This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
p =p
balancing
augmenting
Augmented PAT
balanced PAT
=p
Nw
Stop
>p
Scaled PAT
scaling
Fig. 4. The GO structure For all Node in PAT such that: ((Node.Skel=Farm or Node.Skel=Map or Node.Skel=Reduce or Node.Skel=Scan or Node.Skel=Comp) and marked(Node)=false) Node.Nw=Nopt(Node.skel,M) Node.Tserv=Topt(Node.skel,M) mark(Node)) endfor (phase 1)
For all Node in PAT such that: ((Node.Skel=Pipe or Node.Skel=Comp) and marked(children(Node)) and marked(Node)=false) Update(Node.Skel) Node.Nw=Nopt(Node.skel,M) Node.Tserv=Topt(Node.skel,M) mark(Node)} endfor (phase 2)
Fig. 5. The two phases of the GO init algorithm
1. balancing: in this phase the number of processors employed by each pipeline stage with service time Tserv < Tslow (where Tslow is the service time of the slowest stage) is “reduced”. 2. scaling: when the balancing phase is completed, if Nw is still larger than p, the program must be scaled. The balance procedure computes the minimum number of processors mp such that: T serv ≤ Tslow . In the case: mp = 1 the suitable elimination rule is applied and the PAT is transformed by setting: N ode.skel = Seq. At the end of the balancing phase, three optimization cases may occur: 1. Nw = p GO terminates; 2. Nw < p GO performs the augmenting phase; 3. Nw > p GO performs the scaling phase; The difficulty of scaling the program arises from the fact that many transformations may reduce the number of processors and GO must select which one leads to the optimum. Other works have demonstrated [1] that using a gradient method the optimizer may stop in a local optimum while the global optimum
664
Andrea Zavanella
could have been reached by accepting not optimal moves along the computation. Therefore the choice made in GO is to apply an exhaustive search. GO generates the transformations that reduce the value of Nw to p creating a sequence of sets scal P AT (i) containing the PATs obtained using Nw − i processors. The solution must be located in scal P AT (Nw − p) where all the PATs are filled and a simple search of the minimum service time is performed to find the best implementation. We have the following assertion: Assertion 1 (scaling) The complexity Tscal of scaling is bounded by: Tscal ≤ (Ntr ! Nsk )Nw −p ! Nsk Where Ntr is the number of transformations of our system and Nsk is the number of skeletons in the program. Since the transformations reducing the number of processors are: (a) cutting processors from Farm, Map etc., (b) replacing a Pipe with Comp, (c) collapsing a Pipe to a Seq, in our system Ntr = 3. 4.4
Augmenting Parallelism
This phase exploits the available p − Nw processors to minimize the program cost. Since the starting PAT uses the maximum parallelism for the user program, the program must be transformed in order to increase parallelism. Two types of suitable transformations can be applied: (a) the insertion of Pipe in place of Comp (b) the Farm insertion (since the other parallel constructors already use the optimal number of processors). We have the following assertion: Assertion 2 (Augmenting) the complexity of augmenting is bounded by: Taug < (NSeq + NComp )(p−Nw ) ! Nskel where NComp and NSeq are the number of Comp and Seq constructors respectively. The cost of the augmenting algorithm is furtherly reduced using a pruning technique to decrease the number of generated solutions. In practice the only PATs generated and compared are those in which the number of processors is smaller than p. The pseudo code of the augmenting algorithm is shown in Fig. 6. The procedure Enumerate inserts in solutions the allocations of processors to constructors satisfying the constraint that the number of processors assigned does not exceed Nopt . The Procedure Prune eliminates the solutions where the number of processors exceeds p. Finally the procedure Generate builds the suitable PATs and Findopt selects the solution with the minimum T serv.
5
Case Study
A simple example of the behavior of the GO procedures has been provided using as a case study an image analysis toy-program: IA. IA is a synthetic program
The Skel-BSP Global Optimizer
665
int assigned[1..Nseq+Ncomp]; int max[1..Nseq+Ncomp]; int solutions[1..maxsol;1..Nseq+Ncomp]; int Ntr,Nsol; real Tserv[1..maxsol]; Ntr=Nseq+Ncomp; for i = 1 to Ntr select(Node,i); max[i]=max(Nopt(Node.skel,M),p-Nw) endfor Enumerate(assigned,max,solutions); Prune(Solutions,Nsol); Generate(Solutions,Tserv); Findopt(Solutions,Tserv,assigned);
Fig. 6. The GO augmenting algorithm
including a four stages pipeline where the stages implement: (1) the input from file; (2) the filtering of data; (3) a convolution like computation; (4) the output on file. The sintax tree of IA written in Skel-BSP is shown in Figure 7. The application has been profiled on top of Backus a 10 PCs cluster with PII 266 Mhz CPU and 128 Mbytes RAM running Linux. The machine parameters (M ) and the IA param list are shown in Tab. 2. Using the values in Tab. 3 the GO produces the transformation visualized in Fig. 8. Pipe
Seq
Input
Farm
Seq
Filter
Comp
Map
Seq
Reduce
Seq
Conv
Output
Seq
Binop
Fig. 7. The Skel-BSP structure of IA
6
Conclusions and Related Work
The paper shows that GO may adapt the structure of a skeletal application to optimize its running time on a specific target architecture. Skel-BSP programs
666
Andrea Zavanella g∞ (µ sec) L (µsec) N1/2 (bytes) p (processors) X 0.8 1500 64 10 X tin tf ilter tconv tred tout Tseq (msec) 10 20 80 30 15 din df ilt dconv dred dout sizes (Mbytes) 32 32 32 32 32 M (Backus)
Table 2. The parameters of Backus and the IA param list
Skeleton Nopt (proc) Topt (msec) Farm 6 8 Map 10 10 Reduce 10 4
Table 3. The values of Nopt and Topt for the IA example on Backus
must be simply recompiled to modify their parallel behavior, therefore Skel-BSP seems a promising approach to reach performance portability. The GO exploit a set of formula providing the costs of optimized implementation of each skeletons based on the EdD-BSP cost model. A related approach is the transformational frame proposed by Gorlatch et al. [7] where an “ad hoc” cost model is proposed to drive a semi-automatic transformation system. The future development of the GO will include: – evaluating a general approch to efficiently implement the current GO optimization algorithm; – an extensive validation on several parallel architectures; finally the set of transformation rules will be enlarged.
References [1] M. Aldinucci, M. Coppola, and M. Danalutto. Rewriting skeleton programs: how to evaluate the data-parallel stream-parallel tradeoff. In Proceedings of International Workshop on Constructive Methods for Parallel Programming, number MIP-9805 in Technical Report, University of Passau, 1998. [2] O. Bonorden, B. Juurlink, I. von Otte, and I. Rieping. The paderborn university bsp (pub) library - design, implementation and performance. In Proceeding of 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing (IPPS/SPDP), 1999. [3] M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computation. The MIT Press, Cambridge, Massachusetts, 1989. [4] M. Danelutto, R. Di Meglio, S. Orlando, S. Pelagatti, and M. Vanneschi. A methodology for the development and the support of massively parallel programs. In D. B. Skillicorn and D. Talia, editors, Programming Languages for Parallel Processing. IEEE Computer Society Press, 1994.
The Skel-BSP Global Optimizer pipe([10,20,80,30,15];[32,32,32,32]),18,20
seq ([10];[32]),1,10
farm([20];[32]),6,8
comp([80,30];[32,32]),10,14
667
pipe([10,20,80,30,15];[32,32,32,32]),12,24
seq ([14];[32]),1,15
seq ([10];[32]),1,10
farm([20];[32]),2,15
comp([80,30];[32]),8,24
seq ([14];[32]),1,15
Balancing seq ([20];[32]),1,20
(1)
fully parallel
seq ([20];[32]),1,20
map ([80];[32]),10,10
red ([30];[32]),10,4
map ([80];[32]),8,16
red ([60];[32]),8,8
seq ([80];[32]),1,80
seq ([30];[32]),1,30
seq ([80];[32]),1,80
seq ([30];[32]),1,30
(2)
balanced PAT
pipe(param),10,32
Scaling seq ([10,20];[32,32]),1,30
comp([80,30];[32]),7,32
seq ([14];[32]),1,15
(rules 2-5)
(3) scaled PAT
map ([80];[32]),7,20
red ([60],[32]),7,12
seq([8];[32]),1,80
seq([30];[32]),1,30
Fig. 8. The optimization of IA
[5] J. Darlington, Y.-K. Guo, Hing Wing To, and J. Yang. Functional skeletons for parallel coordination. Lecture Notes in Computer Science, 966:55–67, 1995. [6] P. De La Torre and C. P. Kruskal. Submachine locality in the bulk synchronous setting. Lecture Notes in Computer Science, 1124:352–360, 1996. [7] S. Gorlatch and S. Pelagatti. A transformational framework for skeletal programs: Overview and case study. In Jose Rohlim, editor, Proc. of Parallel and Distributed Processing. Workshops held in Conjunction with IPPS/SPDP’99, volume 1586 of LNCS, pages 123–137, Berlin, 1999. Springer. [8] J. M. D. Hill, B. McColl, D. — C. Stefanescu, M. W. Goudreau, K. Lang, S. B. Rao, T. Suel, T. Tsantilas, and R. H. Bisseling. BSPlib: The BSP programming library. Parallel Computing, 24(14), 1998. [9] R. Hockney. Performance parameters and benchmarking of supercomputers. Parallel Computing, 17(10-11):1111–1130, December 1991. [10] S. Pelagatti. Structured Development of Parallel Programs. Taylor & Francis, 1997. [11] L. G. Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103, August 1990. [12] A. Zavanella. Optimising Skeletal Stream Parallelism on a BSP Computer. In P. Amestoy, P. Berger, M. Dayde, I. Duff, V. Fraysse, L. Giraud, and D. Ruiz, editors, Proceedings of EURO-PAR’99, number 1685 in LNCS, pages 853–857. Springer-Verlag, 1999. [13] A. Zavanella. Skel-Bsp: Performance Portability for Skeletal Programming. In M. Bubak, H. Afsarmanesh, R. Williams, and B. Hertzberger, editors, Proceedings of HPCN 2000, volume 1823 of LNCS, pages 290–299, 2000. [14] A. Zavanella. Skeletons and BSP: Performance Portability for Parallel Programming. PhD thesis, Dipartimento di Informatica Univesita di Pisa, 2000. [15] A. Zavanella and S. Pelagatti. Using BSP to Optimize Data-Distribution in Skeleton Programs. In P. Sloot, M. Bubak, A. Hoekstra, and B. Hertzberger, editors, Proceedings of HPCN99, volume 1593 of LNCS, pages 613–622, 1999.
A Theoretical Framework of Data Parallelism and Its Operational Semantics Philippe Gerner and Eric Violard LSIIT-ICPS, Universit´e Louis Pasteur, Strasbourg {gerner,violard}@icps.u-strasbg.fr
Abstract. We developed a theory in order to address crucial questions of program design methodology. We think that it could unify two concepts of data parallel programming that we consider fundamental as they concern data locality expression: the notions of alignment in HPF and shape in C∗ . In this article, we aim at exploring the impact of program transformations on efficiency. For this, we define a formal operational semantics associated with the aforementioned theory.
1
Introduction
The interest of the concepts that lie behind any programming language is that they express the relationship between what the programmer knows about the problem he wants to address and what the compiler knowns about the architecture on which the program will execute. These concepts consist then in an abstract knowledge that the programmer and the compiler share and can use to interact and eventually perform the best implementation. This is even more true when the architecture is a parallel one, when both the programming and compiling tasks are more complex. The programmer and compiler should influence each other in order to paliate these difficulties. For example, based on these concepts, one could define a language in such a way that a program should be transformed by both entities in a dialog started by the programmer and ended by the compiler. The main point is then to find a good medium, i.e., the concepts that the two entities can handle. In particular, a major interest of the data parallelism paradigm is that it enables the programmer to describe its “parallel” algorithm by choosing a wellknown “sequential” one and then focusing on the expression of data locality, while the compiler can be left in charge of the distribution onto physical processors, communications management, etc. In some sense, it reveals that the concepts for expressing data locality are of main interest in data parallel programming. As examples, let us consider standard data parallel programming languages like HPF [1] and C∗ [6]: they both provide their own way of expressing data locality: alignment of arrays for HPF, and shape of parallel variables for C∗ . In a previous article [7], we proposed a theory which offers a notion of data locality in which those of HPF and C∗ could join up. The purpose of the article was, to provide a model of this theory, to serve as a formal basis for proving the correctness of data parallel programs. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 668–677, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Theoretical Framework of Data Parallelism and Its Operational Semantics
669
In the present article, we focus on the execution of data parallel programs, with the objective of comparing the efficiency of the different versions of a program. Estimating the efficiency of a program needs to define its operational semantics.
2 2.1
Our Theory Objects
Our objects are called shaped data fields. A shaped data field is mainly a container of values. Containers without values are sometimes referred to as shapes–in [2], B. Jay defines a very general and abstract notion of shape via a categorical pullback. The originality of our theory consists in its particular notion of shape. A shape is a relation σ between a set of indices and a set of locations. Both sets are subsets of ZZ n . The indices of a shape serve to access values, and n∈IN
if a value v is accessed through index i, then value v is said to be located at all the locations which are related to i by σ. This relation is such that a location is related to at most one index. In Sect. 4, this relation will serve to describe a placement of the values onto a set of virtual processors. A shaped data field results from the association of a shape with a data field. In the literature, indexed collections of values, without locations, are often referred to as data fields (see Alpha [5] and Lisper’s formalism [4], as examples). The values of our data fields are indexed by integer tuples.
indices
j
indices
j
i
i
3
indices 3
locations 1
j i
1
(a)
3
j
3
1 i
locations j
i
(b)
(c)
Fig. 1. (a) A shaped data field, (b) its shape and (c) its data field
2.2
Operations
The theory defines three kinds of operations on shaped data fields: – a re-indexing concerns both the shape and the data field of a shaped data field: it changes the indices while keeping the values, and their locations unchanged.
670
Philippe Gerner and Eric Violard
– a re-locating does not affect the shape; it changes the locations of some of the values of the data field (with possible duplication or deletion). This change of locations is expressed through a change of indices. Any re-indexing or re-locating is determined by a (partial) function from the indices of the new shaped data field to the indices of the old one. For example, Fig. 2(b) shows two shaped data fields: the shaped data field on the right results from a re-locating applied to the shaped data field on the left. This re-locating is determined by the function which associates point of coordinates (1, j) with any point (i, j) in the rectangle [1..3] × [1..2] of ZZ 2 . Fig. 2(a) illustrates the re-indexing determined by the same function. – a global computation applies the same “scalar” operation on all values. Any classical arithmetical or logical operation on values induces a global computation. In particular, any binary operation on values defines a global computation which combines two shaped data fields having the same shape.
j
j
j
i 2 1
j
i
1
2
2
i 2
1
1
1
2 1
(a)
2
i 2
2
1
1
1
2
2
1
(b)
Fig. 2. (a) A re-indexing and (b) a re-locating
2.3
A Minimal Notation Set
Here we introduce a minimal notation set for expressing problems or programs: both are particular cases of what we call statements. A statement is a finite set of equations whose variables are shaped data fields. Each equation connects two expressions built using the operations on shaped data fields. A variable is an uppercase letter (e.g., A, B, X, . . . ). The classical arithmetical or logical operations are overloaded with global computations (e.g., A + B). We use the following notation for expressing re-indexing and re-locating operations, and also more general cases of dependences: X.F , where F is a pair (h, g) of partial functions, is the result of applying first the re-indexing determined by h, and then the re-locating determined by g, to X. When g is the identity, X.F is a re-indexing, and when h is the identity, X.F is a re-locating. In order to denote partial functions, we use the classical lambda-calculus notation λx.e, and f \ D for the restriction of function f to domain D. For example, assuming A and B are the shaped data fields on Fig. 2(b), then they satisfy the equation B = A.Spread , where Spread is pair (h, g), h is the
A Theoretical Framework of Data Parallelism and Its Operational Semantics
671
identity (on D), and g is partial function λ(i, j).(1, j) \ D, with D = { (i, j)| 1≤i≤3 ∧ 1≤j≤2 } (since h is the identity, Spread defines a re-locating).
3
Theory Adequacy
This section is intended to link our theory with data locality expression in HPF and in C∗ and also to show that our theory makes a better compromise than these languages between compiler and programmer. 3.1
Example 1
Let us consider the two following programs in HPF and C∗ : REAL A(0:3), B(1:4), C(1:4) !HPF$ TEMPLATE X(1:8) !HPF$ ALIGN A(I) WITH X(I+1) !HPF$ ALIGN B(I) WITH X(2*I) ... FORALL ( I=1:4 ) C(I) = A(I-1) + B(I) END FORALL ...
shape [9]vector’; real: vector’ A’,B’,C’; ... C’ = A’ + [.*2]B’; ...
In the HPF program, template X is necessary for aligning arrays A and B the way we want. The C* program is equivalent to the HPF program to the extent that it expresses a correspondance between array elements and virtual processor that respects the alignment constraints of the HPF program. In the C∗ program, arrays A’, B’ and C’ have the same shape, so that for every i, value A’[i] is on the same virtual processor as values B’[i] and C’[i]. A’, B’ and C’ are the result of having the values of A, B, C, placed in such a way that the alignment constraints are respected: for example, A’[4] (resp. B’[4]) contains the value of element A[3] (resp. B[2]), so that A[3] and B[2] are on the same virtual processor. In the HPF program, the array C is not aligned: the placement of values of C is left in charge of the compiler. The C* program is more prescriptive as it explicits a particular placement of values and the required communications. In some sense, translating the HPF program to C* reveals the difficulties of the HPF compiler in handling the alignment directives as much as those of the programmer in writing its program in C*. In our theory, these programs (and their operational meaning) can be expressed by the following statement: C = A.Shift + B.Id
672
Philippe Gerner and Eric Violard
where A, B and C are shaped data fields and represent arrays A, B and C, and Shift , Id are pairs (h, g), (h , g ) of partial functions such that h ◦ g equals λ(i).(i−1) \ [1..4] and h ◦ g equals the identity on [1..4]. This is enough to express the dependences between the values of arrays A, B and C: the functions h ◦ g and h ◦ g express that, for any i ∈ [1..4], C[i] depends on A[i−1] and B[i]. In the HPF program, these dependences are expressed by the FORALL block. Now we precise the functions h, g, h , and g , for expressing the placement used in C∗ program (which respects the alignment constraints of the HPF program), which is depicted on Fig. 3: −1
h = g g = λ(i).(2×i) \ [1..4]
h = λ(i).(i−1) g = id \ [1..4]
A
B
a0
a1 b1 a2
a3 b2
c1
c2
c4
c3
b3
b4
C
Fig. 3. The placement of values
In comparison with HPF, in our theory a shape is associated with each variable, so that each value of any variable is placed on a set of locations, including the values of variable C in the above statement. And in comparison with C∗ , the shapes are inferred from the statement, rather than being found by the programmer, as it was necessary for writing the C∗ program; and also, contrarily to C∗ , the indices are not names for locations. 3.2
Example 2
Here we discuss the Cooley-Tukey algorithm for the one-dimensional, unordered, radix-2 FFT. Our starting point is the iterative formulation of FFT in [3], p.380. The algorithm is defined by the following set of recurrent equations, where vector Y [0..n−1] is the Fourier transform of vector X[0..n−1], and n = 2r : R[i, m] = X[i], for all (i, m) ∈ D0 , (1) R[i, m] = R[i↓m, m−1] + (R[i↑m, m−1]×W [im]), for all (i, m) ∈ D1..r , (2) Y [i] = R[i, m],
for all (i, m) ∈ Dr ,
(3)
where R is a temporary n×r-matrix, W is a given n-vector that contains the powers of the primitive root of unity in the complex plane (also known as twiddle factors), and the following subdomains are considered:
A Theoretical Framework of Data Parallelism and Its Operational Semantics
673
D0 = { (i, m) | 0≤i
(4)
R.Id 1..r = R.Even + (R.Odd × W.Spread ) Y.Vector r = R.Id r
(5) (6)
where the definitions of all pairs of functions and dependence functions are reported in Table 1. R.Id 0 , R.Id 1..r and R.Id r each identifies a piece of R on subdomains D0 , D1..r and Dr , respectively. Table 1. The definition of a particular placement. Id 0 Id 1..r Id r Vector 0 Even Odd Spread Vector r
= = = = = = = =
(h, g) (id, id \ D0 ) (id, id \ D1..r ) (id, id \ Dr ) (λ(i, m).(i) \ D0 , id) (id, λ(i, m).(i↓m, m−1) \ D1..r ) (id, λ(i, m).(i↑m, m−1) \ D1..r ) (λ(i, m).(i m) \ D1..r , id) (λ(i, m).(i) \ Dr , id)
(h ◦ g) id \ D0 id \ D1..r id \ Dr λ(i, m).(i) \ D0 λ(i, m).(i↓m, m−1) \ D1..r λ(i, m).(i↑m, m−1) \ D1..r λ(i, m).(i m) \ D1..r λ(i, m).(i) \ Dr
By this statement, vector X (resp. Y ) is aligned with the first (resp. last) column of matrix R. Moreover, the twiddle factors have been replicated onto the matrix so that the dependence defined by Spread does not involve any communication. This statement describes the binary-exchange algorithm if the indices of variable R are identified with locations. In the following, R will be referred to as the template of this statement.
674
4
Philippe Gerner and Eric Violard
Operational Semantics
All the statements we have seen in the previous section are computable. They are examples of what we call well-formed statements. This class of statements is associated with a proof system [7] and an operational semantics. 4.1
Well-Formed Statements
Informally, any well-formed statement is such that the shape of each variable can be inferred uniquely from the shape of one of them. Moreover, we restrict ourselves to statements in single-assignment form. Definition 1. A well-formed statement, is a statement, say P : – with some variables declared as input variables, other ones declared as output variables, and with a single one declared as the template of the statement (and referred to as T ); – whose every equation has the form X0 .(h0 , g0 )=φ(X1 .(h1 , g1 ), . . . , Xn .(hn , gn )) , is a variable of P , where φ is a global computation, and for all k ∈ [1..n], Xk ZZ n , g0 = id|D , hk and gk are partial functions from I to I, where I = n∈IN
where id is the identity on I, and either h0 = id (case 1), or h0 =id, and ∀k ∈ [1..n], hk = id (case 2); – such that, the following set of equations, denoted S(P ), has a unique solution for any σT . Its variables are partial functions from I to I, and we have one variable σX for each variable X of P . S(P ) is the set of all the equations σXk = hk ◦ σX0 , k ∈ 1..n, induced from every equation of P of case 1, and equations σX0 = h0 ◦ σXk induced from equations of case 2; – and such that, if a variable X appears in more that one left-hand side, in terms X.(h1 , id |D1 ), . . . , X.(hq , id |Dq ), then ∀k, k ∈ 1..q, k =k ⇒ Dk ∩ Dk = ∅. Relative Placement of Variables. The system S(P ) associated with a wellformed statement P defines the placement of the values of each shaped data field on a set of locations. For each variable X, this placement is defined by σX the following way: the values of X accessed by index i are placed at locations σX −1 (i) (in the sequel, this set is denoted loc(X, i)). S(P ) defines the functions σX as follows. The set of locations is I, and the values of template variable T are considered placed on I with σT = id, so that at most one value is accessed through index i, and this value is then located at location l = i. The placement of all the other variables are uniquely inferred from this one.
A Theoretical Framework of Data Parallelism and Its Operational Semantics
4.2
675
States and Transitions
A state is a set of stores. We have one store for each location. The store associated with location l ∈ I is a partial function ρl : Var × I ( V , where Var is the set of variables of statement P , and V is a set of values. Intuitively, ρl (X, i) is the current value of X, indexed (in the index set of X) by i, and placed at location l. A transition describes a change in the store of a location. So, a transition induces a change from one state to another. The transitions can occur concurrently. The execution begins with an initial state. In the initial state, if X is an input variable, and ρl (X, i) is defined, then l ∈ loc(X, i), and ∀l ∈ loc(X, i), ρl (X, i) = ρl (X, i); and if X is not an input variable, then ∀l ∈ I, ∀i ∈ I, ρl (X, i) is undefined. From the initial state, the transitions apply, in a non-deterministic way, and concurrently, until no more transition can apply. Transitions. For every equation X0 .(h0 , g0 ) = φ(X1 .(h1 , g1 ), . . . , Xn .(hn , gn )), of P , we have the following transition rule, where i ∈ I is any index, l0 , l1 , . . . , ln ∈ I are any locations, and dk is for hk ◦ gk , k ∈ 0..n: UNDEF0 ∧ l0 ∈ loc(X0 , d0 (i)) ∀k ∈ 1..n, DEFk ∧ (l0 = lk ∨ l0 ∈loc(Xk , dk (i))) ρl0 −→ ρl0 [(X0 , d0 (i)) ← φ(ρl1 (X1 , d1 (i)), . . . , ρln (Xn , dn (i)))]
(7)
where ρ[(X, i) ← v] is the store ρ except that it maps (X, i) to value v, UNDEF0 means that d0 (i) is defined and ρl0 (X0 , d0 (i)) is undefined, and DEFk (k ∈ 1..n) means that dk (i) and ρlk (Xk , dk (i)) are defined. Computations: Assuming the Owner-Computes Rule, it is to be understood that, in the application of a transition, the computation φ(. . .) is made at location l0 . Note that, when loc(X0 , d0 (i)) contains more than one location, the calculus will be made several times. Such a case is useful when duplicating the calculus avoids some communications. Communications: When l0 =lk , there is a communication for sending the value from location lk , to location l0 , because the computation on l0 needs this value. The transition does not apply if l0 =lk ∧ l0 ∈ loc(Xk , dk (i)), because in this case the value at location lk will be available locally, that is, at location l0 .
5
Example
Let us consider again the FFT example in Sect. 3.2, with n = 24 . Suppose we are given 4 physical processors. Equation (5) is the core of the algorithm. R is the template of the statement, so σR = id, which means that if R contains a value at index (i, m), then this value lies on location (i, m). So, the transition rule associated to this equation breaks down to the following rule:
676
ρi,m
Philippe Gerner and Eric Violard
(i, m) ∈ D1..r ∧ ρi,m (R, (i, m)) is undefined ∧ ρi↓m,m−1 (R, (i↓m, m−1)) is defined ∧ ρi↑m,m−1 (R, (i↑m, m−1)) is defined −→ ρi,m [(R, (i, m)) ← ρi↓m,m−1 (R, (i↓m, m−1)) + (ρi↑m,m−1 (R, (i↑m, m−1)) × ρi,m (W, im))]
(8)
This rule is the only rule, among those induced by the statement, that generates communications. The communications come from the fact that indices (i, m), (i↓m, m−1) and (i↑m, m−1) represent different locations (because R is the template), so that for doing the computation at (i, m), 2 communications are needed for getting the required values which sit at locations (i↓m, m−1) and (i↑m, m−1). We get a total of 2 × card(D1..r ) = 2 × n × r = 2 × 16 × 4 = 128 communications. The rule also shows that the axis of index m follows the direction of time during execution of the program. So, an obvious projection onto physical processors is along this axis. For each (i, m), either i↓m = i or i↑m = i, so that this projection reduces the number of communications by a half, to 64. But we have only 4 processors, not 16, and in this case what is classically done is, distributing 4 contiguous elements of the array (on the first dimension of R) onto one processor. The number of communications is then reduced further, from 64 to 64 − (2 × 16) = 32 communications (because the dependences in the last 2 steps require no communications). A different algorithm could be described, characterized by another placement of values onto the processors: the two-dimensional transpose algorithm. This new algorithm can be expressed by the same statement as in Sect. 3.2, except that we re-index variable R, obtaining a new template variable R . This is expressed by just adding to the statement the following equation: R.(transf , id) = R , √ √ (j+i×√n, m), if 0≤i, j<√n ∧ 0≤m
(9)
This transformation is sketched on Fig. 4. Since we have only changed the template, the number of (virtual) communications is the same for this new algorithm. The interpretation of the new template is the following one: the third axis of R corresponds √ √ to time, and the first two axes give a square of locations of dimensions n × n. Again, an obvious projection onto processors is along the third axis. The idea of the transpose algorithm is, if given 4 processors, we distribute each column onto a processor for the first r/2 = 4/2 = 2 iterations, and each line onto a processor for the last r/2 = 2 iterations: then, with the above projection, no communication is required for the first 2 iterations, nor for the last 2 iterations. But a re-distribution generating communications, which corresponds to the transposition of a matrix of values, is necessary between these two big steps of
A Theoretical Framework of Data Parallelism and Its Operational Semantics
677
m i
0
i
0
1
2
3
m j
1 2 3
Fig. 4. A sketch of the transformation in the case n=16: the domain of R is first rearranged as a parallelepipede and then divided into two parts; last step consists in transposing the matrices forming the first of these two parts. the algorithm. So, the total number of communications is of commu√ the number √ √ nications required for the transposition, which is ( n × n) − n = 16 − 4 = 12.
6
Conclusion
In our opinion, in reference to automatic parallelization, the programmer must provide some additional information to the compiler in order that the compiler can reach more efficiency. In particular, data locality expression is of great importance in data parallelism. Therefore, we proposed a theoretical framework which unify the two main concepts for expressing data locality: alignment of arrays in HPF and shape declaration in C∗ . This framework allows the programmer and the compiler to share a minimum knowledge to reach efficiency. We think that it could help both the programmer and the compiler to transform the program in order to reach the best implementation.
References [1] High Performance Fortran Forum. High Performance Fortran Language Specification, Version 1.0, January 1993. [2] C.B. Jay. A semantics for shape. Sc. of Computer Programming, 25:251–283, 1995. [3] V. Kumar, A. Grama, A. Gupta, and G. Karypis. Introduction to Parallel Computing: Design and Analysis of Algorithms. Benjamin/Cummings, 1994. [4] B. Lisper. Data Parallelism and Functional Programming. LNCS 1132-Tutorial Series, 1996. [5] C. Mauras. Alpha : un langage ´equationnel pour la conception et la programmation d’architectures parall` eles synchrones. PhD thesis, U. Rennes, 1989. [6] Thinking Machines Corp. C* Programming Guide, November 1990. [7] E. Violard. What really is data parallelism? Technical Report RR 00-01, ICPS, Universit´e Louis Pasteur, January 2000.
A Pattern Language for Parallel Application Programs Berna L. Massingill1 , Timothy G. Mattson2 , and Beverly A. Sanders3 1
3
Department of Computer and Information Science and Engineering University of Florida; Gainesville, FL [email protected]† 2 Parallel Algorithms Laboratory, Intel Corporation [email protected] Department of Computer and Information Science and Engineering University of Florida; Gainesville, FL [email protected]
Abstract A design pattern is a description of a high-quality solution to a frequently occurring problem in some domain. A pattern language is a collection of design patterns that are carefully organized to embody a design methodology. A designer is led through the pattern language, at each step choosing an appropriate pattern, until the final design is obtained in terms of a web of patterns. This paper describes a pattern language for parallel application programs. The goal of our pattern language is to lower the barrier to parallel programming by guiding a programmer through the entire process of developing a parallel program.
1
Introduction
Parallel hardware has been available for decades, and is becoming increasingly mainstream. Parallel software that fully exploits the hardware is much rarer, however, and mostly limited to the specialized area of supercomputing. We believe that part of the reason for this is that most parallel programming environments, which focus on the implementation of concurrency rather than higherlevel design issues, are simply too difficult for most programmers to risk using them. A design pattern describes, in a prescribed format, a high-quality solution to a frequently occurring problem in some domain. The format is chosen to make it easy for the reader to quickly understand both the problem and the proposed solution. Because the pattern has a name, a collection of patterns provides a vocabulary with which to talk about the these solutions. A pattern language is a structured collection of design patterns, with the structure chosen to lead the user through the collection of patterns in such a †
We acknowledge the support of Intel Corporation, the National Science Foundation, and the Air Force Office of Scientific Research. Current address: Department of Computer Science, Trinity University, San Antonio, TX; [email protected].
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 678–681, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Pattern Language for Parallel Application Programs
679
way that complex systems can be designed using the patterns. At each decision point, the designer selects an appropriate pattern. Each pattern leads to other patterns, resulting in a final design in terms of a web of patterns. Thus a pattern language embodies a design methodology and provides domain-specific advice to the application designer. (In spite of the overlapping terminology, a pattern language is not a programming language.) The first pattern language was elaborated by Alexander [1] and addressed problems from architecture, landscaping, and city planning. More recently, design patterns were introduced to the software engineering community by Gamma, Helm, Johnson, and Vlissides [3] via a collection of design patterns for object-oriented programming. This paper describes a pattern language for parallel application programs. The goal is to lower the barrier to parallel programming by guiding a programmer through the entire process of developing a parallel program. The target audience is experienced programmers who may lack experience with parallel programming. The programmer brings to the process a good understanding of the actual problem to be solved and works through the pattern language to obtain a detailed parallel design or perhaps working code. The current state of the whole pattern language may be seen at our Web site [7]; the patterns are extensively hyperlinked, allowing the programmer to work through the pattern language by following links. In this paper, space constraints allow only an overview of the pattern language and a brief discussion of related approaches; see [7] and [8] for complete patterns and examples of their use.
2
Organization of the Pattern Language
The pattern language is organized into four design spaces — FindingConcurrency, AlgorithmStructure, SupportingStructures, and ImplementationMechanisms — which form a linear hierarchy, with FindingConcurrency at the top. FindingConcurrency. This design space is concerned with structuring the problem to expose exploitable concurrency; the programmer working at this level focuses on high-level algorithmic issues and reasons about the problem to expose potential concurrency. The space includes four major patterns: GettingStarted helps the programmer understand where to start in solving a problem using a parallel algorithm. DecompositionStrategy helps the programmer decide whether the problem should be decomposed based on a data decomposition, a task decomposition, or a combination of the two. DependencyAnalysis helps the programmer understand how the elements of the decomposed problem depend on each other. Finally, Recap helps the programmer bring together the results from the other patterns and prepare to work in the AlgorithmStructure design space. AlgorithmStructure. This design space is concerned with structuring the algorithm to take advantage of the potential concurrency exposed in the previous level. Patterns in this space describe overall strategies for exploiting concurrency.
680
Berna L. Massingill, Timothy G. Mattson, and Beverly A. Sanders
They can be divided into the following four groups, plus ChooseStructure, which helps the programmer select an appropriate pattern from those in this space. “Organize by ordering” patterns. These patterns are used when the ordering of groups of tasks is the major organizing principle for the parallel algorithm. An example is PipelineProcessing, which decomposes the problem into ordered groups of tasks connected by data dependencies. “Organize by tasks” patterns. These patterns are those for which the tasks themselves are the best organizing principle (“task parallelism”). Two examples are EmbarrassinglyParallel, which decomposes the problem into a set of independent tasks that can be executed concurrently (examples are algorithms based on task queues and random sampling); and DivideAndConquer, which solves the problem by recursively dividing it into subproblems, solving the subproblems independently, and then combining the subsolutions. “Organize by data” patterns. These patterns are those for which the decomposition of the data is the major organizing principle in understanding the concurrency (“data parallelism”). An example is GeometricDecomposition, which decomposes the problem space into discrete subspaces and solves the problem by computing solutions for the subspaces, with the solution for each subspace typically requiring data from a small number of other subspaces (examples include grid-based computations). SupportingStructures. This design space represents an intermediate stage between the problem-oriented AlgorithmStructure patterns and the machine-oriented ImplementationMechanisms “patterns”; it includes both patterns that represent program-structuring constructs (e.g., ForkJoin) and patterns that represent commonly used shared data structures (e.g., SharedQueue). Ideally patterns in this space would be implemented as part of a library. ImplementationMechanisms. This design space is concerned with how the patterns of the higher-level spaces are mapped into particular programming environments. We use it to provide pattern-based descriptions of common mechanisms (e.g., barriers and message-passing) for managing and coordinating processes or threads. Patterns in this design space, like those in the SupportingStructures space, describe entities that strictly speaking are not patterns at all, but which we include in our pattern language to provide a complete path from problem description to code.
3
Related Work
Design patterns and pattern languages were first introduced into software engineering in [3]. Early work on patterns dealt mostly with object-oriented sequential programming, but more recent work [10, 4, 9] addresses concurrent programming as well. Algorithmic skeletons and frameworks capture very high-level patterns that provide the overall program organization, with the user providing
A Pattern Language for Parallel Application Programs
681
lower-level code specific to an application. Skeletons, as in [2], are typically envisioned as higher-order functions, while frameworks often use object-oriented technology. Particularly interesting is recent work [5] on using design patterns to generate frameworks for parallel programming from pattern template specifications. Programming archetypes [6] combine elements of all of the above categories, capturing common computational and structural elements at a high level and also providing a basis for implementations including both frameworks and code libraries, but do not directly address the question of how to choose an appropriate archetype for a particular problem.
4
Conclusions
We have described a pattern language for parallel programming. Currently, the top two design spaces (FindingConcurrency and AlgorithmStructure) are relatively mature, with several of the patterns having undergone scrutiny at a writers’ workshop for design patterns [8]. The lower two design spaces are still under construction, but the pattern language is usable, and several case studies are in progress (see [7]). Preliminary results of the case studies and feedback from the writers’ workshop leave us optimistic that our pattern language can indeed achieve our goal of lowering the barrier to parallel programming.
References [1] Christopher Alexander, Sara Ishikawa, and Murray Silverstein. A Pattern Language: Towns, Buildings, Construction. Oxford University Press, 1977. [2] M. I. Cole. Algorithmic Skeletons: Structured Management of Parallel Computation. MIT Press, 1989. [3] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995. [4] Doug Lea. Concurrent Programming in Java: Design Principles and Patterns. Addison-Wesley, 1997. [5] S. MacDonald, D. Szafron, J. Schaeffer, and S. Bromling. From patterns to frameworks to parallel programs, 1999. Submitted to IEEE Concurrency, August 1999; see also http://www.cs.ualberta.ca/~stevem/papers/IEEECON99.ps.gz. [6] Berna L. Massingill and K. Mani Chandy. Parallel program archetypes. In Proceedings of the 13th International Parallel Processing Symposium (IPPS’99), 1999. [7] Berna L. Massingill, Timothy G. Mattson, and Beverly A. Sanders. A pattern language for parallel application programming. Available at http://www.cise.ufl.edu/research/ParallelPatterns. [8] Berna L. Massingill, Timothy G. Mattson, and Beverly A. Sanders. Patterns for parallel application programs. In Proceedings of the Sixth Pattern Languages of Programs Workshop (PLoP99), 1999. [9] Jorge Ortega-Arjona and Graham Roberts. Architectural patterns for parallel programming. In Proceedings of the 3rd European Conference on Pattern Languages of Programming and Computing, 1998. [10] D. C. Schmidt. The ADAPTIVE Communication Environment: An objectoriented network programming toolkit for developing communication software. http://www.cs.wustl.edu/~schmidt/ACE-papers.html, 1993.
Oblivious BSP Jesus A. Gonzalez 1, Coromoto Leon 1, Fabiana Piccoli 2, Marcela Printista2, José L. Roda 1, Casiano Rodriguez 1, and Francisco de Sande1 1
Dpto. Estadística, Investigación Operativa y Computación, Universidad de La Laguna, Tenerife, Spain [email protected] 2 Universidad Nacional de San Luis Ejército de los Andes 950, San Luis, Argentina [email protected]
Abstract. The BSP model can be extended with a zero cost synchronization mechanism, which can be used when the number of messages due to receive is known. This mechanism, usually known as "oblivious synchronization" implies that different processors can be in different supersteps at the same time. An unwanted consequence of these software improvements is a loss of prediction accuracy. This paper proposes an extension of the BSP complexity model to deal with oblivious barriers and shows its accuracy.
1 Introduction The asynchronous nature of some parallel paradigms like farms and pipelines hampers the efficient implementation in the scope of a flat-data-parallel global-barrier Bulk Synchronous Programming (BSP [4]) software like the BSPLib [3]. To overcome these limitations, the Paderborn University BSP library (PUB) offers the use of collective operations, processor-partition operations and oblivious synchronizations [1]. In addition to the BSP most common features, PUB provides the capacity to partition the current BSP machine into several subsets, each of which acts as an autonomous BSP computer with their own processor numbering and synchronization points. One of the most novel features of PUB is oblivious synchronization. It is implemented through the bsp_oblsync(bsp,n) function, which does not return until n messages have been received. Although its use mitigates the synchronization overhead, it implies that different processors can be in different supersteps at the same time. The BSP semantic is preserved in PUB by numbering the supersteps and by ensuring that the receive thread buffers messages that arrive out of order until the correct superstep is reached. An unwanted consequence of these software improvements is a loss of accuracy when using the BSP complexity model to predict the running time. Runtime systems oriented to the flat BSP model try to bring actual machines closer to the BSP ideal machine by packing individual messages generated during a A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 682-685, 2000. Springer-Verlag Berlin Heidelberg 2000
Oblivious BSP
683
superstep and optimizing communication time by rearranging the order in which messages are sent at the end of the superstep [3]. This policy reduces the influence of the communication pattern, since it gives place to an AllToAll communication pattern at the end of each superstep. According to this, the actual overlapping of supersteps produced by PUB machine partitioning and oblivious synchronization makes unfeasible the former implementation approach and may lead to congestion (hot spots) and therefore to a wider variability in the observed bandwidth. In the next section we propose an extension of the BSP model for PUB parallel programs: the Oblivious BSP model. The model intends to cover the complete class of PUB programs. The computational times on a CRAY T3E of a PUB Divide and Conquer algorithm were predicted with an error percentage under 3%.
2 The Oblivious BSP Model The execution of a PUB program on a BSP machine X={0,1,...,p-1} consists of supersteps. As a consequence of the oblivious synchronizations, processors may be in different supersteps at a given time. However, still is true that: Supersteps can be numbered starting at 1. The total number of supersteps R, performed by all the p processors is the same. Although messages sent by a processor in superstep s may arrive at another processor executing an earlier superstep r, communications are made effective only when the receiver processor reaches the end of superstep s. Lets assume in first instance that no processor partitioning is performed in the analyzed task T. If the superstep s ends in an oblivious synchronization, we define the set Ωs,i for a given processor i and superstep s as the set • • •
Ωs,i = { j∈X / Processor j sends a message to processor i in superstep s } ∪ { i}
(1)
while Ωs,i = X when the superstep ends in a global barrier synchronization. In fact, this last expression can be considered a particular case of formula (1) if it is accepted that a barrier synchronization carries an all-to-all communication. Processors in the set Ωs,i are called "the incoming partners of processor i in step s". Usually it is accepted that all the processors start the computation at the same time. The presence of partition functions in PUB forces us to consider the most general case in which each processor i joins the computation at a different initial time ξi. The Oblivious BSP time Φs,i(T, X, ξ) when processor i∈X executing task T finishes superstep s is recursively defined by the formulas:
Φ1i = max { w1,j + ξi / j ∈ Ω1,i} + (g h1,i + Lb), i = 0,1,..., p-1 Φs,i = max{Φs-1,j+ws,j / j∈Ωs,i } + (g hs,i + Lb), s=2,..,R, i=0,1,..., p-1
(2)
where ws,i and hs,i respectively denote the time spent in computing and the number of packet units communicated by processor i in step s: hs,i = max {ins,j @ outs,j / j ∈ Ωs,i}, s = 1,...,R, i = 0,1,..., p-1
(3)
684
Jesus A. Gonzalez et al.
and ins,i and outs,i respectively denote the number of packet units incoming/outgoing to/from processor i in the superstep s. The @ operation is defined as max or the sum depending on the input/output capabilities of the network interface. While the gap g has the same meaning that in BSP, the value Lb is related with the start-up latency. This oblivious latency value Lb is different from the barrier synchronization latency value L used in global synchronization superteps. Another issue to discuss is what is the appropriate size of a unit communication packet. This size depends on the target architecture. Rather than exhibiting linear behavior, the communication time takes the form of a piecewise linear function. There is always a slope for small size messages different from the slope for large messages. We define the OBSP Packet Size as the size in which the time/h-relation size curve of the architecture has the first inflection point. Special gap g0 and latency L0 values have to be used for messages of sizes smaller than the OBSP unit packet size. The formula (2) says that processor i in its step s can not start the reception of its messages before it has finished its computing time Φs-1,i + ws,i and so have done the other processors j sending messages to processor i. The formulas charge the communication time of processor i with the maximum communication time of any of its incoming partner processors. The times of a PUB task T in OBSP are given by the vector of times ΦR, j (T, X,ξ) when processor j=0,...,p-1 finishes task T. R is the total number of supersteps, X is the set of processors executing the task T, and ξ = (ξ0 , ..., ξp-1) is the vector of starting times. At any time, the processors are organized in a hierarchy of processor sets. A processor set in PUB is represented by a structure called a BSP object. Let be Q a set of processors (i.e. a BSP object) executing a task T. When all the processors in Q call the function bsp_partition with arguments (t_bsp* Q, t_bsp* S, int r, int* Y) the set Q is divided in r disjoint subsets Si: Q = ∪0 ≤ j ≤ r-1 Sj ; Si = {Y[i-1], ..., Y[i]-1} 1 ≤ i ≤ r-2 and S0 = {0, ..., Y[0]-1} After the partition step each subgroup Si acts as an autonomous BSP computer with its own processor number, message queue and synchronization points. The time when processor j∈Si finishes the task Ti executed by the object BSP Si is given by
ΦRi, j (Ti, Si, Φs-1,j+w*s,j) such that j∈ Si i=0, ..., r-1
(4)
where Ri is the number of supersteps performed in task Ti and w*s,j is the computing time performed by processor j before its call to the bsp_partition function and s is the current superstep number. Observe that subgroups are created in a stack-like order. The two functions bsp_partition and bsp_done incurs no communication. This implies that different processors in a given subset can arrive at the partition process (and leave it) at different times. From the point of view of the parent submachine, the code executed between the call to bsp_partition and bsp_done behaves as computation (i.e. like a call to a subroutine). Another essential difference of the Oblivious Model proposed here with the BSP model is the way the computing time W is carried out. Instead of using a single parameter s to characterize the computational power, our proposal is to return to an approach nearer to the original RAM model: a sequential algorithm A with input of
Oblivious BSP
685
size n has complexity of order O(f(n)) if and only if there are constants C and D such that the time TA,M(n) invested by a computer M executing A on such input holds the equation TA,M(n) ≤ C*f(n)+D. The values of constants C and D can be estimated from the actual times TA,M(n) of the algorithm A on the worst (or average) cases of size n. Table 1 presents the prediction accuracy of the OBSP for a nested parallel recursive Discrete Fast Fourier Transform PUB algorithm [2]. Table 1. Real and OBSP predicted times for the FFT algorithm in the CRAY T3E. Processors Real Predicted OBSP % Error
2 6.8967 6.8001 1.40
4 3.7712 3.6838 2.32
8 2.2552 2.2118 1.92
16 1.5385 1.5226 1.03
Acknowledgements We would like to thank to Centre de Computació i Communicacions Catalunya, Edinburgh Parallel Computing Centre, Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas, Universidad de La Laguna and Universidad Nacional de San Luis. This research has been partially supported by Comisión Interministerial de Ciencia y Tecnologaí under project TIC1999-0754-C03.
References 1.
2. 3.
4.
Bonorden O. Juurlink B. von Otte I. Rieping I. The Paderborn University BSP (PUB) Library - Design Implementation and Performance. Technical Report TR-RSFB-98-063. Heinz Nixdorf Institute. Paderborn University. 1998. Gonzalez J.A. , Leon C., Piccoli F., Printista M., Roda J.L., Rodriguez C., Sande F. Oblivious BSP. Internal Report. http://nereida.deioc.ull.es/html/obsp.ps.gz Hill J. McColl W. Stefanescu D. Goudreau M.. Lang K. Rao S. Suel T. Tsantilas T. Bisseling R. BSPLib: The BSP Programming Library. Parallel Computing. 24(14) pp. 1947-1980. 1988. Valiant L.G. A Bridging Model for Parallel Computation. Communications of the ACM. 33(8). pp. 103-111. 1990.
A Software Architecture for HPC Grid Applications Steven Newhouse, Anthony Mayer, and John Darlington Department of Computing, Imperial College of Science, Technology and Medicine, 180 Queen’s Gate, London, SW7 2AZ, UK sjn5,aem3,[email protected] http://hpc.doc.ic.ac.uk/environments/ Abstract. We introduce a component software architecture designed for demanding grid computing environments that allows the optimal performance of the assembled component applications to be achieved. Performance over the assembled component application is maintained through inter-component static and dynamic optimisation techniques. Having defined an application through both its component task and data flow graphs we are able to use the associated performance models to support application level scheduling. By building grid aware applications through reusable interchangeable software components with integrated performance models we enable the automatic and optimal partitioning of an application across distributed computational resources.
1
Introduction
The emergence of local, national and international high-bandwidth networking allows physcially distributed computing hardware resources to be linked enabling the development of ‘computational grids’. These emerging grids present new challenges in automatic scheduling, application partitioning and resource management to deliver an effective computing environment. Effectively exploiting these dynamic and heterogeneous networking and storage resources requires automatic data partitioning to enable automatic run-time scheduling. Such scheduling only becomes feasible once performance information is integrated into the application. We introduce a scientific component framework for HPC applications which provides relevant abstractions to the end-user, scientist and numerical programmer. Application performance is maintained through static and dynamic component optimisations which allow component implementations to be matched to the execution architecture. By adopting object and component based programming techniques we can deliver the functionality of skeletons through a conventional abstraction mechanism. We present a simple example to illustrate these concepts and refer the reader to our complete paper (http://www-icpc.doc.ic.ac.uk/components/papers). A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 686–689, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Software Architecture for HPC Grid Applications
687
Component
Boundary Condition
Heat Flow
Region
Heat Flow FD
FD Grid Abstract Component Sequential
Parallel
Dirchlet
Neumann
Concrete Component
Diffusion BC
Heat Source BC
Fig. 1. Component Repository for a simple Finite Difference Problem showing abstract (white boxes) and concrete (grey boxes) components.
2
An Example: Heat Flow in an Insulated Bar
We will illustrate this software architecture with a simple scientific example of heat flow in an insulated beam. A simplified component repository is illustrated in Figure 1 to demonstrate the implementation hierarchy and the use of abstract and concrete components. 1. Problem Definition. The problem is defined within a graphical PSE linked to a component repository (Figure 2). A component will have context dependent representations. It may be a code segment in an execution context, have a visual representation when being visually composed to build an application or have a three dimensional representation within the PSE. In the example, the Region component is selected from the repository, placed within the graphical PSE and manipulated to define the physical domain of the bar. Properties such as the boundary and initial conditions are attached to the edges and domain to characterise the behaviour of the Region. 2. The Component Network. The problem defined within the graphical PSE can also be expressed as a component network (Figure 3). This allows further non-graphical components to be added into the component assembly to complete the application which will be used to solve the problem. The parameters within the components can be adjusted to further define the physical problem. 3. Valid Implementation Options. The component network which has been defined by the user is validated for its correctness. In the heat flow example
688
Steven Newhouse, Anthony Mayer, and John Darlington
HEAT FLOW
DIFFUSION BC
REGION
HEAT SOURCE BC
Fig. 2. A graphical Problem Solving Environment being used to physically define the analysis by manipulating graphical components.
a boundary condition must be applied to each edge of the physical Region and one or more properties may be applied to the whole Region to give it some physical characteristics (eg. heat flow, elastic material, etc). These conditions are essential pre-requisites for any valid component network and can be incorporated at a low-level into the Region component. The validated component network is compared to the available components in the repository. The static optimisation process described earlier is used to examine all of the feasible implementation options. In this simple example a choice has to be made between solving the problem using a sequential or parallel approach. 4. Scheduling. The valid implementation options are passed to the scheduler to match the application to a computational resource. This process uses the performance models derived from the assembled application and the performance of the computational resources to determine, within the constraints specified by both the user and resource provider, the resources needed to execute the application. In this example, the application’s overall performance model, the target architectures and the problem size will show when it is appropriate to move from a serial to a parallel implementation. Further examination of the parallel performance model will yield the ideal domain decomposition for this problem size. The effectiveness of cross platform scheduling can also be assessed as the component assembly will provide a profile of the computational requirements over time. If a matrix is being generated and then solved, it may be quicker to generate the matrix on a scalar parallel machine and transfer the data
A Software Architecture for HPC Grid Applications
689
to a vector machine for the solution phase rather than execute both tasks on the vector machine. The data flow and execution tasks graphs within the application, developed during the component composition process allow computing and networking performance models to be used to assess these alternative implementation options. 5. Execution. The component assembly is finally passed to the resources for execution. The execution and data flow graphs can still be exploited to examine the benefits of migrating the application to faster resources if they become available or the benefits of adding additional resources to the current calculation.
3
Conclusions
The proposed architecture, which we are developing and implementing across several applied scientific domains, is an extension of current or established research in compositional programming techniques. The skeletons work has already demonstrated the validity of using performance models to guide implementation and data layout decisions. We are able to exploit standard component models, such as Java Beans and CORBA, to contain the skeleton code fragments. The repository will contain a variety of software components presenting usable abstractions to the end-user, the scientist and the numerical programmer. By defining standard interfaces, software will become easier to reuse between applications simplifying the development of large multi-disciplinary projects. By breaking the link between the problem definition and execution we are able to find the optimal implementation on the currently available computational resources. Having built an application from components with integral performance models we are able to make sophisticated scheduling decisions and match an application to the appropriate computational resources. This will become essential for the emerging heterogeneous computational grids that will dominate high performance computing in the future.
Diffusion BC
Heat Source BC Region
Diffusion BC
Diffusion BC
Heat Flow
Fig. 3. The Component Network extracted from the graphical Problem Solving Environment.
Satin: Efficient Parallel Divide-and-Conquer in Java Rob V. van Nieuwpoort, Thilo Kielmann, and Henri E. Bal Dept. of Mathematics and Computer Science, Vrije Universiteit, Amsterdam, The Netherlands [email protected], [email protected], [email protected] http://www.cs.vu.nl/albatross/
Abstract. Satin is a system for running divide and conquer programs on distributed memory systems (and ultimately on wide-area metacomputing systems). Satin extends Java with three simple Cilk-like primitives for divide and conquer programming. The Satin compiler and runtime system cooperate to implement these primitives efficiently on a distributed system, using work stealing to distribute the jobs. Satin optimizes the overhead of local jobs using on-demand serialization, which avoids copying and serialization of parameters for jobs that are not stolen. This optimization is implemented using explicit invocation records. We have implemented Satin by extending the Manta compiler. We discuss the performance of ten applications on a Myrinet-based cluster.
1
Introduction
There is currently much interest in divide and conquer systems for parallel programming [2, 6, 10, 11, 15]. Divide and conquer style programs start by dividing the problem into subproblems. Each subproblem is then recursively solved, again by dividing it into smaller subproblems. An example of such a system is Cilk [6], which extends C with divide and conquer primitives. Cilk runs these annotated C programs in parallel, in an efficient way, but is mainly targeted at shared memory machines. Atlas [2], an extension of Java, is a divide and conquer system designed for distributed memory machines. Its primitives have a high overhead, however, so it runs fine-grained parallel programs inefficiently. In this paper, we introduce a new system, called Satin, which also is a divide and conquer system based on Java. Satin (as the name suggests) was inspired by Cilk. In Satin, single-threaded Java programs are parallelized by annotating methods that can run in parallel. Our ultimate goal is to use Satin for distributed supercomputing applications on hierarchical wide-area clusters (e.g., the DAS [8]). We think that the divide and conquer model will map efficiently on such systems, as the model is also hierarchical. In this paper, however, we focus on the implementation of Satin on a single local cluster computer. In contrast to Atlas, Satin is designed as a compiler-based system in order to achieve high performance. Satin is based on the Manta [12] native compiler, which supports highly efficient serialization and communication. Parallelism is achieved in A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 690–699, 2000. c Springer-Verlag Berlin Heidelberg 2000
Satin: Efficient Parallel Divide-and-Conquer in Java
691
Satin by running different spawned method invocations on different machines. The system load is balanced by work stealing. One of the contributions we make in the paper is the use of explicit invocation records, to enable the on-demand serialization of parameters to spawned method invocations. This optimization is possible because of Satin’s parameter semantics. Furthermore, we demonstrate that Satin can run efficiently on distributed memory machines. Satin also cleanly integrates divide and conquer programming into Java, and solves some problems that are introduced by this integration (e.g., by garbage collection).
2
The Programming Model
Satin’s programming model is an extension of the single-threaded Java model. Satin programmers thus need not use Java’s multithreading and synchronization constructs or Java’s Remote Method Invocation mechanism, but can use the much simpler divide and conquer primitives described below. 2.1
Spawn and Sync
We have introduced three new keywords to the Java language, spawn, sync, and satin. The spawn keyword must be placed in front of a method invocation, which will then be called a spawned method invocation. When spawn is placed in front of a method invocation, conceptually a new thread is started which will run the method. (The implementation of Satin, however, eliminates thread creation altogether.) The spawned method will run concurrently with the method that executed the spawn. In Satin, spawned methods always run to completion. The sync operation waits until all spawned calls in this method invocation are finished. The return values of spawned method invocations are undefined until a sync is reached. The satin modifier must be placed in front of a method declaration, if this method is ever to be spawned. To illustrate the use of spawn and sync, an example program is shown in Fig. 1. This code fragment calculates Fibonacci numbers, and is a typical example of a divide and conquer program. Note that this is a benchmark, and not a suitable algorithm for efficiently calculating the Fibonacci numbers. The program is parallelized just by inserting spawn in front of the recursive calls to fib. The two subproblems will now be solved concurrently. Before the results are combined, the method must wait until both subproblems have actually been solved, and have returned their value. This is done by the sync operation. A well known optimization in parallel divide and conquer programs is to make use of a threshold on the number of spawns. When this threshold is reached, work is executed sequentially. This approach can easily be programmed using Satin. Satin does not provide shared memory, because this is hard to implement efficiently on distributed memory machines. Moreover, our ultimate goal is to run Satin on wide-area systems, which clearly do not have shared memory. The only way of communicating between threads is via the parameters and the return
692
Rob V. van Nieuwpoort, Thilo Kielmann, and Henri E. Bal
class Fibonacci { SATIN int fib(int n) { if(n < 2) return n; int x = SPAWN fib(n - 1); int y = SPAWN fib(n - 2); SYNC; return x + y; } public static void main(String[] args) { Fibonacci f = new Fibonacci(); int result = f.fib(10); System.out.println("Fib 10 = " + result); } }
Fig. 1. A Satin example: Fibonacci.
value. The parameter passing mechanism, as described in Sect. 2.2, assures that all data that can be accessed via parameters will be sent to the machine that executes the spawned method invocation. 2.2
The Parameter Passing Mechanism
Because Satin does not provide shared memory, objects passed as parameters in a spawned call to a remote machine will not be available on that machine. Therefore, Satin uses call-by-value semantics when the runtime system decides that the method will be spawned remotely. This is semantically similar to the standard Java Remote Method Invocation (RMI) mechanism [17]. Call-by-value is implemented using Java’s serialization mechanism, which provides a deep copy of the serialized objects [16]. For instance, when the first node of a linked list is passed as an argument to a spawned method invocation (or a RMI), the entire list is copied. It is important to minimize the overhead for work that does not get stolen and is executed by the machine that spawned the work, as this is the common case. For example, in almost all applications we have studied so far, at most 1 out of 400 jobs gets stolen. Because copying all parameter objects (i.e., using call-by-value) in the local case would be prohibitively expensive, parameters are passed by reference when the method invocation is local. Therefore, the programmer cannot assume either call-by-value or call-by-reference semantics for satin methods (normal methods are unaffected and have the standard Java semantics). It is therefore erroneous to write Satin methods that depend on the parameter passing mechanism. (A similar approach is taken in Ada for parameters of a structured type.) An important characteristic of Satin is that when the extensions satin, spawn, and sync are removed, a sequential standard Java program remains.
Satin: Efficient Parallel Divide-and-Conquer in Java
693
This program produces the same result as the parallel Satin program. This always holds, because Satin does not specify the parameter passing mechanism. Using call-by-reference in all cases (as normal Java does) is thus correct.
3
The Implementation
The large majority of jobs will not be stolen, but will just run on the machine the jobs were spawned on. Therefore, it is important to reduce the overhead that the Satin runtime system generates for such jobs as much as possible. The key problem here is that the decision whether to copy the parameters must be made at the moment the work is executed or stolen, not when the work is generated. To be able to defer this important decision, Satin’s runtime system uses invocation records, which will be described below. The large overhead for creating threads or building task descriptors (copying parameters) was also recognized in the lazy task creation work by Mohr et al. [13]. When a program executes a spawn, Satin redirects the method call to a stub. This stub creates an invocation record (see Fig. 2), describing the method to be invoked, the parameters that are passed to the method, and a reference to where the method’s return value has to be stored. For primitive types, the value of the parameter is copied. For reference types (objects, arrays, interfaces), only a reference is stored in the record. In the example of Fig. 2, a satin method is invoked with an integer, an array, and an object as parameters. The integer is stored directly in the invocation record, but for the array and the object, references are stored, to avoid copying these data structures. The compiler allocates space for a counter on the stack of all methods executing spawn operations. This counter is called the spawn counter, and counts the number of pending spawns, which have to be finished before this method can return. The address of the spawn counter is also stored in the invocation record.
foo &result &spawn_counter
i
d
o
foo &result &spawn_counter
i
d
o
d o d foo &result &spawn_counter
i
d
o
SATIN int foo(int i, double[] d, Object o);
d
int result = SPAWN foo(i, d, o);
Fig. 2. Invocation records in the job queue.
The stub that builds an invocation record for a spawned method invocation is generated by the Manta compiler, and is therefore very efficient, as no runtime type inspection is required. From an invocation record, the original call can
694
Rob V. van Nieuwpoort, Thilo Kielmann, and Henri E. Bal
be executed by pushing the value of the parameters (which were stored in the record) onto the stack, and by calling the Java method. The invocation record for a spawn operation is stored in a queue. The spawn counter (located on the stack of the invoking method) is incremented by one, indicating that the invoking method now has a pending spawned method invocation. The invoking method may then continue running. After the spawned method invocation has eventually been executed, its return value will be stored at the return address specified in the invocation record. Next, the spawn counter (the address of which is also stored in the invocation record) will be decremented by one, indicating that there now is one less pending spawn. The sync operation executes work stored in the job queue, and waits for the spawn counter to become zero. When this happens, there are no more pending spawned method invocations, so the method may continue. Serialization is Java’s mechanism to convert objects into a stream of bytes. This mechanism always makes a deep copy of the serialized objects: all references in the serialized object are traversed, and the objects they point to are also serialized. The serialization mechanism is used in Satin for marshaling the parameters to a spawned method invocation. Satin implements serialization on demand: the parameters are serialized only when the work is actually stolen. In the local case, no serialization is used, which is of critical importance for the overall performance. In the Manta system, the compiler generates highly-efficient serialization code. For each class in the system a so-called serializer is generated, which writes the data fields of an object of this class to a stream. When an object has reference fields, the serializers for the referenced objects will also be called. Furthermore, Manta uses an optimized protocol to represent the serialized objects in the byte stream. Manta’s implementation of the serialization mechanism is described in more detail in [12]. The invocation records describing the spawned method invocations are stored in a double ended job queue. A Dijkstra-like protocol [6] is used to avoid locking in the local case. Satin registers the invocation records at the garbage collector, keeping parameter objects alive when they are referenced only via the invocation record, and not via a Java reference. (Otherwise, the garbage collector might free objects that are needed to execute the spawn operations, but are no longer referenced via the Java program). Satin’s work stealing is implemented on top of the Panda communication library [1], primarily using Panda’s message passing primitives. On the Myrinet network (which we use for our measurements), Panda is implemented on top of the LFC [3] network interface protocol. Satin uses the efficient, user level locks that Panda provides for protecting the work queue.
4
Performance Evaluation
We evaluated Satin’s performance using ten application kernels. All measurements were performed on a cluster of the Distributed ASCI Supercomputer (DAS), each containing 200 MHz Pentium Pros that are locally connected by Myrinet. The machines run the Linux (RedHat 6.2) operating system.
Satin: Efficient Parallel Divide-and-Conquer in Java
4.1
695
Basic Spawn Overhead (Fibonacci)
An important indication of the performance of a divide and conquer system is the overhead of the parallel application on one machine, compared to the sequential version of the same application. The sequential version is obtained by filtering the keywords satin, spawn, and sync out of the parallel program. The difference in run times between the sequential and parallel programs is caused by the creation, the en-queuing and de-queuing of the invocation record, and the construction of the stack frame to call the Java method. Fibonacci gives an indication of the worst-case overhead, because it is very fine grained. Cilk is very efficient, the parallel Fibonacci program on one machine has an overhead of only a factor of 3.6 (measured on a Sun Enterprise 5000, with 167 MHz UltraSPARC processors) [6]. Atlas is implemented completely in Java and does not use ondemand serialization. Therefore its overhead is much worse, a factor of 61.5 (hardware unknown) [2]. The overhead of Satin is a factor 7.25, substantially lower than that of Atlas. These overhead factors can be reduced at the application level by introducing threshold values that spawn only large jobs. For Fibonacci, for example, we tried a threshold value of 20 for a problem of size 45, so all calls to fib(n) with n<20 are executed sequentially, without using spawn. This simple change to the application reduced the overhead to almost zero. Still, 22.8 · 106 jobs were spawned, leaving enough parallelism for running the program on large numbers of machines. For Fibonacci, the threshold can easily be determined by the programmer, while for other applications this may be difficult or impossible. In general, however, it still is important to keep the sequential overhead of a divide and conquer system as small as possible, as it allows the creation of more fine-grained jobs and thus a better load balancing. The overhead for the other applications we implemented is much lower than for the (original) Fibonacci program, as shown in Table 1. Here, ts denotes the run time of the sequential program, t1 the run time of the parallel program on one machine. In general, the overhead depends on the number of parameters to spawned methods. All parameters have to be stored in the invocation record when the work is spawned, and pushed on the stack again, when executed. 4.2
Parallel Applications
We ran ten applications on the DAS cluster, using up to 32 CPUs. Figure 3 shows the achieved speedups while Table 2 provides detailed information about the parallel runs. All speedup values were computed relative to the sequential applications, with the Satin-specific annotations removed from the code. There is a strong correlation between measured speedup and the sequential overhead value, as already shown in Table 1: the lower the overhead, the higher the speedup we achieved. In Table 2 we compare the measured speedup with its upper bound, computed as the number of CPUs divided by the overhead on a single CPU. We also show the percentage of this upper bound as actually achieved by the measured speedup. This percentage is very high for most applications,
Rob V. van Nieuwpoort, Thilo Kielmann, and Henri E. Bal
Table 1. Application overhead factors, times in seconds. application problem size ts t1 overhead adaptive integration 0, 2.0E5, 1.0E-4 363.137 451.117 1.24 58, 29 1983.723 2071.333 1.04 set covering problem 41 65.517 475.133 7.25 fibonacci 45 473.749 473.834 1.00 fibonacci threshold 60 220.131 250.001 1.14 Iterative deepening A* 28 1064.220 1150.016 1.08 knapsack problem 1024 x 1024 137.982 141.742 1.03 matrix multiplication 34, 17 971.991 977.847 1.01 n over k 15 1861.318 1909.942 1.03 n-queens 1234567890 874.504 930.954 1.06 prime factorization 17 982.864 1352.617 1.38 traveling sales person
linear fib. threshold n over k n queens prime factors integrate IDA* TSP knapsack mmult set cover fibonacci
30 25 20 speedup
696
15 10 5 0
0
5
10
15 20 number of processors
25
30
Fig. 3. Application speedups Table 2. Parallel performance breakdown for 32 CPUs #CPUs/ % max application overhead speedup overhead speedup jobs integrate 1.24 26.09 25.81 101 % 63.3 · 106 1.04 6.85 30.8 22.2 % 51.0 · 106 set cover 7.25 4.31 4.4 98.0 % 536 · 106 fibonacci 1.00 31.77 32.0 99.3 % 22.8 · 106 fib. threshold 1.14 25.82 28.1 91.9 % 33.6 · 106 IDA* 1.08 12.36 29.6 41.8 % 33.5 · 106 knapsack 1.03 9.49 31.1 30.5 % 37.4 · 103 mmult 1.01 31.27 31.7 98.6 % 1.05 · 106 n over k 1.03 31.02 31.1 99.7 % 2.47 · 106 n-queens 1.06 28.98 30.2 96.0 % 33.6 · 106 prime factors 1.38 22.79 23.2 98.2 % 200 · 106 tsp
stolen 2187 579 2906 1951 3866 417 8567 2458 3027 2609 3026
Satin: Efficient Parallel Divide-and-Conquer in Java
697
denoting that Satin’s communication costs are low. The actual percentage depends (like the sequential overhead) on the number of method parameters and their total serialized size. Table 2 also lists the total number of spawned jobs and the number of stolen jobs, which is less than 1 out of 400 for all applications, except for mmult. Because the number of stolen jobs is so small, speedups are mainly determined by sequential overhead. A good example is Fibonacci, which achieves 98% of the upper bound, but still has a low speedup due to the sequential overhead. Satin’s sequential efficiency thus is important for the successful deployment of the divide and conquer paradigm for parallel computing. Mmult does not get good speedups, because the problem size is small due to memory constraints, the run time on 32 cpus is only 14 seconds. Also, much data is transferred, in total over all CPUs, 31 MByte is sent per second. The mediocre speedup of knapsack, a very irregular application, is caused by load imbalance. The search space is pruned by both the weights and the values of the elements in the knapsack, making it difficult to estimate the grain size of a job. Therefore, many small jobs get stolen. The same holds for the set-covering problem, where a large percentage of the time is spent in finding work. On 32 nodes, only 1.2 percent of the work stealing attempts were successful.
5
Related Work
We discussed Satin, a divide and conquer extension of Java. Satin has been designed for distributed memory machines, while most divide and conquer systems use shared memory machines (e.g. Cilk [6]). There is also a version of Cilk for distributed memory machines, called CilkNOW [5], but it only supports functional Cilk programs (without shared memory), and it does not make a deep copy of the parameters to spawned methods. Our own previous work on parallel divide and conquer [9] was based on the C language while having similar restrictions as CilkNOW. Alice [7] and Flagship [18] offer a hardware solution for parallel divide and conquer programs (e.g., a reduction machine with one global address space for the parallel evaluation of declarative languages), while Satin is purely software based, and does not require, or provide, a single address space. Mohr et al. [13] describe the importance of avoiding thread creation in the common, local case (lazy task creation). Satin also avoids creating threads in the local case, targeting distributed memory adds the problem of copying the parameters of parallel invocations (marshalling). Satin builds on the ideas of lazy task creation, and avoids both the starting of threads and the copying of parameter data by choosing a suitable parameter passing mechanism. Another divide and conquer system based on Java is Atlas [2]. Atlas is not a Java extension, but a set of Java classes that can be used to write divide and conquer programs. While Satin is targeted at efficiency, Atlas was designed with heterogeneity and fault tolerance in mind, and aims only at a reasonable performance. Because Satin is compiler based, it is possible to generate code to create the invocation records, thus avoiding all runtime type inspection. The
698
Rob V. van Nieuwpoort, Thilo Kielmann, and Henri E. Bal
Java classes presented in [11] can also be used for divide and conquer algorithms. However, they are restricted to shared-memory systems. A compiler-based approach is also taken by Javar [4]. In this system, the programmer uses annotations to indicate divide and conquer and other forms of parallelism. The compiler then generates multi-threaded Java code, which runs on any JVM. Therefore, Javar programs run only on shared memory machines and DSM systems, whereas Satin programs run on distributed memory systems. Java threads impose a large overhead, which is why Satin does not use threads at all, but provides light weight invocation records. There are many other projects which use Java for parallel processing, for instance [14] and the work referenced in this paper.
6
Conclusions and Future Work
We have described our experiences in building a parallel divide and conquer system for Java, which runs on distributed memory machines. We have shown that an efficient implementation is possible by choosing convenient parameter semantics. An important optimization is the on-demand serialization of parameters to spawned method invocations. This was implemented using explicit invocation records. Our Java compiler generates code to create these invocation records for each spawned method invocation. We have also demonstrated that divide and conquer programming can be cleanly integrated into Java, and that problems introduced by this integration (e.g., through garbage collection) can be solved. Our ultimate goal is to use Satin for distributed supercomputing applications on hierarchical wide-area clusters. We believe that divide and conquer programs will map efficiently on such systems, as the model is also hierarchical. Our intention is to carry out research on the scheduling of divide and conquer programs on hierarchical wide-area systems.
Acknowledgments This work is supported in part by a USF grant from the Vrije Universiteit. The widearea DAS system is an initiative of the Advanced School for Computing and Imaging (ASCI). We thank Aske Plaat for his contribution to this research, and Ronald Veldema, Jason Maassen, Ceriel Jacobs, and Rutger Hofman for their work on the Manta system. We thank Kees Verstoep and John Romein for keeping the DAS in good shape. We also thank the anonymous referees for their useful comments on this paper.
References [1] H. E. Bal, R. Bhoedjang, R. Hofman, C. Jacobs, K. Langendoen, T. R¨ uhl, and F. Kaashoek. Performance Evaluation of the Orca Shared Object System. ACM Transactions on Computer Systems, 16(1):1–40, Feb. 1998.
Satin: Efficient Parallel Divide-and-Conquer in Java
699
[2] J. Baldeschwieler, R. Blumofe, and E. Brewer. ATLAS: An Infrastructure for Global Computing. In Proceedings of the Seventh ACM SIGOPS European Workshop on System Support for Worldwide Applications, 1996. [3] R. A. F. Bhoedjang, T. R¨ uhl, and H. E. Bal. User-Level Network Interface Protocols. IEEE Computer, 31(11):53–60, Nov. 1998. [4] A. Bik, J. Villacis, and D. Gannon. javar: A prototype Java restructuring compiler. Concurrency: Practice and Experience, 9(11):1181–1191, November 1997. [5] R. Blumofe and P. Lisiecki. Adaptive and reliable parallel computing on networks of workstations. In In Proceedings of the USENIX 1997 Annual Technical Conference on UNIX and Advanced Computing Systems, Anaheim, California, 1997. [6] R. D. Blumofe, C. F. Joerg, B. C. Kuszmaul, C. E. Leiserson, K. H. Randall, and Y. Zhou. Cilk: An efficient multithreaded runtime system. In Proceedings of the 5th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP’95, pages 207–216, Santa Barbara, California, July 1995. [7] J. Darlington. Alice: a multi-processor reduction machine for the parallel evaluation of applicative languages. In Arvind, editor, 1st Conference on Functional Programming Languages and Computer Architecture, pages 65–76, Wentworth-bythe-Sea, Portsmouth, New Hampshire, 1981. [8] The Distributed ASCI Supercomputer (DAS). http://www.cs.vu.nl/das/. [9] B. Freisleben and T. Kielmann. Automated Transformation of Sequential Divide– and–Conquer Algorithms into Parallel Programs. Computers and Artificial Intelligence, 14(6):579–596, 1995. [10] K. S. Gatlin and L. Carter. Architecture-cognizant divide and conquer algorithms. In SuperComputing ’99, November 1999. [11] D. Lea. A java fork/join framework. In ACM Java Grande 2000 Conference, San Francisco, California, June 2000. [12] J. Maassen, R. van Nieuwpoort, R. Veldema, H. Bal, and A. Plaat. An Efficient Implementation of Java’s Remote Method Invocation. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 173–182, Atlanta, GA, May 1999. [13] E. Mohr, D. Kranz, and R. Halstead. Lazy task creation: a technique for increasing the granularity of parallel programs. In Proceedings of the 1990 ACM Conference on Lisp and Functional Programming, pages 185–197, June 1990. [14] M. Philippsen and M. Zenger. JavaParty—Transparent Remote Objects in Java. Concurrency: Practice and Experience, pages 1225–1242, Nov. 1997. [15] R. Rugina and M. Rinard. Automatic parallelization of divide and conquer algorithms. In Seventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 72–83, Atlanta, May 4-6 1999. Massachusetts Institute of Technology. [16] Sun MicroSystems, Inc. Java (TM) Object Serialization Specification, 1996. ftp://ftp.javasoft.com/docs/jdk1.1/serial-spec.ps. [17] J. Waldo. Remote procedure calls and Java Remote Method Invocation. IEEE Concurrency, pages 5–7, July–September 1998. [18] I. Watson, V. Woods, P. Watson, R. Banach, M. Greenberg, and J. Sargeant. Flagship: A parallel architecture for declarative programming. In 15th IEEE/ACM Symp. on Computer Architecture, pages 124–130, Honolulu, Hawaii, 1988. ACM SIGARCH newsletter,16(2).
Implementing Declarative Concurrency in Java Rafael Ramirez, Andrew E. Santosa, and Lee Wei Hong National University of Singapore School of Computing S16, 3 Science Drive 2, Singapore 117543 Tel. +65 8742909, Fax +65 7794580 {rafael,andrews,leeweiho}@comp.nus.edu.sg
Abstract. We describe the implementation of a high-level language based on first order logic for expressing synchronization in multi-threaded Java programs. The language allows the programmer to declaratively state the system safety properties as temporal constraints on specific program points of interest (events). The constraints are enforced by the runtime environment, i.e. the program points of interest are traversed only in the order specified by the constraints. The implementation is based on the incremental and lazy generation of partial orders among events. Although the implementation reported in this paper is concerned only with the synchronization of Java programs, the general underlying synchronization model we present is language independent in that it allows the programmer to glue together separate concurrent threads regardless of their implementation language and application code.
1
Introduction
The task of programming concurrent systems is substantially more difficult than the task of programming sequential systems in respect of both correctness (to achieve correct synchronization) and efficiency. One of the reasons is that it is very difficult to separate the concurrency issues in a program from the rest of the code. Synchronization concerns cannot be neatly encapsulated into a single unit which results in their implementation being scattered throughout the source code. This harms the readability of programs and severely complicates the development and maintenance of concurrent systems. Furthermore, the fact that program synchronization concerns are intertwined with the rest of the code, also complicates the formal treatment of the concurrency issues of the program which directly affects the possibility of formal verification, synthesis and transformation of concurrent programs. In Java, for instance, a common problem of writing multi-threaded applications is that synchronization code ensuring data integrity tends to dominate the source code completely. This produces code which is difficult to understand, modify and treat formally [11]. We believe that the system concurrency issues are best treated as orthogonal to the system base functionality. In this paper we describe the implementation of a first order language in which the safety properties of concurrent Java programs A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 700–708, 2000. c Springer-Verlag Berlin Heidelberg 2000
Implementing Declarative Concurrency in Java
701
are declaratively stated as constraints. Basic Java programs are annotated at points of interest so that the run-time environment enforces specific temporal constraints between the visit times of these points. The declarative nature of the constraints provides great advantages in writing, reasoning and deriving concurrent programs [4,5]. The constraints are language independent in that the application program can be specified in any conventional programming language. The model has a procedural interpretation which is based on the incremental and lazy generation of constraints, i.e. constraints are considered only when needed to reason about the execution order of current events. Section 2 describes some related work. Section 3 presents the language used for specifying the synchronization constraints of concurrent Java programs. In Section 4 we describe the implementation of the language and finally Section 5 summarizes the contributions and indicates some areas of future research.
2
Related Work
Various attempts have been done by the object-oriented community to separate concurrency issues from functionality. Recently, some researchers have proposed aspect-oriented programming (AOP ) [6]. In this area, the closest related work is the work by De Volder and D’Hondt [14]. Their proposal utilizes a fullfledged logic programming language as the aspect language. In order to specify concurrency issues in the aspect language, basic synchronization declarations are provided which increase program readability. Unfortunately, the declarations have no formal foundation. This reduces considerably the declarativeness of the approach since correctness of the program concurrency issues directly depend on the implementation of the the declarations. Closer to our work are path expressions (e.g., PROCOL [2]) and constructs similar to synchronization counters (e.g., Guide [8] and DRAGOON [1]). These proposals, as ours, differ from the AOP approach in that the specification of the system concurrent issues are part of the final program. Unfortunately, synchronization counters have limited expressiveness since it is not possible to order methods explicitly. Path expressions are more expressive in this respect but they cannot express some important synchronization (e.g., producers-consumers) without embedding guards that increases complexity. An important issue is that in all of the other proposals mentioned above, method execution is the smallest unit of concurrency. This is impractical in actual concurrent programming where we often need finer-grained concurrency. On the other hand, several approaches to incorporate declarative programming into concurrent programming have been proposed. Traditional approaches to declarative concurrent programming include concurrent logic programming (e.g. Parlog [3], KL1 [13]) and concurrent constraint programming [12]. Although these approaches preserve many of the benefits of the abstract declarative model, such as the logical reading of programs and the use of logical terms to represent data structures, important program properties, namely safety and progress properties, remain implicit. These properties have to be preserved by using con-
702
Rafael Ramirez, Andrew E. Santosa, and Lee Wei Hong
trol features such as modes and sequencing, producing programs with little or no declarative reading. Also, in such languages, there is no clear separation of program application functionality and concurrency control.
3
Logic Programs for Concurrent Programming
3.1
Events and Constraints
Many researchers, e.g. [7,9], have proposed methods for reasoning about temporal phenomena using partially ordered sets of events. Our approach to concurrent programming is based on the same general idea. The basic idea here is to use a constraint logic program to represent the (usually infinite) set of constraints of interest. The constraints themselves are of the form X < Y , read as “X precedes Y ” or “the execution time of X is less than the execution time of Y ”, where X and Y are events, and < is a partial order. The constraint logic program is defined as follows1 . Constants range over events classes E, F, . . . and there is a distinguished (postfixed) functor +. Thus the terms of interest, apart from variables, are e, e+, e + +, . . . , f, f +, f + +, . . .. The idea is that e represents the first event in the class E, e+ the next event, etc. Thus, for any event X, X+ is implicitly preceded by X, i.e. X < X+. We denote by e(+N ) the N -th event in the class E. Programs facts or predicate constraints are of the form p(t1 , . . . , tn ) where p is a user defined predicate symbol and the ti are ground terms. Program rules or predicate definitions are of the form p(X1 , . . . , Xn ) ← B where the Xi are distinct variables and B a rule body whose variables are in {X1 , . . . , Xn }. A program is a finite collection of rules and is used to define a family of partial orders over events. Intuitively, this family is obtained by unfolding the rules with facts indefinitely, and collecting the (ground) precedence constraints of the form e < f . Multiple rules for a given predicate symbol give rise to different partial orders. For example, since the following program has only one rule for p: p(e, f ). p(E, F ) ← E < F , p(E+, F +). it defines just one partial order e < f , e+ < f +, e + + < f + +, . . .. In contrast, p(e, f ). p(E, F ) ← E < F , p(E+, F +). p(E, F ) ← F < E, p(E+, F +). defines a family of partial orders over {e, f, e+, f +, e + +, f + +, e + + + . . .}. We will abbreviate the set of clauses H ← Cs1 , . . ., H ← Csn by the disjunction constraint H ← Cs1 ; . . . ; Csn (disjunction is specified by the disjunction operator ’;’). 1
For a complete description, see [10]
Implementing Declarative Concurrency in Java
3.2
703
Markers and Events
In order to refer to the visit times at points of interest in the program we introduce markers. A marker declaration consists of an event name enclosed by angle brackets, e.g. <e>. Markers annotations can be seen simply as program comments (i.e. they can be ignored) if only the functional semantics of an application is considered. Markers are associated with programs points between instructions, possibly in different threads. Constraints may be specified between program points delineated by these markers. For a marker M , time(M ) (read as “the visit time at M ”) denotes the time at which the instruction immediately preceding M has just been completed. In the following, we will refer to time(M ) simply by M whenever confusion is unlikely. Given a pair of markers, constraints can be stated to specify their relative order of execution in all executions of the program. If the execution of a thread T1 reaches a program point whose execution time is constrained to be greater than the execution time of a not yet executed program point in a different thread T2 , thread T1 is forced to suspend execution. In the presence of loops and procedure calls a marker is typically visited several times during program execution. Thus, in general, a marker M associated with a program point p represents an event class E where each of its instances e, e+, e + + . . . corresponds to a visit to p during program execution (e represents the first visit, e+ the second, etc.). 3.3
Example
An example discussed in almost every textbook on concurrent programming is the producer and consumer problem. The problem considers two types of processes: producers and consumers. Producers create data items (one at a time) which then must be appended to a buffer. Consumers remove items from the buffer (if it is not empty) and consume them, i.e. perform some computation which uses the data item. Thus, producer processes can be defined by an infinite cycle containing produce (producing an item) and append (appending the item to the buffer). Similarly, consumer processes can be defined by an infinite cycle containing remove (removing an item from the buffer) and consume (consuming the item). Producers and consumers may be defined as follows (the code has been annotated with markers p1, p2, c1 and c2). class Producer extends Runnable { ... public void run() { while (true) { produce (X); < p1 > append (X); < p2 > }} ... }
class Consumer extends Runnable { ... public void run() { while (true) { < c1 > remove(X); < c2 > consume(X); }} ... }
704
Rafael Ramirez, Andrew E. Santosa, and Lee Wei Hong
If we assume an infinite buffer, the only safety property needed is that the consumer never attempts to remove an item from an empty buffer. This property can be expressed by p(p2, c1). p(P, C) ← P < C, p(P +, C+). In practice however, buffers are finite. Thus, in practice, a producer is allowed to append items to the buffer only when the buffer is not full. For instance, this safety property for a system with a buffer of size 3 can be expressed by p(c2, p1 + ++). p(C, P ) ← C < P , p(C+, P +).
4
Implementation
A prototype implementation of the ideas presented here has been written using the language Java. Java was used both to implement the constraint language and to write the code of a number of applications. We discuss the implementation in this section. 4.1
Architecture
The architecture of our implementation (Figure 1) consists of four main parts: – The interface is an object of class Tempo which decides whether or not threads suspend upon reaching a marker during execution. When a thread reaches a marker m, a request is sent to the interface to determine whether the current event e associated with m is disabled, i.e. it appears on the right of a precedence constraint X < e, or enabled, i.e. otherwise, wrt the constraint store. If e is found to be disabled, the thread is blocked until e becomes enabled, otherwise the thread proceeds execution at the instruction immediately after m. – The constraint store contains the system synchronization constraints. As shown in Figure 1, it can be decomposed into two parts: the predicate definition store (DS) and the constraint store (CS). The constraint store contains precedence, predicate and disjunction constraints. – The user program is the main program and typicly specifies the system synchronization constraints, creates the Tempo object and spawn a number of threads which may contain markers. – The verifier examines the specification of the system synchronization constraints to detect any errors such as infinite loops in predicate definitions, e.g. p(X) ← p(X), before the Tempo object is created. The overall mechanism is as follows: Once the Tempo object has been created and a set of threads have been spawned, whenever one of the threads, say T ,
Implementing Declarative Concurrency in Java
Predicate Definition Store (DS)
Precedence Store
705
Disjunction Store
Predicate Store
constraint checking
User Program
Verifier
Interface (Tempo)
check("c")
check("a") Thread 1
Thread 2
Thread n check("b")
spawns
.............
Fig. 1. Implementation architecture reaches a marker in their code, a communication between the thread and the constraint store is triggered. Currently the communication is implemented as a request from T to the Tempo object. The request is of the form check (Str ), in which Str is a string denoting marker’s identifier (e.g., “p1,” “p2,” “c1,” “c2” in the producers-consumers example). 4.2
The Constraint Interpreter
The decision on whether to suspend a thread or not is based on a procedural interpretation of the constraint logic programs used to specify synchronization constraints. The procedural interpretation allows a correct specification to be executed in the sense that events are only executed as permitted by the constraints represented by the program. This procedural interpretation is based on an incremental execution of the program and a lazy generation of the corresponding partial orders. Constraints are generated by the constraint logic program only when needed to reason about the execution times of current events. Figure 2 shows the complete interpreter for a constraint logic program specifying the synchronization constraints of a Java program. The algorithm is implemented in the class Tempo. To see how the interpreter works, let us use the producers-consumers example explained in Section 3.3. Initially, DS contains the predicate definition p(X, Y ) ← X < Y, p(X+, Y +), and CS contains two predicate constraints: p(p2, c1) and p(c2, p1 + ++). Suppose a consumer thread reaches marker c1. At this point, the constraint store is checked to determine whether c1 is enabled or disabled which requires the expansion of p(p2, c1). Based on the predicate definition in DS, p(p2, c1) is expanded to p2 < c1, p(p2+, c1+) at which point
706
Rafael Ramirez, Andrew E. Santosa, and Lee Wei Hong
place system constraints in CS; place predicate definitions in DS; when thread T reaches marker E: get current occurrence e of the event class associated with E; while e is neither enabled nor disabled in CS (e appears in user-defined constraint C): replace C in CS by its definition; if e is disabled in CS then suspend T if e is enabled in CS then execute e else if e is conditionally enabled in CS then reduce CS by e; execute e else if e is unknown in CS then error. execute e: for each precedence constraint in CS with e on the left e < R (including those inside disjunctions): delete e < R; resume threads waiting for R. reduce CS by e: for each disjunction D in CS: delete all alternatives of D in which e is disabled.
for a disjunction D: if e is enabled in every alternative of D then e is enabled in D else if e is disabled in every alternative of D then e is disabled in D else if e is enabled in some alternatives of D and disabled in all others then e is conditionally enabled in D else e is unknown in D. for a conjunction CS: if e is disabled in at least one constraint in CS then e is disabled in CS else if e is unknown in at least one constraint in CS then e is unknown in CS else if e is conditionally enabled in at least one constraint in CS then e is conditionally enabled in CS else e is enabled in CS.
Fig. 2. Synchronization constraints interpreter
c1 is found to be disabled since it appears on the right hand side of the precedence constraint p2 < c1. Thus, execution of the consumer thread is suspended at marker c1. At some point, a producer thread reaches marker p1 and after checking the constraint store it is determined that p1 is enabled (it does not appear on the right hand side of any precedence constraint). It then proceeds to add an item to the buffer, and then it reaches marker p2. Since p2 is enabled execution of the producer thread proceeds, constraint p2 < c1 is deleted from CS since it has already been satisfied, and threads suspended for event c1 are awaken so they can re-checks the constraint store. At this point, since c1 does not appear on the right side of any precedence constraints, the consumer thread may continue its execution. This has the effect of not allowing retrieval of items when the buffer is empty. A similar situation occurs when a producer thread attempts to add an item to a full buffer. Our implementation is still in a prototype stage, thus several efficiency issues have still to be addressed. The main issue is the time taken for a thread to check whether a marker is enabled or not. Ideally, this time should be close to zero. In our current implementation, the check is linear to the size of the constraint store which produces an acceptable performance of the system. We are investigating further improvements to the implementation including porting it to a more efficient programming language.
Implementing Declarative Concurrency in Java
707
Fairness is implicitly guaranteed by our implementation. Every event that becomes enabled will eventually be executed (provided that the program point associated with it is reached). This is implemented by dealing with event execution requests in a first-in-first-out basis. Although fairness is provided as the default, users, however, may intervene by specifying priority constraints among events. It is therefore possible to specify unfair scheduling.
5
Conclusion
We have described the implementation of a high-level language based on first order logic for expressing synchronization constraints in multi-threaded Java programs. In the language, the safety properties of the system are explicitly stated as temporal constraints. Programs are annotated at points of interest so that the run-time environment enforces specific temporal relationships between the visit times of these points. Constraints are language independent in that the application program can be specified in any conventional concurrent objectoriented language. The constraints have a procedural interpretation that allows the specification to be executed. The procedural interpretation is based on the incremental and lazy generation of constraints, i.e. constraints are considered only when needed to reason about the execution time of current events. This paper presents work in progress so several important issues are still to be considered. Our implementation is still in a prototype stage, thus several efficiency issues have still to be addressed. In particular, we will focus on how the two key features of incrementality and laziness may be most efficiently achieved. Another important issue is how to deal with progress properties. Currently, constraints explicitly state all safety properties of programs. However, the progress (liveness) properties of programs remain implicit. It would be desirable to be able to express these properties explicitly as additional constraints, but so far we have not devised a way to do that. Future versions may also include deadlock detection feature. We are considering a mechanism that checks user constraints for cycles (e.g., A < B, B < A) whenever a timeout occurred. We are also looking into developing a methodology that uses the generative technique for engineering multi-threaded Java programs using constraints. Using this technique, programs may be generated using high-level descriptions. The declarative nature of our language particularly fits this approach.
References 1. Atkinson, C. 1991. Object-Oriented Reuse, Concurrency and Distribution: An Ada-Based Approach. Addison-Wesley. 2. Van den Bos, J. and Laffra, C. 1989. PROCOL: A parallel object language with protocols. ACM SIGPLAN Notices 24(10):95–112, October 1989. Proc. of OOPSLA ’89. 3. Gregory, S. 1987. Parallel Logic Programming in PARLOG, Addison-Wesley. 4. Gregory, S. and Ramirez, R. 1995. Tempo: a declarative concurrent programming language. Proc. of the ICLP (Tokyo, June), MIT Press, 1995.
708
Rafael Ramirez, Andrew E. Santosa, and Lee Wei Hong
5. Gregory, S. 1995. Derivation of concurrent algorithms in Tempo. In LOPSTR95: Fifth International Workshop on Logic Program Synthesis and Transformation. 6. Kiczales, G., Lamping, J., Mendhekar, A., Maeda, C., Lopes, C., Loingtier, J.M. and Irwin, J. 1997. Aspect-oriented programming. In ECOOP ’97—ObjectOriented Programming, Lecture Notes in Computer Science, number 1241, pp. 220–242, Springer-Verlag. 7. Kowalski, R.A. and Sergot, M.J. 1986. A logic-based calculus of events. New Generation Computing 4, pp. 67–95. 8. Krakowiak, S., Meysembourg, M., Nguyen Van, H., Riveill, M., Roisin, C. and Rousset de Pina, X. 1990. Design and implementation of an object-oriented strongly typed language for distributed applications. Journal of Object-Oriented Programming 3(3):11–22. 9. Pratt, V. 1986. Modeling concurrency with partial orders. International Journal of Parallel Programming 15, 1, pp. 33–71. 10. Ramirez, R. 1996. A logic-based concurrent object-oriented programming language, PhD thesis, Bristol University. 11. Ramirez, R., Santosa, A. E. 2000. Declarative concurrency in Java, Proceedings of the 5th International Workshop on High-Level Parallel Programming Models and Supportive Environments (HIPS 2000). Springer-Verlag. 12. Saraswat, V., and Rinard, M. 1990. Concurrent Constraint Programming. Proceedings of the 7th Annual ACM Symposium on Principles of Programming Languages. pp.232-245. 13. Ueda, K. and Chikayama, T. 1990. Design of the kernel language for the parallel inference machine. Computer Journal 33, 6, pp.494-500. 14. De Volder, K. and D’Hondt, T. 1999. Aspect-oriented logic meta programming. In Meta-Level Architectures and Reflection, Lecture Notes in Computer Science number 1616, pp. 250–272. Springer-Verlag.
Building Distributed Applications Using Multiple, Heterogeneous Environments Paul A. Gray1 and Vaidy S. Sunderam2 1
2
University of Northern Iowa, Dept. of Mathematics, Cedar Falls, IA, 50614-0506, USA [email protected] Emory University, Dept. of Mathematics and Comp. Sci., Atlanta, GA, 30322, USA [email protected]
Abstract. There continues to be genuine interest in expanding application environments to include geographically-distributed resources and to be able to dynamically re-configure the environment to suit the needs of the application. The expansion of an application’s resource pool and dynamicity of the environment introduces fresh nuances in programming paradigms and demands novel characteristics of the substrate exportable to the application. This paper describes our research efforts toward the goal of providing applications with such an environment; one that is able to actively adopt and relinquish resource pools, is able to pre-configure and dynamically re-configure attributes to suit the needs of the application, and maintains reliability and persistence in the presence of failures. This paper discusses our findings relating to portability and utility aspects of shared-library usage over heterogeneous environments.
1
Introduction
Distributed computing environments have evolved considerably in the last decade. The Parallel Virtual Machine ([5] environment and the Message-PassingInterface ([13]) (MPI) are two noteworthy projects that have evolved over this time frame. PVM and MPI provide users with an abstract virtual machine (VM) and tools to manage the collection of resources underneath the abstraction. With the APIs that these packages provide, application programmers are given a welldefined and robust environment for distributed and parallel application development. Indeed, the virtual machine approach has been shown to be a very compelling paradigm for a wide variety of applications. However, there are some limitations. For example, one limitation is that the current virtual machine approach limits a process’ interaction with entities outside of the virtual machine. Processes running in two distinct virtual environments are unable to use the virtual machine paradigm to interact. Another related limitation is the lack of a
Research supported in part by NSF grant ACI-9872167
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 709–717, 2000. c Springer-Verlag Berlin Heidelberg 2000
710
Paul A. Gray and Vaidy S. Sunderam
model whereby processes inside of a virtual machine may interact with “open” services such as database and web servers, ftp services and other services which are so prevalent in the networking aspects of today. Further, the issue of process migration is one that is also fundamentally difficult in such environments, where process-naming conventions and check-pointing issues are paramount issues ([12]). Simply stated, naming difficulties and resource-specific attributes such as open files and sockets present good arguments for binding processes to a single resource. In general, the VM paradigm possesses many strong features and will continue being a steadfast aspect of distributed computing in the future. At the same time, the commodity networks and workstations upon which the VM paradigm has its stronghold is evolving at such a rapid rate that commensurate evolutions in the fundamentals of virtual machines are necessary. Microprocessors which drive todays workstations keep getting faster – a lot faster. Further, the cost of a moderately-configured workstation in today’s market is quite reasonable, which allows one easy access to off-the-shelf, state-of-the-art componentry. This is most evident in the growing number of SMP workstations, the wide-reaching grasp of computational grids, and the proliferation of “Beowulf”-type cluster environments available in the market today. All of this means that the future of distributed computing will need to support traditional applications as well as servers, SMPs, and various configurations of clusters. They will also need to be able to adjust to underlying attributes so as to allow application-specific optimizations; being able, for example, to detect and leverage upon a Non-Uniform Memory Access (NUMA) configuration, the availability of high-speed communication attributes such as Myrinet ([1]), and so forth. It is our view that a strong trend in distributed computing exists that incorporates a deep awareness of the underlying architectures of its component resources and also includes the utilization of multiple virtual environments, which are brought together for collaborative or enterprise-type utilization. In abstract terms, virtual environments merge together for a specific collaborative phase and are then able to split apart, in-tact, along arbitrary boundaries. Once merged, applications that were previously restricted to one environment are able to collaborate and to utilize the resources found on the complement environment (left). Further, if permitted, applications are able to instantiate new processes on any capable resource in the combined environment. Figure 1 also shows that these resources may split apart arbitrarily (right). The succession of groups of resources and applications is determined by applications, the owners of the resources or even by faults in the network environment. The processes in the succeeding environments are left in-tact. These processes continue with their designated processing duties, provided they are able and permitted to do so. Note that if the fissure of the environment was caused by a failure in the network, the separate groups may re-merge when the network recovers. Thus, this would allow for more resilient processes, which would be able to survive and recover from environmental catastrophes. Note also that the component environments may re-merge at a later time or merge with alternate environments at any time.
Building Distributed Applications
711
Fig. 1. The merging of virtual environments provides applications access to the resources and applications on the complement. Virtual environments are also permitted to split apart along arbitrary boundaries as illustrated in the depiction on the right. In creating such an environment, methodologies are needed that will enable distinct parallel applications running on top of the environment to discover each other, to synchronize and permit communication, to promote collaboration, and to cleanly detach from each other upon separation. The subject of this paper focuses on the specific task of creating processes upon the resources of the complementary environment once merging of the resources has occured. The difficulties which arise involve foreign architectures which cause binary incompatibilities and locating the appropriate binary executables appropriate for the remote architectures. The approach presented here involves details of how environments are explored in order to detect compatibilities and missing attributes and how the environment may be dynamically re-configured so as to include all of the facilities that a process may need prior to its instantiation. Our approach utilizes both exploration using Java-based front ends and dependence on shared libraries formats for the soft-installation of processes, described in detail in the next section.
2
Designing Dynamic Environments
As mentioned above, one of the issues that comes up when discussing the merging of two distinct resources pools is the utilization of these newly-acquired resources. Take for example, the issue of instantiating a new process; that is the task of physically loading a process upon a remote resources. Resource pools are not assumed to share any common filesystem, nor are they assumed to be binary compatible. Referring back to Figure 1, suppose that the resources within the environment on the left were of different architectures; some running Windows NT as their operating system, others consisting of Sun Sparcs and DEC Alphas running Solaris and Linux as their operating systems. These entities would be unable to instantiate native forms of processes on the complement environment
712
Paul A. Gray and Vaidy S. Sunderam
without some help. This process of probing the application’s needs, exploring the attributes of the environment, re-configuring the environment to make it able to support the application, and the ultimate instantiation of a process onto a remote system is referred to as the soft-installation of a process. The soft-installation process that we’ve employed involves two main phases: a Java-based setup and exploration phase; and phase that involves the locating, relocating, and loading of shared libraries needed by the process. These two phases are explained below. A major prototype for the soft-installation mechanism is part of our project termed “IceT” ([10], [11], [9]), and the sections below describe the manner in which code portability is achieved through the use of Java and shared libraries. The investigations to date involve applications which consist of a Java-based application wrapper and C, C++ or FORTRAN computational substrates. The two are linked together using the Java Native Interface (JNI) and encapsulation of the C, C++, or FORTRAN codes into shared libraries. 2.1
The Role of Java
The main attribute of the Java programming language which we make use of is the bytecode representation. In bytecode form, a Java process is significantly platform independent. We leverage upon this attribute of Java in the process of “handshaking,” i.e. as a way as a way to “introduce” and declare the needs of the application to a remote resource. The major portion of the application code is wrapped by a Java class. The Java-based application wrapper is glued to the application using the Java Native Interface (JNI) and Java-based classes which make calls into the native code. The underlying application code is compiled in the form of a shared library and linked into the Java application wrapper using the Java directive: “System.loadLibrary("foo"),” where “foo” is the name of the application core in its shared library form. The simple Java call: static{ System.loadLibrary("foo")} is embedded in the Java application wrapper’s compiled bytecode. It becomes part of the application’s static bytecode representation which is passed between resources and used for handshaking. That is, the need for the shared library “foo” is embedded in the bytecode that we use for the handshaking process1 . By dis-assembling the bytecode instruction, one can elicit the name of the shared library required for this process’ execution. This describes the methodologies used for the soft-installation process. The application’s bytecode wrapper is disassembled and analyzed for shared library components that will be needed for the process to run. For more details on the detection of the library calls and on the dis-assembly process involving the Java bytecode, see [7]. Once the shared library requirements are determined, the next step is to locate the appropriate form of the shared library for the resource and, if necessary, 1
The embedding of the shared library call into the Java bytecode is part of the Java Virtual Machine definition, and accomplished by the standard Java compiler. No special pre- or post-processing of the bytecode is necessary.
Building Distributed Applications
713
to soft-install the shared library locally. This scenario is depicted in Figure 2. The use of the “foo” shared library is detected by the parsing of the Java bytecode front end, described above. The next task is to fill this foo library requirement with the appropriate binary form, based upon the operating system and architecture of the host system.
FOO.DLL Windows NT Platform DEC Alpha Architecture
Application Code
Shared−library code and data area for "foo"
foo.so Linux Platform Intel x86 Architecture foo.so Solaris 7 Platform Sun Sparc Architecture
Fig. 2. Since shared libraries are not embedded in the static form of the application’s executable, one is able to locate, transport, and “plug-in” these library dependencies according to the particular environment upon which the application is to run. Once a shared library requirement “foo” is detected from the bytecode analysis, the environment looks first to determine if the shared library foo is already present; “FOO.DLL” in the case of a Windows environment or “libfoo.so” in the case of a Unix environment. In the event that the foo library is not found locally, a query is made to a local library server as to the location of the “foo” library for the appropriate operating system (say Windows NT or Linux) and platform (say a DEC Alpha or Intel x86). If the appropriate form of the shared library is unavailable, the creation of the process fails. 2.2
Shared Library Aspects
It may seem like many of the attributes of the environment that we seek can be delivered by the Java runtime environment itself: Heterogeneous execution, portability, and such are all associated with the Java programming language. However, there are many reasons why a pure Java implementation falls short. A Java program simply cannot access the low-level aspects of a system which are ultimately necessary for high-performance. While the environment that we seek to provide allows for highly portable codes, application performance is of equal concern. The environment should facilitate high-performance applications, written to take specific advantage of specialized software. That is, the environment under development will be able to provide portability and high performance gains to applications that depend upon
714
Paul A. Gray and Vaidy S. Sunderam
high-performance tools such as Nexus ([4]), the Virtual Interface Architecture (VIA) ([3]), numerical applications based upon LINPACK routines ([2]) and so on. To illustrate the feasibility of such an implementation, experiments that we have performed using this paradigm of Java-wrapped shared libraries include the facilitation of applications based upon highly-tuned BLAS (Basic Linear Algebra Subroutine), LINPACK routines, CCTL (a high-performance multicast communication package) ([7]) and LAM-MPI ([6]). The performance seen in these native-language packages remains unattainable in Java. It should also be mentioned that there is a slight performance penalty for using Java as an application wrapper as well, and determining the extent of the penalty is also part of our investigations ([8]). An ultimate goal of the environment under development is the transparent utilization of resources in a combined, heterogeneous environment. A major fissure that we have had to cross is when the environment contains both Microsoft Windows and Unix-based workstations (Solaris and/or Linux, for example). This presents a major roadblock inasmuch as the inner workings of the respective operating systems are vastly different. However, both operating systems share enough commonalities so as to permit the mutual soft-installation of a single application built upon shared libraries onto Windows and Unix platforms using the technique outlined above. 2.3
Shared and Static Libraries
Generally speaking, executable programs are applications which receive and process messages, signals, and user input. Typically, libraries are not directly executable and do not receive messages or signals directly. Libraries are separate files which contain functions that can be called by applications and other libraries so as to perform certain tasks. In languages such as C, C++, and FORTRAN, application code is linked with various libraries. The result is an executable file. These libraries which application code are linked may be static or shared. In the Windows environment, shared libraries take the form of dynamic link libraries or “.DLL”s, but the fundamental attributes of dynamic link libraries and shared libraries in Unix environments are the same. Shared libraries in Unix environments are typically identified by a “.so” extension. When an application is linked to a static library, the code within that library becomes an inseparable part of the application’s executable file. Changes made to a static library after the linking process would not be reflected in the executable form of the application. When an application is linked to a static library, the library function calls are resolved at link time. In contrast, when an application utilizes a shared library, linking is done at run time. Shared libraries do not become intertwined with the executable form of the application in the way a static library is embedded into the an application’s executable form. This allows changes made to the library core to be incorporated into the application without re-compilation or changes made of the application. The operating system loads the shared library into memory and resolves the calls into the library during the execution of the application.
Building Distributed Applications
715
These are fundamental differences between shared and static libraries. Another major difference occurs when the application is running. A major difference is that static library codes linked into an application are loaded into memory as part of the executing application whereas shared library code is able to be loaded into memory on-demand. This feature of shared libraries facilitates our mechanism for soft-installation as we can link the appropriate library with our applications on the host where the application is to be executed and at the time the application requires it. Other differences between static and shared libraries effect the manner in which one writes an application. Each application that is linked to a static library embeds a copy of the static library’s codes into its own executable. Thus, every application has its own copy of the static library code necessary for its execution. A shared library, on the other hand, exists uniquely. Applications that have linked to a shared library simultaneously share the single instance of the library. That is, if several applications are running at the same time, the shared library code is loaded only once into the environment, and mutually shared and accessible to each application that it has been linked to. Thus, care must be taken when one writes a shared library so as to avoid unintended manipulation of the shared library’s resources. So, while building a shared library may be as simple as changing a compiler flag, there are some significant attributes of shared libraries that effect how applications are executed. The singular shared library aspect disadvantage is evident in the intrinsic way an application interacts and links with a shared library. If two or more applications that utilize the same shared library are running simultaneously, the shared library is loaded into the operating system’s kernel only once. Thus, each application is sharing a single copy of common code and resources. For this reason, it is necessary that the code in the shared library be reentrant. As there is a single shared library instance shared by all applications depending upon it, global variables in the shared library code are at risk of being manipulated simultaneously by independent processes — a situation to be avoided at all costs. This puts additional semaphore and mutex coding responsibilities on the application programmer.
3
Conclusion
There are significant benefits to shared-library utilization in the soft-installation approach. Shared libraries are linked independently of the applications that use them which allows us to detect and relocate shared libraries as necessary and permits the use of highly-tuned libraries that are maintained outside of the application. Our preliminary experimentation in providing portability based upon shared library usage has been very successful. However, shared library dependence also brings with it some disadvantages which need to be considered when developing applications, such as the issues of reentrant and thread safe code. These disadvantages are not insurmountable. First, the primary disadvantage is that shared libraries are much more difficult
716
Paul A. Gray and Vaidy S. Sunderam
to develop: One must be aware of the aspects which allow a single instance of the library to be utilized by multiple applications at the same time. Thus, a necessary aspect of shared libraries which are able to be utilized by multiple applications is the that the library must be reentrant. Preliminary investigations have shown feasibility and proof of concept for the portability and distribution of processes as described in this paper. These experimentations include extending portability to BLAS, CCTL, LINPACK, and MPI libraries. Our ongoing investigations involve incorporation of user authentication, process validation, and pinpointing the conflicts in a multiple-application, single-library setting.
References 1. Nanette Boden, Danny Cohen, Robert Felderman, Alan Kulawik, C. L. Seitz, and J. N. Seizovic. Myrinet: A gigabit per second local area network. IEEEMicro, 15(1), February 1995. 2. Jack Dongarra, Jim Bunch, Cleve Moler, and Pete Stewart. The FORTRAN– based LINPACK routines. Available from NAG, Downers Grove, IL. 3. Dave Dunning, Greg Regnier, Gary McAlpine, Don Cameron, Bill Shubert, Frank berry, Anne Marie Merritt, Ed Gronke, and Chris Dodd. The Virtual Interface Architecture. IEEE Micro, 18(2), March/April 1998. 4. I. Foster, C. Kesselman, and S. Tuecke. The Nexus Approach to Integrating Multithreading and Communication. J. Parallel and Distributed Computing, 37:70–82, 1996. 5. G. A. Geist and V. S. Sunderam. The PVM system: Supercomputer level concurrent computation on a heterogeneous network of workstations. In Proceedings of the Sixth Distributed Memory Computing Conference, pages 258–261. IEEE, 1991. 6. Vladimir Getov, Paul Gray, Sava Mintchev, and Vaidy Sunderam. Multilanguage programming environments for high performance Java computing. Scientific Programming, 9(11):1161–1168, November 1999. 7. Vladimir Getov, Paul Gray, and Vaidy Sunderam. Aspects of portability and distributed execution for JNI-wrapped code. To appear in a special issue on MPI in Concurrency: Practice and Experience. 8. Vladimir Getov, Paul Gray, and Vaidy Sunderam. MPI and Java-MPI: Contrasts and comparisons of low-level communication performance. In Proceedings of Supercomputing 99, November 1999. 9. P. Gray and V. Sunderam. IceT: Distributed Computing and Java. Concurrency: Practice and Experience, 9(11):1161–1168, November 1997. 10. P. Gray and V. Sunderam. Native Language-Based Distributed Computing Across Network and Filesystem Boundaries. Concurrency: Practice and Experience, 10(1), 1998. 11. Paul Gray and Vaidy Sunderam. The IceT Environment for Parallel and Distributed Computing. In Y. Ishikawa, R. R. Oldehoeft, J. V. W. Reynders, and M. Tholburn, editors, Scientific Computing in Object-Oriented Parallel Environments, number 1343 in Lecture Notes in Computer Science, pages 275–282, New York, December 1997. Springer Verlag.
Building Distributed Applications
717
12. M. Litzkow, M. Livny, and M. W. Mutka. Condor – A hunter of idle workstations. In Oriceedubgs if the 8th International Conference of Distributed Computing Systems, pages 104–111, June 1988. 13. Marc Snir, Steve W. Otto, Steven Huss-Lederman, David W. Walker, and Jack Dongarra. MPI, The Complete Reference. MIT Press, November 1995.
A Multiprotocol Communication Support for the Global Address Space Programming Model on the IBM SP Jarek Nieplocha Jialin Ju Tjerk P. Straatsma Pacific Northwest National Laboratory, Richland, WA 99352, USA http://www.emsl.pnl.gov/docs/parsoft/armci
Abstract. The paper describes an efficient communication support for the global address space programming model on the IBM SP, a commercial example of the SMP (symmetric multi-processor) clusters. Our approach integrates shared memory with active messages, threads and remote memory copy between nodes. The shared memory operations offer substantial performance improvement over LAPI, IBM one-sided communication library, within an SMP node. Based on the experiments with the SPLASH-2 LU benchmark and a molecular dynamics simulation, our multiprotocol support for the global address space is found to improve performance and scalability of applications. This approach could also be used in optimizing the MPI-2 one-sided communication on the SMP clusters.
1 Introduction This work is motivated by applications that require support for a shared-memory programming style rather than just message passing. Many of them are characterized by irregular data structures, and dynamic or unpredictable data access patterns. For certain types of applications, the shared-memory programming model can be substituted or supported with the global address space model. Systems with a global address space usually do not offer coherent shared memory at the operating system level. Instead, in a distributed-memory environment they provide remote memory operations, for example as in the SHMEM library [1] on the Cray T3E, or one-sided communication operations in the MPI-2. The global address space can be used directly by applications, or indirectly supporting a shared-memory view of data structures emulated through a user-level library interface, such as the Global Arrays (GA) [2], that transparently to the user performs an appropriate translation of shared to distributed memory references. The programming model based on the global address space has the added benefit of preserving data locality information (the application is explicitly aware of, and controls data distribution) that well-written distributedmemory message-passing applications can exploit to increase performance. It also allows access the data in a fashion similar to shared memory without the interprocess synchronization imposed by the traditional cooperative message passing model. In recent years, clustered systems with symmetric multi-processor (SMP) nodes have become increasingly popular as the cost effectiveness of SMP nodes and highperformance networks improved. An example of such an architecture is the IBM SP, a distributed-memory machine with SMP nodes, a network supporting highperformance user-space communication, and a rich programming environment that offers active messages and remote memory copy through the LAPI system library, threads, thread-safe MPI, and the standard Unix shared-memory interfaces within the A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 718-728, 2000. © Springer-Verlag Berlin Heidelberg 2000
A Multiprotocol Communication Support
719
SMP nodes. Our goal is to optimize communication support for the global address space on clustered SMP-based systems for applications that use one process/task per processor (rather than the explicitly multithreaded single process with per SMP node). The main contribution of this paper is an integration of multiple communication protocols such as active messages, threads, and remote memory copy with shared memory to support the global address space model efficiently. Performance of onesided operations is substantially improved by mapping the global address space to the shared-memory segments on SMP nodes. This technique makes a low-level messaging library such as LAPI responsible only for internode communication while the intranode communication is handled directly by shared memory. Performance advantages of this approach are shown for the remote-memory operations in the ARMCI portable communication library[3] and two applications, the SPLASH-2 LU benchmark and NWChem molecular dynamics code. The performance of NWChem was improved by 12.6% on 64 processors by relinking this large and already well tuned application unchanged with a different version of the ARMCI library. Although this paper focuses on the IBM SP, we used the multiprotocols for supporting global address space through ARMCI on other cluster platforms. We chose the SP since it 1) is a major cluster platform widely used for technical computing, 2) offers a richer set of vendor supported protocols (including active messages) relevant to our objectives than most other cluster platforms, and 3) all the discussed protocols are widely available in the user mode and supported on the current SP system configurations. Shared memory has been exploited before in low-level one-sided messaging systems like Active Messages[4] or Nexus [5], and higher-level two-sided messagepassing interfaces like MPI [6] on SMP clusters. However, as the programming models based on the global-address space and message passing are fundamentally distinct, different strategies are needed to pass benefits of shared memory to the applications. For example, the IBM implementation of MPI on SMP nodes exploits shared memory to move data between a pair of MPI tasks through an internal buffer in shared memory with one task copying data into the shared memory buffer and then the other copying into its separate address space. ARMCI places the application data in shared memory thus the data can be accessed without any intermediate memory copies and message buffer management overheads. It helps achieve 15 times better latency and 67% better bandwidth than in the vendor version of MPI within the SMP node. Similar performance gains are realized through the shared memory in ARMCI over LAPI within the node. On the IBM SP, we found that the shared memory optimizations benefit more the remote memory/one-sided operations than the message passing. The rest of this paper is organized as follows: Section 2 describes integration of shared memory with LAPI and thread-based protocols; Section 3 reports the results of basic communication operations; Section 4 presents experimental results of two applications, the SPLASH-2 LU benchmark and the molecular dynamic simulation; Section 5 discusses related work. Finally, we conclude in Section 6.
2 SMP-Aware Communication Protocols We developed ARMCI[3] to support remote memory operations in the context of distributed array libraries such as GA and compiler run-time systems such as the Parallel Runtime Consortium [7] Adlib. ARMCI supports remote memory copy, accumulate, and synchronization operations. It is portable and compatible with message-passing libraries such as MPI or PVM. Unlike most existing similar facilities,
720
Jarek Nieplocha, Jialin Ju, and Tjerk P. Straatsma logically partitioned shared memory segment
local process memory
P0
P1
SMP node
P2
P3
put
LAPI
LAPI
adapter
adapter
P0
P1
P2
P3
SMP node
network
memc py
Figure 1: Combining shared memory with LAPI on the IBM SP
such as Cray SHMEM, it focuses on the noncontiguous data transfers that correspond to the data structures used in scientific applications (sections of multi-dimensional dense or sparse arrays, scatter, gather). Such transfers are optimized thanks to the noncontiguous data interfaces available for ARMCI data transfer operations: multi-strided and generalized UNIX I/O vector interfaces. ARMCI offers a simpler model and lower-level interface than the MPI-2 one-sided communication. The standard parallel programming environment of the IBM SP includes LAPI, a low-level one-sided communication system that supports active messages (AM) and remote memory copy operations. LAPI offers competitive performance to MPI; however, unlike MPI it does not support noncontiguous data transfers which are common in scientific codes. ARMCI on the IBM SP uses active messages and threads rather than remote memory copies to optimize noncontiguous data transfers. Each process in a LAPI program uses at least three threads: the main application thread, and two extra threads for executing the LAPI active message handlers. In applications that use MPI in addition to LAPI, there are three additional threads introduced by the thread-safe IBM MPI library. For example, on the 4-way SMP node with 4 user processes (explicitly single threaded) that use MPI and LAPI, there are 4x6=24 threads. LAPI is supported in the SMP environment. However, since it uses the network adapter [8] to move data between processes on the same SMP node as if they were on separate nodes, its performance is not optimal. In particular, to transfer data between address spaces of two processes, LAPI performs at least two memory copies (to and from the adapter DMA area), and in many cases generates an interrupt. We developed hybrid communication protocols for the ARMCI to exploit the SMP locality information and shared memory communication. The locality information is determined at start time from the process-to-node mapping, and used to select appropriate communication protocols. Within the SMP node, all operations are implemented on top of shared memory rather than with LAPI, see Figure 1. For internode communication, depending on the request size and shape, ARMCI chooses between active messages (AM) or remote memory copies to optimize bandwidth, see Figure 2. An implementation of the gather operation used by ARMCI is described in [9]. Since ARMCI includes a memory allocation routine to be used in the context of remote memory copy, memory is allocated using the System V shared memory
A Multiprotocol Communication Support
Y
Target
ORIGIN PROCESS
Strided remote memory copy using multiple nonblocking LAPI_Get
process on the same SMP node? N Wait until previous store operations to the same location complete (LAPI_Waitcntr)
Y
small section or single/large columns
Wait until data arrives (LAPI_Waitcntr)
TARGET PROCESS AM header handler returns thread t1 address of completion handler function and saves request info thread t2 strided copy from the specified data source location into the temp buffer b1 (AM completion handler)
721
Strided shared memory copy
N Acquire local buffer b0 Packetize request to match the buffer size Send AM request to target, include address of buffer b0 (LAPI_Amsend) Wait for data to arrive into the buffer (LAPI_Waitcntr) Strided copy data from the buffer into the user location
send data from buffer b1 to buffer b0 at origin processor(LAPI_Put) Figure 2: SMP-aware implementation of the ARMCI strided get operation on the IBM SP
interfaces. This critical function helps reduce the put/get operations to a memory copy and avoid at least one memory copy that LAPI must do. As explained below, the main difficulty is posed by the requirement to make multithreaded active-message internode protocols compatible with intranode shared memory operations. ARMCI, in addition to remote memory copy, supports atomic operations: accumulate (reduction) as well as read-modify-write. The mutual exclusion embedded in the semantics of these operations requires special care when hybrid protocols are used. ARMCI on uniprocessor nodes uses active messages, threads, and Pthread mutexes. Since the same memory region in address space of process A can be addressed concurrently by a process B executing on the same SMP node and process C on a remote node, the mutual exclusion primitives must synchronize multiple threads in both the same and different process spaces. In AIX, Pthread mutexes cannot be used in this context. We designed a mutex lock replacement for both thread mutexes and System V semaphores. It offers a very low overhead and is free of the disadvantages of spin locks that can waste substantial amounts of CPU time by not yielding the processor when the mutex cannot be acquired for an extensive amount of time. We use the AIX atomic operation check_lock to check the content of a word. If the mutex is already acquired by another thread we use a spin lock with limited asymptotic backoff before finally yielding the processor to another thread.
722
Jarek Nieplocha, Jialin Ju, and Tjerk P. Straatsma 140
140
bandwidth [MB/s]
120
shmem
120
100 LAPI remote
80 60
80 60
LAPI SMP
40 20 0 1
shmem
100
LAPI remote
40 20
100
10000
1000000
0 1
bytes
LAPI SMP
100
10000
1000000
bytes
Figure 3: Performance of contiguous get (left) and contiguous accumulate (right) operations implemented using shared memory(shmem) and LAPI on the same SMP or remote node.
3 Performance of Communication Operations We used a 16-node, 4-way SMP IBM SP with 64 PPC-604e processors and the TB3MX adapter at PNNL to study the performance of remote memory operations, the SPLASH-2 LU kernel benchmark and the molecular dynamics simulation. In this section, we discuss performance of the ARMCI get and accumulate operations in accessing contiguous and strided data on the same and remote SMP node. For the same node, the performance using the LAPI-based protocols and the shared-memory operations is presented. Figures 3-4 demonstrate that our SMP-aware protocol outperforms LAPI by a large margin within a node. Interestingly, the observed bandwidth in the LAPI protocols used for intranode communication is in many cases worse than for the internode communication. This phenomenon is attributed to the fact that LAPI uses the network adapter even when communicating within the same node. For the intranode communication, the same adapter is used for sending and receiving the data, whereas for the internode communication the adapter handles only one side of the data transfer. Despite the obvious differences between one-sided protocols in ARMCI and twosided point-to-point message-passing protocols in MPI, we present performance of these systems to demonstrate how these interfaces take advantage of shared memory. Unlike LAPI, the IBM implementation of MPI already uses shared memory for communication within the SMP node. Table 1 shows latency and bandwidth numbers for the ARMCI get and the MPI send/receive operations. In this test, we used contiguous data transfers and tried to assure that the data is not already in cache. The MPI latency below is reported as 1/2 of the roundtrip time between two tasks for 1byte message size, and we measured bandwidth for 512KB data size. The primary reason ARMCI outperforms MPI by a wide factor within the SMP node is that ARMCI get operation reduces simply to a memory copy whereas the MPI protocol adds the cost of message queue management and requires two memory copies and cooperation of two tasks to move data through an MPI internal shared memory buffer. Of course,
A Multiprotocol Communication Support
723
these advantages apply to the intranode environment. For the internode communication ARMCI uses LAPI, and in this case MPI and LAPI performances are similar [10]. The SMP-aware protocols, by improving communication performance within an SMP node, expose one additional layer of memory hierarchy in the IBM SP and provide an additional optimization opportunity to the applications. Table 1: Performance of MPI send/receive and ARMCI get on the SMP node interface
latency [µs]
bandwidth [MB/s]
MPI
13.2
65.63
ARMCI
0.85
109.64
4 Application Study We used the SPLASH-2 LU benchmark and the NWChem molecular dynamics code to study the implication of SMP-aware communication protocols on the application performance. Neither of these codes was developed to exploit performance characteristics of the SMP-based clustered systems, and in this respect, are representative of the majority of scientific parallel codes. Since both codes perform internode and intranode communication, it was not clear what degree of performance improvement should be expected. 4.1 LU SPLASH-2 Benchmark The SPLASH-2 benchmark suite [11] is a set of parallel applications for use in the design and evaluation of shared-memory multiprocessing systems. The suite contains two types of codes: full applications and kernels. We chose the LU program, which is one of the kernel programs from SPLASH-2, to evaluate the performance of our approach. The LU program factors a dense matrix into the product of a lower triangular and an upper triangular matrix. The factorization uses blocking to exploit temporal locality w.r.t. individual submatrix elements [12]. Originally designed to run on shared memory systems, this benchmark can only be used on a single SMP node of the IBM SP. Some modifications were needed to use the global address space model. 120
160 140
shmem
100
shmem
120
80
100 80
60
LAPI remote
60
LAPI remote 40
40
LAPI SMP
20 0 1
LAPI SMP
20
0 100
10000 bytes
1000000
1
100
10000 bytes
10000 00
Figure 4: Performance of strided get (left) and strided accumulate (right) operations imple-
mented using shared memory (shmem) and LAPI on the same SMP or remote node.
724
Jarek Nieplocha, Jialin Ju, and Tjerk P. Straatsma 9
4
8 7
3 speedup
6 5
2
4 cyclic armci
block lapi+shm
3
block armci
1
block lapi
2
cyclic pthreads
cyclic lapi+shm
1
block pthreads
cyclic lapi
0
0 0
1
2
3
number of processors
4
5
0
4
8
12
16
20
number of processors
Figure 5: Speedup in the Pthread and ARMCI Figure 6: Speedup in the SPLASH-2 LU
versions of the SPLASH-2 LU benchmark using benchmark using block and block cyclic block and block cyclic distributions on one SMP distributions and SMP-aware or SMP-oblivious communication protocols node
We also developed a Pthread version of the benchmark to evaluate the performance of our modifications within an SMP node. The primary issues to be addressed included memory allocation, access to shared data, and interprocess/thread synchronization. The Pthread version uses threads while the ARMCI version uses processes. In the first case, shared data is located in the process memory and accessed directly by the threads as needed. In order to replace shared memory with global address space, we had to divide the shared data, assign it to individual processes, and allocate the corresponding storage on each process. To synchronize processes, we used MPI_Barrier. Threads are synchronized with a Pthread mutex and a condition variable. During the LU factorization, if required data blocks are in the local process memory, they are accessed directly. Otherwise, ARMCI_Get is used to copy the data block from the remote process that owns it to the local temporary storage. The computation requires transferring data blocks from the same row and column (for diagonal blocks). The original benchmark uses block cyclic distribution for load balancing. We also used block decomposition, as it had better locality of the data accesses. To take advantage of the SMP performance, the blocks are distributed according to a block pattern, such that the block that needs to be transferred has a better chance of residing in the local memory or neighboring memory on the same node. We used a matrix size of 3072 and a block size of 32 to study performance of the SPLASH-2 LU benchmark. The performance results given in Figure 5 indicate that the global address space and shared memory versions of the benchmark have similar performance for the same distribution types. The block cyclic distribution gives better performance than the block distribution because of better load balancing properties. As the Pthread version works only on a single SMP node of the SP, we used only the ARMCI version of the benchmark on multiple nodes to study performance of the SMP-aware communication. We relinked the program with two alternative versions of ARMCI - one that uses only LAPI and the other that uses LAPI for internode communication and shared memory on a node. Since the block cyclic decomposition yields better load balancing the code scales better on a single SMP node. However, the lower memory reference locality compared to block decomposition causes
A Multiprotocol Communication Support
725
communication patterns to be spread out across all the nodes. Consequently, the benchmark performance on multiple nodes is better with block decomposition. To exploit performance advantages of both decomposition schemes, we also developed a third version of the benchmark that uses block cyclic decomposition on a single SMP node and block decomposition on multiple SMP nodes. Its performance is a good as its component protocols in their optimal operational regimes, see Figure 6. The impact of using faster communication within SMP nodes is more significant with the block rather than block cyclic decomposition because it better exploits data locality. Improved performance is observed for the approach using LAPI and shared memory, although the difference is small when few processors are used. This is because more time is spent in computation than in communication. With more processors, the multi-protocol approach shows better performance, proving the benefits of shared memory support for the global address space. Figure 6 shows some performance degradation effects in the LAPI-only version with four processors. We contribute this phenomenon to the hardware and software interaction involving the heavily used (for all the intranode communications) network adapter and scheduling of the 24 threads for a particular communication pattern in the benchmark. As our approach relieves the adapter from handling intranode communication any similar performance degradations have never been experienced. 4.2 Molecular Dynamics Simulation NWChem is a massively parallel computational chemistry software package developed on top of Global Arrays (GA). This large (>600,000 code lines) package contains multiple methods for quantum-mechanical and classical computational chemistry including molecular dynamics. The recent version of GA employs ARMCI as its runtime system. GA maintains only the distributed array infrastructure, performs array index translation to the global address space, and implements collective array operations. All the GA one-sided communication is supported through ARMCI. Molecular dynamics (MD) simulations typically evaluate atomic interactions within a specified cutoff distance. The implementation of the method in NWChem uses this locality of atomic interactions in the way the data is distributed on a distributed-memory MPP. This domain decomposition of the physical simulation volume exploits the locality of the atomic interactions such that all-to-all interprocessor communication is avoided. On computers where the cost of communication between processor pairs is not homogeneous, such as SMP clusters, an additional optimization of the domain decomposition is in principle possible for example by avoiding the assignment of physically distant parts of the simulation volume to processors capable of fast communication. In practice, this means that the locality in the simulation space is reflected in the choice of processor assignment. The processors on a given SMP node should typically be assigned adjacent parts of the simulation volume, for which communication requirements are always high. Molecular dynamics simulations of liquid water were carried out with three versions of the code, see Figure 7. The application uses numbers of processors that are powers of 2. The original version is based on the ARMCI library that uses only LAPI for communication between and within SMP nodes. The other versions use the SMPaware ARMCI library, with and without an additional permutation of the process numbers (inside GA) to improve communication locality based on our analysis of the MD communication patterns. No modification of the application code was needed.
726
Jarek Nieplocha, Jialin Ju, and Tjerk P. Straatsma 72 64 56 speedup
48 40 32 24
smp-permuted
16
smp
8
original
0 0
16
32
48
64
number of processors
Figure 7: Performance of the MD simulation with SMP-aware and unaware (original) versions of ARMCI. A permutation of process numbers is used to improve communication locality
The three versions were produced by simply relinking the application with different versions of the ARMCI and GA libraries. Similarly to the LU benchmark, the shared memory multiprotocols improved performance and scaling of this application. The improvement rate depends on the number of processors used and ratio of communication to calculation. The SMP optimizations are more effective when communication is made through a permutation of the process numbers. On 64 processors a 12.6% performance improvement is achieved over the original version. Since this well optimized application had already scaled almost linearly up to 32 CPUs, this is a substantial improvement, and it was achieved without any explicit modifications to this complex code. Moreover, 3dimensional MD simulations on 4-way SMP clusters could not take full advantage of the communication locality. Our analysis indicates that with the increased number of CPUs per SMP node, the performance improvements should be even better.
5 Related Work Multiprotocols involving shared memory has been used before to optimize performance of two- [6] and one-sided messaging systems [4,5]. Husbands and Hoe [6] used the shared memory mechanism for intra-SMP communication in an MPI implementation for a cluster interconnected by the StarT-X network. They optimized contiguous data transfer through a shared memory transfer facility in the MPICH channel layer on the node. ARMCI unlike MPI-1 supports one-sided communication. Lumetta el., [4] proposed multi-protocol Active Messages on SMP cluster. Thir multiprotocol implementation of Active Message directs message traffic through the appropriate medium, either shared memory or network. Separate message queues are maintained for these two media. The shared memory queue block must first be mapped into address space of processes on the node. Message polling operations are ubiquitous in Active Message layers even with shared memory implementation, whereas ARMCI does not require polling. Also unlike [4], ARMCI incorporates threads among other protocols used. Foster, el., [5] discussed the multi-protocol communication in the Nexus multi-threaded one-sided communication system that extends to heterogeneous platforms. Two-level protocols are available: one better suited for small, latency-
A Multiprotocol Communication Support
727
sensitive communications, and the other for large communications. The application performance can be optimized by using one link for synchronization and the other for data transfer. In ARMCI, selecting the protocol is not an issue anymore. Nexus unlike ARMCI does not include active messages among the protocols considered in [5]. The other differences between these papers and our work arise from the fact that ARMCI: 1) supports one-sided remote memory operations rather than one- or twosided message-passing interfaces; 2) emphasizes noncontiguous data transfers (not addressed in papers [4-6]); and 3) supports atomic remote memory operations. In our experience, one-sided messaging libraries such as Nexus or Active Messages are very well suited for implementing inter-node communication but they add unnecessary overhead on the SMP node (e.g., related to message queue management, flow control, buffering) that can be avoided in the context of global address space model by using shared memory directly.
6 Conclusions and Future Work We described a multiprotocol communication support for the global address space on the IBM SP that integrates shared memory within SMP nodes with the LAPI active messages, threads and remote memory copy between nodes. Shared memory offers substantial performance improvements over LAPI within a node for both contiguous and noncontiguous data transfers. Based on the SPLASH-2 LU benchmark and molecular dynamics simulation, the multiprotocol support for global address space is found to improve both performance and scalability of these applications. In the application context we have also found that 1) distribution plays an important role in exploiting the shared memory effectively and 2) replacing a shared memory programmming style (Pthreads) with the global address space model does not lead to performance losses within the SMP domain and provides good scaling across the cluster. Another important benefit of this approach is that the intranode communication is no longer involving the network adapter and its resources thus they can now be devoted exclusively to inter-node communications. This technique can also be used in implementations of the MPI-2 (not yet available on the IBM SP) on SMP clusters since it also offers a memory allocation interface (MPI_Alloc_mem) in the context of the one-sided operations. The described integrated protocols are employed in ARMCI for supporting the global address space model on more SMP-based systems than just the IBM SP. We intend to extend the ARMCI capabilities (e.g., support for heterogenous systems) and use it to implement other interfaces. For example, ARMCI is well suited for a portable implementation of the SHMEM library and we are also considering using it to support the MPI-2 1-sided “passive communication model”.
References 1. R. Barriuso, Allan Knies, SHMEM User’s Guide, Cray Research, SN-2516, 1994. 2. J. Nieplocha, R. J. Harrison, R.J. Littlefield. Global Arrays: A shared memory programming model for distributed memory computers. Proc. Supercomputing’94. 3. J. Nieplocha, B.Carpenter, ARMCI: A Portable Remote Memory Copy Library for Distributed Array Libraries and Compiler Run-time Systems, Proc. RTSPP IPPS/ SDP’99, 1999. 4. S. Lumetta, A. M. Mainwaring, D.E. Culler, “Multi-Protocol Active Messages on a Cluster of SMP’s”. Proc. Supercomputing’97, 1997.
728
Jarek Nieplocha, Jialin Ju, and Tjerk P. Straatsma
5. I. Foster, J. Geisler, C. Kesselman, S. Tuecke, “Managing Multiple Communication Methods in High-Performance Networked Computing Systems,” Journal of Parallel and Distributed Computing, Vol. 40, January 1997. 6. P. Husbands, J.C. Hoe, “MPI-StarT: Delivering Network Performance to Numerical Applications”. Proc. Supercomputing’98, 1998 7. Parallel Compiler Runtime Consortium, Common Runtime Support for High-Performance Parallel Languages, Proc. Supercomputing’93, 1993. 8. R. Govindaraju, IBM Power Parallel Systems, personal communication, 1999. 9. S.Andersson,G.Bhanot, J.Hague, F.Johnston, S. Kandalai, D. Klepacki, J. Levesque, J. Nieplocha, F. O’Connell, F. Parpia, C. Pospiech, Scientific Applications in IBM RS/6000 SP Environments, IBM Corp., ISBN: 0738415189, 1999. 10. M. Banikazemi, R. Govindraju, R. Blackmore, D. Panda, Implementing Efficient Implementation of MPI for IBM RS/6000 SP Systems, Proc. IPPS/SDP’99, 1999. 11. S.C. Woo, M.O. Ohara, E. Torrie, J.P. Singh, A. Gupta. "The SPLASH-2 Programs: Characterization and Methodological Considerations". Proc. 22nd International Symposium on Computer Architecture, 1995. 12. S.C. Woo, J.P. Singh, J.L. Hennessy, The Performance Advantages of Integrating Block Data Transfer in Cache-Coherent Multiprocessors. Proc.6th ASPLOS, 1994.
A Comparison of Concurrent Programming and Cooperative Multithreading Takashi Ishihara, Tiejun Li, Eugene F. Fodor, and Ronald A. Olsson Department of Computer Science, University of California, Davis, CA 95616 USA {ishihara,liti,fodor,olsson}@cs.ucdavis.edu
Abstract. This paper presents a comparison of the cooperative multithreading models with the general concurrent programming model. It focuses on the execution time performance of a range of standard concurrent programming applications. The results show that in many, but not all, cases programs written in the cooperative multithreading model outperform those written in the general concurrent programming model. The contributions of this paper are an analysis of the performances of applications in the different models and an examination of the parallel cooperative multithreading programming style.
1
Introduction
The general concurrent programming execution model (CP) typically provides independent processes as its key abstraction. Processes execute nondeterministically. That is, processes run in some unknown order, which can vary from execution to execution, and context switches can occur arbitrarily. Multiple processes within a given program may execute at the same time on multiple processors, e.g., on a shared-memory multiprocessor or in a network of workstations. This model of execution is found in many concurrent programming languages — e.g., Ada [9], CSP [8], Java [5], Orca [3], and SR [1]. These languages provide various synchronization mechanisms to coordinate the execution of processes (e.g., semaphores, monitors, or rendezvous). The cooperative multithreading execution model (CM) is a more specialized model of execution. Threads execute one at a time. A thread executes until it chooses to yield the processor or to wait for some event to become true. The kinds of events for which a thread can wait include a shared variable meeting a particular condition, a device completing some operation, or a timeout occurring. This model of execution is especially well-suited for writing programs for realworld programmable controllers for embedded systems [2], such as those found in irrigation control systems and railroad crossing control systems. One language for writing these controllers is Z-World’s Dynamic C [15]. The CM model as defined above allows only one thread to be active at any given time. A natural generalization of CM (called PCM, for Parallel CM) is
This work is partially supported by Z-World, Inc. and the University of California under the MICRO program.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 729–738, 2000. c Springer-Verlag Berlin Heidelberg 2000
730
Takashi Ishihara et al.
to allow multiple threads to be active simultaneously, so that a CM program can run on multiple processors. However, to preserve some of the advantages of CM (described later), some restrictions need to be placed on which particular threads can be run simultaneously. For example, only one thread per module, as in Lynx [12, 13], or one thread per group of threads with possibly interfering variable usages may be active at any time. Two significant advantages of CM have been pointed out in the key work [12, 13] and specifically for controllers in [2]: CM is a simpler conceptual model and threads often do not need to synchronize explicitly because threads yield the processor at fixed places in the code. Two other tradeoffs [6] involve how the effect of I/O can vary in the different models and the relationship of execution fairness to program determinacy. In this paper, we present a comparison of the cooperative multithreading models (CM or PCM) with the general concurrent programming model (CP). We focus on the execution time performance of a range of standard concurrent programming applications. The results show that in many, but not all, cases programs written in the CM- or PCM-style outperform those written in the CP-style. The contributions of this paper are an analysis of the performances of applications in the different models and an examination of the PCM programming style. (This paper extends our preliminary performance comparison of CM and CP [6].) Although the programs used in our experiments are written in SR, the general performance results should apply to other languages and systems. The specific performance results will vary depending on relative costs of synchronization and context switches, etc. The rest of this paper is organized as follows. Section 2 briefly compares language features typical in the three models. Section 3 presents execution time performance comparisons of programs written in the CM- or PCM-style with their counterparts written in the CP-style for several standard CP applications. Section 4 addresses additional issues. Finally, Section 5 concludes the paper.
2
Language Features
We assume most readers are familiar with CP languages (such as Ada, etc. mentioned in Section 1) but are less familiar with CM or PCM languages. We therefore briefly present the essential ideas of two such languages — Dynamic C [15] and Lynx [12, 13]. Dynamic C extends the C language with various features to support CM. A thread executes until it chooses to yield the processor or to wait for some event to become true. Yielding the processor is accomplished via explicit statements: yield and waitfor. yield context switches to another ready thread, if any, or resumes the current thread if no other thread is ready. waitfor evaluates the condition. If true, the thread continues; otherwise, the thread yields and will, therefore, reevaluate the condition when it runs again. (Some notations use await rather than waitfor.)
A Comparison of Concurrent Programming and Cooperative Multithreading
731
Lynx is conceptually similar to Dynamic C. A Lynx program consists of a single module, called a process. Each process may contain multiple threads. As in Dynamic C, only one thread may be active at a time and threads execute until they block. An important feature of Lynx is that Lynx programs (processes) can communicate via messages using a link mechanism; the receipt of a message creates a new thread to run code to handle the message. Lynx falls under the PCM model because multiple threads — but at most one in each process within a group of processes — may execute at a time. To illustrate the differences between the CP and CM models, Figure 1 shows
do true -> do # think ... # get forks # (use semaphore operations # on shared sem array fork; # left and right are indices # of neighboring philosophers) P(fork[left]);P(fork[right]) # eat ... # release forks V(fork[left]);V(fork[right]) od od (a) CP-style
true -> # think ... # get forks, by simulating: # await fork[left]=1 & fork[right]=1 do not(fork[left]=1 & fork[right]=1)-> nap(0) # i.e., yield od fork[left] := 0; fork[right] := 0 # eat ... # release forks fork[left] := 1; fork[right] := 1
(b) CM-style
Fig. 1. Code for a Philosopher in Dining Philosophers
how the classic dining philosophers problem can be solved in CP and CM. The CP code uses standard SR features; for synchronization, it uses the shared array of semaphores fork. The CM code use only CM-like features from SR; for synchronization, it uses the shared array of integers fork and a simulated await statement. This simulation uses SR’s nap function to explicitly yield. The CM code is compiled with an option that prevents normal, implicit context switches. The key difference in these program fragments is how the philosopher checks the status of its two neighboring philosophers to decide whether it can eat. In CP, synchronization is required to avoid race conditions. By contrast, in CM, a context switch can occur only explicitly. Thus, no context switch can occur within evaluation of the conditions of the (simulated) await or between that test, if true, and the subsequent setting of the two elements of fork. (Note that the CM code here is not valid under PCM; see Section 3.2.)
732
3
Takashi Ishihara et al.
Experimental Results
We wrote in the CM and PCM programming styles several standard CP applications including grid computations such as Jacobi Iteration (JI) (for approximating the solution to a partial differential equation), http servers, Producer/Consumer (PC1 ), Dining Philosophers (DP), Readers and Writers (RW), Matrix Multiplication (MM), and Traveling Salesman Problem (TSP). For the PC problem, we use the notation xPyCzS to mean x producers, y consumers, and a buffer with z slots. We focus below on the DP, PC, and JI applications. We programmed the applications in the SR language, using standard CP features or CM-like features, as we did for the DP code in Figure 1. The PC and JI programs are fairly straightforward (the CP versions are taken from [1]). Below, we compare CP with CM (both running on a single processor)2 and CP with PCM (both running on a multiprocessor) by looking at the execution times for some of the applications mentioned above. We ran on many different problem sizes for each application. For example, for JI we tested with different size matrices, convergence values, and initial values. For PC, we tested with different numbers of consumer processes, producer processes, and number of slots in the buffer. For PC and DP, we tested with different amounts of time spent inside and outside of critical sections. The results we report below are representative of and summarize the observed results; see [10] for complete details. To understand better where execution time was being spent, we modified the SR run-time system to report, upon program termination, the number of context switches, the total number of semaphore P operations performed, and the number of P operations that block. We also ran several micro-benchmarks to determine the basic costs of context switches and costs of P operations that block and of those that do not block. Finally, we used gprof to profile the code. 3.1
CP versus CM (Single Processor)
These tests were run on several workstations including a Sun Sparc 5 workstation running SunOS 5.5, various Intel-based PCs running various versions of Linux, a DEC 5000/240 running ULTRIX 4.3, and a DEC Alpha running OSF1 V3.2. The data presented in this section are from the tests run on the Sparc 5 or the Pentium Pro 200 running Linux 2.2.5-22. The overall patterns of results on all platforms are similar. We also focus on relative performance and so give most results in terms of the ratio of execution times multiplied by 100%, i.e., Tcm /Tcp ∗ 100%. Table 1 shows representative results for the DP, PC (1P1C1S), and RW problems. The “work” in the table indicates how much non-critical activity a process 1 2
The PC problem with more than one slot is usually called the Bounded Buffer problem. We use PC for both in this paper. CM applications are typically run on a single processor; some CP applications are run on a single processor.
A Comparison of Concurrent Programming and Cooperative Multithreading WORK 100 1000 10000 100000 1000000
DP 20.0 44.3 52.8 53.8 52.9
PC 92.7 92.9 93.1 93.3 93.1
733
RW 77.8 74.8 70.8 69.8 70.2
Table 1. Execution time ratio CM/CP (%) for three applications
performs. For example, in PC, it indicates how much relative work a producer takes to produce a new item. We expect that most practical applications would fall within the lower work categories, which incur fewer context switches and synchronization points.3 For the applications in Table 1, the CM programs perform better for two reasons: they perform no P/V operations (i.e., they synchronize using shared variables and less often) and they make fewer context switches. Of these two factors, the former is more important because it is more costly. For example, on a Sparc-5, it takes 20-30 µs per P operation versus 5 µs per context switch. These results prompted us to look further into how we could reduce costs. We observed that, in CM programs, processes were sometimes busy waiting more than necessary due to the order in which processes execute. For example, in the code for 2P2C1S, if a producer that has just deposited an item yields to the other producer, then that producer will be awakened, see that it cannot proceed, and in turn will yield. Some context switching could be eliminated if the producer instead yields to a consumer. We call this ability to select the process to which to yield a named yield and the usual yield an unnamed yield. (A named yield is like a coroutine resume.) We simulated named yield for the PC problem. The graph in Figure 2 shows the results. The results are better than those shown for PC in Table 1. The results are also better on some tests for small amounts of work (not shown in Table 1), where the execution time ratios (%) using unnamed yields ranged from 90% to 120%. We also looked at TSP, MM, and JI. For TSP and and MM, the programs written in the CM model obtain execution time ratios (%) between 97-99%, due to the small amount of synchronization in those applications. For JI, the CM model obtains execution time ratios (%) of between 98.5-100%, despite somewhat frequent barrier synchronization. For this application, the CM’s barriers — busy waiting on shared variables with yields — turned out to cost about the same as the CP’s barriers — performing P/V semaphore operations.
3
Note that “work” means iterations of some computational loop. It does not necessarily correspond directly to elapsed time because, for example, if a process is waiting a long time for I/O to complete, other processes can execute during that time.
734
Takashi Ishihara et al. 100
10p10c1s 1p1c1s 5p10c1s 5p10c5s
CM/CP (%)
95
90
85
80
75
0
10
10^2
10^5
10^4
10^3
10^6
Amount of Work (log scale)
Fig. 2. Execution time ratio CM/CP (%) for the xPyCzS Problem
3.2
CP versus PCM (Multiprocessor)
We ran further experiments on a dual 550 MHZ processor PC running Red Hat Linux 6.0. These experiments used the SR implementation (MultiSR) with support for multiprocessors. We ran the applications mentioned previously. Again, the results for TSP and MM differed little due to the small amount of synchronization in those programs. Note that the PCM versions do need to synchronize between groups of processes (more on that below). The more interesting tests were DP, PC, and JI.
16
1
15
2
14
3
13
4 West (processor 2)
East (processor 1) 5
12
11
6
10
7 9
8
Fig. 3. Layout of PCM Dining Philosophers for 16 philosophers and 2 processors
For the DP problem, we used the same CP version as before (Figure 1(a)). The PCM version, though, is new. The basic approach, illustrated in Figure 3, splits philosophers into two regions: East and West. Each of these regions is
A Comparison of Concurrent Programming and Cooperative Multithreading
735
assigned to a processor. Synchronization within a region uses shared variables, represented by dashed lines in Figure 3. Synchronization between the two regions, however, uses a semaphore, represented by solid lines in Figure 3. The code is, therefore, a hybrid of the code in the two parts of Figure 1 with three kinds of philosophers: interior philosophers 2-7 and 10-15, each of which uses shared variables to get both of its forks; borders 1 and 9, each of which uses a semaphore to get its right fork but a shared variable to get its left fork; and borders 8 and 16, each of which uses a semaphore to get its left fork but a shared variable to get its right fork. The MultiSR implementation, unfortunately, does not support processor affinity. (The same holds true for the underlying LinuxThreads on which MultiSR is built.) So, different processes in the same region (East or West) could run at the same time, which violates the PCM assumption and could lead to a race condition on the shared variables. In our simulation, therefore, we tested two versions — DP1 and DP2 — of PCM DP. Both include extra (semaphore) synchronization to protect the shared variables (“protection synchronization”). DP2 includes additional synchronization to ensure that only one process in a region runs at the same time (“region synchronization”). DP1 , on the other hand, allows more than one from the same region to run at the same time. DP1 is a reasonable, conservative approximation to how PCM DP would perform. Given the characteristics of the tests (e.g., number of philosophers), it is likely that multiple philosophers from each region can run at the same time; so the overall performance is not likely to be improved by running two philosophers from the same region at the same time. The extra, protection synchronization means the measured costs are (most likely) higher than they would be for a pure PCM DP solution. Table 2 shows some representative speed-ups for the two PCM programs compared with the CP DP program. DP1 outperforms the CP version of DP for all tested workloads, even though DP1 has extra, protection synchronization. DP2 performs considerably worse for most workloads due to its use of extra, region synchronization. WORK DP1 2 45.4 4 54.6 10 81.8 20 86.6 100 99.7 200 95.0 400 98.1 1000 95.9
DP2 57.9 83.0 126.0 139.1 178.2 171.9 172.2 167.4
test 1 2 3 4
PC 101.5 102.8 101.8 100.8
test 1 2 3 4
JI 52.6 94.8 88.4 83.8
Table 2. Execution time ratio PCM/CP (%) for three applications We tested a PCM version of the PC program. The synchronization requirements of the program led us to place all producers on one processor and all
736
Takashi Ishihara et al.
consumers on the other because producers (consumers) need to synchronize in accessing the variable that indicates which item was removed (inserted). Unfortunately, the extra, protection synchronization required to protect shared variables results in code whose semaphore structure is identical to that in the CP version and has additional synchronization to simulate the await statements. Accordingly, its performance was slightly worse than the CP version, as indicated in Table 2. The CP and PCM versions of JI each use a barrier. The difference, though, is in how the barrier is coded. In the CP version, the barrier uses semaphores shared by all processes. On the other hand, the PCM version is similar in spirit to the PCM version of DP (DP1 ). The processes are divided into two groups. Each group of processes uses shared variables to implement the barrier among the group. Two semaphores are used to synchronize between the two groups. (The actual code uses the extra, protection synchronization, as in DP1 , to prevent race conditions.) Table 2 shows some representative speed-ups for JI. As shown, the PCM version performs better, despite the extra, protection synchronization, than the CP version. The reason is its use of the simple shared variable barrier versus the more expensive semaphore based barrier. Note that the improvement for JI here is significant, whereas the improvement for JI in CM was nearly non-existent (see Section 3.1). The difference is that processes do fewer busy waits since they are now split onto two processors.
4
Discussion
Applications such as MM, TSP, and JI run on a uni-processor system will (almost always) be faster if they are written as sequential rather than concurrent programs. However, applications such as PC, DP, and RW represent servers. Those applications and others such as http or file servers lend themselves to multiple logical threads. For example, an http server multiplexes multiple connections and might have one thread for each request being serviced. These threads might be expressed as processes within a CP program or as threads within a CM program (e.g., as in the Boa server [4]). Having logical threads can also be important with respect to I/O [6]. The results we gave are based on measurements of SR programs, both standard, CP-style programs and CM- and PCM-style programs. The results might be skewed a bit in favor of the CP-style programs because SR’s underlying runtime system was designed for CP. However, having all applications written in the same language was useful: the implementation of other language features (e.g., code generated for accessing array elements) is consistent between the different versions of programs and therefore does not unfairly influence the results as it might when comparing programs written in different languages. The style in which programs are written differs between the models. The difference between CP and CM can be seen clearly in Figure 1. The difference between CP and PCM was described, for example, for the DP problem in Sec-
A Comparison of Concurrent Programming and Cooperative Multithreading
737
tion 3.2. There, the code is a hybrid of styles and the argument made regarding simplicity in [12, 13] is not as solid — the programmer needs to understand both models of execution to program in PCM. The PCM DP example also raises the issue of load balancing. As presented, the processes were split statically into two groups (regions), under the implicit assumption that, overall, each group of processes was doing about the same amount of work. If that assumption is not correct, then philosophers could be moved. Specifically, a border philosopher could be moved from one group to the other; such a change would directly affect three philosophers and their roles as border or interior. The actual code to effect such a move would be complicated, especially when the philosophers are in the midst of synchronizing. In the CP model, load balancing can happen implicitly without any change to how the processes synchronize. A potential advantage of PCM-style programs is that they might benefit from cache affinity [14]. That is, processes that use the same variables will be placed on the same processor, for example, as in the PCM DP example. Our work is related generally to other work that attempts to eliminate synchronization or replace synchronization by less expensive forms. Examples: eliminating barrier synchronization from parallel programs [7] and replacing more costly forms of message passing with less costly ones [11]. Both of those approaches employ compiler analysis, whereas the approach in this paper is aimed at the higher, language level. We are investigating how to transform CP programs into PCM or CM programs. Even if some programmers prefer to express their code within the CP model, that code can be transformed to PCM and run more efficiently by eliminating synchronization. For example, a P/V pair of semaphore operations used for mutual exclusion can simply be eliminated under certain conditions. Note that some implementations of CP languages essentially map a CP into a PCM program anyway, but they generally need to assume the worst case of when context switches will occur. It is also desirable to automatically transform programs like the CP version of DP into a PCM version. However, it is not clear how to devise general transformations that would work for many programs.
5
Conclusion
We have presented a comparison of the cooperative multithreading models (CM and PCM) with the general concurrent programming model (CP). We examined execution time performance of a range of standard concurrent programming applications. The results showed that in many, but not all, cases programs written in the CM- or PCM-style outperform those written in the CP-style. The key factor is that cooperative multithreading allows less costly synchronization to be used or even some synchronization to be eliminated. We also examined the PCM programming style. Our experience indicates that the cooperative multithreading models (CM or PCM) are viable alternatives to the general concurrent programming model (CP) and are worthy of further exploration.
738
Takashi Ishihara et al.
Acknowledgement Joel Baumert made valuable technical suggestions on this work. The anonymous reviewers provided detailed comments that helped us to improve this paper.
References [1] G.R. Andrews and R.A. Olsson. The SR Programming Language: Concurrency in Practice. Benjamin/Cummings Publishing Company, Inc., Redwood City, CA, 1993. [2] Tak Auyeung. Cooperative multithreading. Embedded Systems Programming, pages 72–77, December 1995. [3] H. E. Bal, M. F. Kaashoek, and A. S. Tanenbaum. Orca: A language for parallel programming of distributed systems. IEEE Transactions on Software Engineering, 18(3):190–205, March 1992. [4] http://www.boa.org, 1999. [5] G. Cornell and C. S. Horstmann. Core Java. Sun Microsystems, Inc., Mountain View, CA, 1996. [6] E.F. Fodor and R.A. Olsson. Cooperative multithreading: Experience with applications. In The 1999 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA ’99), pages 1953–1957, July 1999. [7] H. Han, C.-W. Tseng, and P. Keleher. Eliminating barrier synchronization for compiler-parallelized codes on software DSMs. International Journal of Parallel Programming, 25(5):591–612, October 1998. [8] C.A.R. Hoare. Communicating Sequential Processes. Communications ACM, 21(8):666–677, August 1978. [9] Intermetrics, Inc., 733 Concord Ave, Cambridge, Massachusetts 02138. The Ada 95 Annotated Reference Manual (v6.0), January 1995. ftp://sw-eng.falls-church.va.us/public/Ada- IC/standards/95lrm_rat. [10] Takashi Ishihara, Tiejun Li, Eugene F. Fodor, and Ronald A. Olsson. Cooperative multitasking versus general concurrent programming: Measuring the overhead associated with synchronization mechanisms. Unpublished Manuscript, University of California, Davis, September 1999. [11] C. M. McNamee. Transformations for optimizing interprocess communication and synchronization mechanisms. International Journal of Parallel Programming, 19(5):357–387, October 1990. [12] M. L. Scott. Language support for loosely coupled distributed programs. IEEE Transactions on Software Engineering, 13(1):88–103, January 1987. [13] M. L. Scott. The Lynx distributed programming language: Motivation, design and experience. Computer Languages, 16(3/4):209–233, 1991. [14] R. Vaswani and J. Zahorjan. The implications of cache affinity on processor scheduling for multiprogrammed, shared memory multiprocessors. In Proceedings of the Thirteenth ACM Symposium on Operating System Principles, pages 26–40, December 1991. [15] Z-World, Inc. Dynamic C 5.x Integrated C Development System Application Frameworks (Rev.1), 1998. Dynamic C 5.x.
The Multi-architecture Performance of the Parallel Functional Language GpH Philip W. Trinder1 , Hans-Wolfgang Loidl1 , Ed. Barry Jr.† , M. Kei Davis2 , Kevin Hammond3 , Ulrike Klusik4 , Simon L. Peyton Jones5 , and ´ Alvaro J. Reb´on Portillo3 1
Heriot-Watt University, Edinburgh, U.K; {trinder,hwloidl}@cee.hw.ac.uk 2 Los Alamos National Laboratory, U.S.A; [email protected] 3 University of St. Andrews, U.K; {kh,alvaro}@dcs.st-and.ac.uk 4 Philipps–University Marburg, Germany; [email protected] 5 Microsoft Research Ltd, Cambridge, U.K; [email protected]
Abstract. In principle, functional languages promise straightforward architecture-independent parallelism, because of their high level description of parallelism, dynamic management of parallelism and deterministic semantics. However, these language features come at the expense of a sophisticated compiler and/or runtime-system. The problem we address is whether such an elaborate system can deliver acceptable performance on a variety of parallel architectures. In particular we report performance measurements for the GUM runtime-system on eight parallel architectures, including massively parallel, distributed-memory, shared-memory and workstation networks.
1
Introduction
Parallel functional languages have several features that should, in theory, enable good performance on a range of platforms. They are typically only semi-explicit about parallelism, containing limited explicit control of parallel behaviour. Instead the compiler and runtime-system extract and exploit parallelism, with the programmer controlling a few key aspects of the parallelism explicitly. Purely functional languages also have deterministic parallelism: the value computed by a program is not dependent on its parallel behaviour, thereby avoiding the complications of race conditions and deadlocks. Many pure functional language implementations support dynamic resource allocation: the resources of the parallel machine are allocated during program execution. Dynamic resource allocation †
This paper is dedicated to the memory of Ed Barry Jr., who died an untimely death in May 1999.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 739–743, 2000. c Springer-Verlag Berlin Heidelberg 2000
740
Philip W. Trinder et al.
relieves the programmer from architecture dependent tasks such as specifying exactly what computations are to be executed where. The cost of high-level, dynamically-managed parallelism is a complex compiler and/or runtime-system. Can such a sophisticated system deliver acceptable performance on very different parallel architectures? We tackle this question in the context of our GUM implementation of Glasgow Parallel Haskell (GpH), a non-strict functional language. GUM performs parallel graph reduction, and many aspects of the parallelism are determined dynamically e.g. threads are dynamically created and allocated to processors. GUM is designed to be portable, and uses a message-passing model (Sect. 2). Performance is measured for a simple test program with good parallel behaviour (Sect. 3) as well as for one larger application with irregular parallelism and complex data structures (Sect. 4). This complements our earlier research on parallelising substantial Haskell applications [1] and developing a suite of simulation and profiling tools.
2
The GUM Runtime System
GUM is the runtime-system for GpH [7], a parallel variant of the Haskell lazy functional language. Being a parallel graph reduction machine [3], GUM represents an architecture-independent abstract machine-model appropriate to both shared- and distributed-memory architectures. In this model both data and program are represented via graph structures. Executing a program means rewriting a graph with its result. Semi-explicit parallelism in GpH requires the programmer to annotate expressions that can be evaluated in parallel. The runtimesystem then dynamically distributes data and work among the available processors. Potential parallelism may be subsumed [3] by existing threads in a similar way as in the lazy task creation mechanism [2], thereby dynamically increasing thread granularity. This dynamic granularity control, together with overlapping computation with communication (latency hiding), is crucial for achieving high performance on very different parallel architectures. Communication between threads is realised via a shared heap with implicit synchronisation on graph structures shared between several threads (implemented via message-passing on top of PVM or MPI). For efficient and portable compilation we use the Glasgow Haskell Compiler [4] (GHC), a state-of-the-art optimising compiler for Haskell. The design and implementation of GUM are discussed in detail in [7].
3
Measurement Setup
In our measurements we have used eight machine configurations: one MPP, a 97processor Connection Machine CM-5 with the native CMMD communications library; one DMP, a 16-processor IBM SP/2 with MPI; one SMP, a 6-processor Sun SparcServer with PVM; a 56-node Beowulf cluster with 450MHz Intel Pentium II processors, 384MB RAM and 8GB local disk; and four networks of workstation (NOWs) all with PVM. The Beowulf uses a 100Mb/s fast Ethernet switch, the NOWs use standard Ethernet on the same subnet.
The Multi-architecture Performance of GpH
741
Table 1. Single-processor efficiency of blackspots Class and Sequential Efficiency Class and Sequential Efficiency Architecture Runtime (s) Architecture Runtime (s) SMP Workstation-net Sun-SMP PVM 135.2 77% Digital Alpha PVM 378.6 63% Sun-4/15 PVM 815.4 84% Sun-10 PVM 289.1 96% Pentium PVM 109.4 96% Beowulf PVM 19.8 90%
In order to assess the overhead of the GUM runtime-system we have measured parfact, a simple binary divide-and-conquer program computing the sum of a given interval with little communication and no software bound for the achievable parallelism. The parfact program and additional details and measurements are available in [5]. The CM-5 achieves the best relative speedup of 74.1 on 97 processors, without approaching a parallelism bound imposed by either the hardware or the GUM runtime-system. These speedups are significantly better than the PVM and MPI versions, 5.82 and 6.53 on 8 processors, respectively, indicating the high costs of the portable communication libraries. Furthermore the MPI version on the IBM SP/2 suffers from a slow (sequential) startup. The Beowulf achieves a relative speedup of 33.1 on 56 processors. However, its parallel efficiency (33.1/56 = 59%) is not as good as the CM-5’s (54.0/63 = 86%).
4
Accident Blackspots: A Larger GpH Program
The Accident Blackspots program determines locations where two or more traffic accidents have occurred, based on a set of police accident records as input. A number of criteria can be used to determine whether two accident reports are for the same location, and each criteria partitions the set. The problem amounts to combining several partitions of a set into a single partition, or union find. The program comprises 1,500 lines of Haskell code and additional details are available in [1]. The parallel (GpH) version of the algorithm uses a geometric partitioning of the input data into 32 small and 8 large tiles. Evaluation strategies [6] are used to define the parallel evaluation over these tiles. The best sequential efficiency (see Table 1) is obtained for machines based on SPARC and Pentium processors, since these architectures are best supported by the GHC compiler itself. The simpler parfact program has higher efficiency in most cases [5]. In GUM sequential efficiency is dominated by the costs for managing potential parallelism (essentially adding a pointer to an array of possible tasks) and for locking graph structures representing work. The former, although cheap in itself, prohibits optimisations because it requires the graph structures to exist in the heap. The latter requires several instructions and, without the help of sharing analysis, increases with program size.
742
Philip W. Trinder et al. Blackspots (Absolute Speedup) 14
Beowulf PVM Pentium PVM Alpha PVM Sun-4 PVM Sun-SMP PVM Sun-10 PVM Linear
12
Speedup
10 8 6 4 2 0
0
5
10
15 Processors
20
25
30
Fig. 1. Absolute speedups for blackspots Fig. 1 shows that the small amount of communication required by the geometric partitioning enables good speedups even on the NOWs: 11.94 relative, 10.00 absolute on 16 Suns and 9.46 relative, 5.96 absolute on 12 Alphas. The Beowulf cluster profits from its high efficiency and its scalability. As a result it delivers the highest absolute speedup for this application: 11.39 (with larger input data 18.69) on 32 processors. Occasional drops in performance reveal that the dynamic scheduling is not always effective for this rather coarse-grained application. The overall poorer performance for the Pentium, Alpha and Sun-10 PVM NOWs is due to the higher ratio of communication costs to processor speed, exacerbated by the unusually low efficiency of the Digital Alpha. In this configuration the available parallelism is not sufficient to effectively hide communications latency. In the Beowulf cluster, with lower communications costs, this effect is less pronounced. In order to obtain good performance on machines with such characteristics we could perform architecture-dependent tuning, e.g. splitting the data into more, smaller tiles. The Sun-10 NOW exhibits a super-linear speedup for 3 processors. We believe this is due to reduced garbage collection costs in a parallel setting (with n processors we have n times the sequential heap available). The rather low speedup on the Sun SMP (2.82 relative, 2.16 absolute) is partly due to the higher overall performance of the processors and partly due to the competition with other user processes when performing the measurements. Our implementation would also profit from direct support of shared-memory — currently we use PVM or MPI even on SMPs.
5
Conclusion
In this paper we have assessed the architecture-independent performance of the GUM runtime-system for GpH by taking measurements on eight platforms,
The Multi-architecture Performance of GpH
743
ranging from MPPs to networks of workstations. From the parfact results we conclude that, for a program with little communication and no software bound on the parallelism, GUM is efficient on all platforms (at least 83%), capable of delivering acceptable speedups on all architectures, and also capable of massive parallelism (a relative speedup of 74 on a 97-processor CM-5). From the blackspots results we conclude that GUM can achieve absolute speedups (up to 18.69 on 32 processors) for symbolic programs with irregular parallelism. We consider GpH to be best suited for such symbolic applications, obtaining moderate speedups with only minimal code changes. Our measurements suggest several improvements of GUM, which are currently being examined in the new implementation of the runtime-system. The implicit work and data distribution could be improved, e.g. by refining the workstealing algorithm, by constructing clusters of data to avoid excessive communication or by supporting the migration of running threads. Furthermore, better interaction between granularity control and the generation of parallelism could be achieved via a low-watermark scheme maintaining a minimal amount of parallelism despite thread subsumption.
References 1. H-W. Loidl, P.W. Trinder, K. Hammond, S.B. Junaidu, R.G. Morgan, and S.L. Peyton Jones. Engineering Parallel Symbolic Programs in GPH. Concurrency — Practice and Experience, 11(12):701–752, Oct. 1999. Available from [8]. 2. E. Mohr, D.A. Kranz, and R.H. Halstead Jr. Lazy Task Creation: a Technique for Increasing the Granularity of Parallel Programs. IEEE Transactions on Parallel and Distributed Systems, 2(3):264–280, Jul. 1991.
Novel Models for Or-Parallel Logic Programs: A Performance Analysis V´ıtor Santos Costa1 , Ricardo Rocha2 , and Fernando Silva2 1
COPPE Systems Engineering, Federal University of Rio de Janeiro, Brazil [email protected] 2 DCC-FC & LIACC, University of Porto, Portugal {ricroc,fds}@ncc.up.pt
Abstract. One of the advantages of logic programming is the fact that it offers many sources of implicit parallelism, such as and-parallelism and or-parallelism. Arguably, or-parallel systems, such as Aurora and Muse, have been the most successful parallel logic programming systems so far. Or-parallel systems rely on techniques such as Environment Copying to address the problem that branches being explored in parallel may need to assign different bindings for the same shared variable. Recent research has led to two new binding representation approaches that also support independent and-parallelism: the Sparse Binding Array and the CopyOn-Write binding models. In this paper, we investigate whether these newer models are practical alternatives to copying for or-parallelism. We based our work on YapOr, an or-parallel copying system using the YAP Prolog engine, so that the three alternative systems share schedulers and the underlying engine.
1
Introduction
One of the advantages of logic programming (LP) is the fact that one can exploit implicit parallelism in logic programs. Implicit parallelism reduces the programmer effort required to express parallelism and to manage work. Logic programs have two major forms of implicit parallelism: or-parallelism (ORP) and andparallelism (ANDP). Given an initial query to the logic programming system, ORP results from trying several different alternatives simultaneously. In contrast, ANDP stems from dividing the work required to solve the alternative between the different processors. One particularly interesting form of ANDP is independent and-parallelism (IAP), found in divide-and-conquer problems. Arguably, ORP systems, such as Aurora [16] and Muse [2], have been the most successful parallel logic programming systems so far. One reason is the large number of logic programming applications that require search, including structured database querying, expert systems and knowledge discovery applications. Parallel search can be also useful in constraint logic programming. Two major issues must be addressed to exploit ORP. First, one must address the multiple bindings problem. This problem arises because alternatives being exploited in parallel may give different values to variables in shared branches A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 744–753, 2000. c Springer-Verlag Berlin Heidelberg 2000
Novel Models for Or-Parallel Logic Programs: A Performance Analysis
745
of the search tree. Several mechanisms have been proposed for addressing this problem [14]. Second, the ORP system itself must be able to divide work between processors. This scheduling problem is made complex by the dynamic nature of work in ORP systems. Most modern parallel LP systems, including SICStus Prolog [5], Eclipse [1], and YAP [12] use copying as a solution to the multiple bindings problem. Copying was made popular by the Muse ORP system, a system derived from an early release of SICStus Prolog. The key idea for copying is that workers maintain separate stacks, but the stacks are in shared memory. Whenever a processor, say W1 , wants to give work to another, say W2 , W1 essentially copies its own stacks to W2 . In contrast to other approaches, Muse [3] showed that copying has a low overhead over the corresponding sequential system. On the other hand, copying has a few drawbacks. First, it is expensive to exploit more than just ORP with copying, as the efficiency of copying largely depends on copying contiguous stacks, but this is difficult to guarantee in the presence of ANDP [15]. A second issue is that copying makes it more expensive to suspend branches during execution. This is a problem when implementing cuts and side-effects. Recent research in the combination of IAP and ORP has led to two new binding representation approaches: the SBA (Sparse Binding Array) [8] and the αCOWL (copy-on-write design) [9]. The SBA is an evolution of Warren’s Binding Array (BA) representation [20]. In BA systems, the stacks form a cactus-tree representing the search-tree, and processors expand tips of this tree. Workers thus use a shared pool of memory except when storing bindings that are private to a worker. These are stored in a local data-structure, the Binding Array. The αCOWL scheme uses a copy-on-write mechanism to do lazy copying. Both of these approaches elegantly support IAP and ORP. The question remains of how these systems fare against copying for ORP, in order to verify whether they are indeed practical alternatives to copying. To address this question, we experimented with YapOr, an ORP copying system using the YAP engine [18], and we implemented the SBA and the αCOWL over the original system. The three alternative systems share schedulers and the underlying engine: they do only differ in their binding scheme. We then used a set of well known ORP all-solutions benchmarks to evaluate how they perform comparatively. The paper is organised as follows. We first review in more detail the three models. Next, we discuss their implementation. We then present and discuss experimental results. Last, we make some concluding remarks.
2
Models for Or-Parallelism
A goal in our research is to develop a system capable of exploring implicitly all forms of parallelism in Prolog programs. A key point to achieve such a goal is to determine a binding model that simplifies the exploitation of the combined forms of parallelism. In this paper we concentrate in three binding models: environment copying, sparse binding arrays and copy-on-write. We assume a multisequencial
746
V´ıtor Santos Costa, Ricardo Rocha, and Fernando Silva
system, where the computational agents, or workers, do not initially inform the system that they have created new alternatives, and thus have exclusive access to these alternatives. This is called private work. At some point these alternatives may be made available to other workers, and we say they become public work. The Environment copying model was introduced by Ali and Karlson in the Muse system [2]. In this model each computing agent (or worker) maintains a separate environment, almost as in sequential Prolog, in which the bindings it makes are independently recorded, hence solving the multiple bindings problem. When a worker becomes idle (that is, when it has no work), it searches for a busy worker from whom to request work. Sharing work among workers thus involves the actual copying of the computation state (WAM stacks) from the busy worker to the requester. After copying both workers have exactly the same state, and will diverge by executing alternative branches at the choice-point where the work sharing took place. Efficient implementations of copying depend on Incremental copying to reduce the overheads of copying. With this technique, one just copies the parts of the execution stacks that are different among the workers involved. The scheduler plays an important role here by guiding idle workers to request work from the nearest busy workers. Bottom-most scheduling strategies have been very successful with this binding model, because they increase the number of choice-points shared between workers, thus preventing unnecessary copying. The Copy-On-Write model, or αCOWL, was proposed by Santos Costa [9] towards supporting and/or parallelism. In the αCOWL, similarly to environment copying, each worker maintains a separate environment. Moreover, whenever a worker wants to share work from a different worker, it also logically copies all execution stacks. The insight here is that although stacks will be logically copied, they will be physically copied only on demand. To do so, the αCOWL applies the Copy-On-Write mechanism provided by most modern Operating Systems. The αCOWL has two major advantages. First, we can copy anything. We can copy standard Prolog stacks, the store of a constraint solver, or a set of stacks for ANDP. Indeed, we might not even have a Prolog system at all. Second, because copying is done on demand, we do not need to worry about the overheads of copying large, non-contiguous, stacks. This is an important advantage for ANDP computations. The main drawback of the αCOWL is that the actual setting up of the COW mechanism can be itself quite expensive, and in fact, more expensive than just copying the stacks. In the next sections we discuss an implementation and its performance results. The Sparse Binding Array (SBA) derives from Warren’s Binding Arrays. Binding arrays were originally proposed for the SRI model [20]. In this model execution stacks are distributed over a shared address space, forming the socalled cactus-stack. In more detail, workers expand the stacks in the parts of the shared space they own, whilst they also can access stacks originally created by other workers. Note that the major source of updates to public and private work are bindings of variables. Bindings to the public part of the tree are tentative, and in fact different alternatives of the search tree may give different values,
Novel Models for Or-Parallel Logic Programs: A Performance Analysis
747
or even no value, to the same variable. These bindings are called conditional bindings, and WAM-based systems will also stored them in the Trail data-area, so that they can later be undone. Conditional bindings cannot be stored in the shared tree. Instead, in the original BA scheme workers use a private array data structure associated with each computing agent to record conditional bindings. The Sparse or Shadow Binding Array (SBA) [8] is a simplification of the BA designed to handle IAP. In the SBA each worker has a private virtual address space that fully shadows the system’s shared address space. In other words, every work has its own shadow of the whole shared stacks. This “shadow” will be used to store to shared variables. Thus, the execution data structures and unconditional bindings are still stored in the shared address space, only conditional bindings are stored in the shadow area. The SBA thus preempts the problem of managing a BA in the presence of IAP [13], at the cost of having to allocate much more virtual space than the BA. A further optimisation in the SBA is that each SBA is mapped at the same fixed location for each worker in the system. We thus can maintain pointers from the shared stacks to the SBA.
3
Implementation Issues
The literature includes several comparisons of copying-based versus BA-based systems, and particularly of Aurora vs. Muse [4, 7]. One problem with these studies is that Aurora and Muse have very different implementations: it is quite difficult to know whether the differences stem from the model or from the actual implementation. In contrast, we experimented with the three models by implementing them over the same YapOr system [18]. The system is derived from the Yap engine [10]. This is one of the fast emulator-based Prolog systems currently available, and only 2 to 3 times slower than systems that generate native-code. We would expect Yap to be between 2 to 4 times faster than the sequential Aurora engine on the same hardware. 3.1
YapOr with Copying
The YapOr system was originally designed to implement copying. The system is based on the Yap engine. The main changes were required on the instructions that manipulate choice-points. Other changes are in the initialisation code for memory allocation and worker creation, some small changes in the compiler to provide extra information for managing ORP, and lastly a change designed to support built-in synchronisation. In a nutshell, the adapted engine communicates with YapOr through a fixed set of interface functions and through two special instructions. The functions are entered when choice-points are activated, updated, or removed. The two instructions are activated whenever a worker backtracks to the shared part of the tree and they call the scheduler to do work search. One instruction processes
748
V´ıtor Santos Costa, Ricardo Rocha, and Fernando Silva
parallel choice-points, and the other sequential choice-points (that is, choicepoints such that their alternatives must be explored in sequential order). The scheduler is the major component of YapOr. Work is represented as a set of or-frames in a special shared area. Idle workers consult this area and the GLOBAL and LOCAL data structures, which contains data on work and the status of each worker, until they find work. If there is no work in the shared tree, idle workers try to share work with a busy worker. This sharing is implemented by two model dependent functions: p share work(), for the busy worker, and q share work(), for the idle one. After sharing the previously idle worker will backtrack to a newly shared choice-point, whereas the previously busy worker continues execution from the same point. Note that before sharing, workers will try to move up in the tree to simplify incremental copying. In copying, sharing is implemented by the following algorithm: Busy Worker P __________________________ Compute stacks to copy . Share private nodes . . . Help Q in copy ? . . Wait copy_done signal Back to Prolog execution . Backtrack to shared node ? Wait ready signal
Signals Idle Worker Q ________________ __________________________ Wait sharing signal ----sharing----> . Copy trail ? Copy heap ? Wait nodes_shared signal --nodes_shared-> . Copy local stack ? <---copy_done--- . ---copy_done---> . Wait copy_done signal Install conditionals <-----ready----- . Fail to top shared node .
Initially, the idle worker waits for a sharing signal while the busy worker computes the stacks to copy. Next, the busy worker prepares its private nodes for sharing whilst the idle worker performs incremental copying. The busy worker may help in the copying process to speed it up. The two workers then synchronise to determine the end of copying. At the end, the busy worker goes back to Prolog execution and the idle worker installs the conditional bindings from the busy worker that correspond to variables in the maintained part of the stacks. To guarantee the correctness of the installation step, the busy worker can not backtrack to a shared node until the idle worker do not completes installation. 3.2
αCOWL
To support sharing of work in the αCOWL we had to change p share work() and q share work(). We applied the main function where Unix-style Operating Systems implement COW: the fork() function. The idea is that whenever a worker P accepts a work request from another worker Q, worker P forks a child process that will assume the identity of worker Q, whilst the older process executing Q exits. At this point, the new process Q has the same state as that of P. The process Q then is forced to backtrack to the same choice-point. Note that
Novel Models for Or-Parallel Logic Programs: A Performance Analysis
749
scheduling is realised in exactly the same way as for the environment copying model, that is through the use of a public tree of or-frames in shared space. The synchronisation algorithm for sharing work in αCOWL is as follows: Busy Worker P Signals Idle Worker Q __________________________ ________________ __________________________ . Wait sharing signal . ----sharing----> . fork() exit() . Child takes Q’s id Back to Prolog execution Fail to top shared node
Note that fork() is a rather expensive operation. For programs which have parallelism of high granularity, one expects that the workers will be busy most of the time and the number of sharing operations be small. In this case the model is expected to be efficient. On the other hand, we would expect worse results for fine-grained applications. Note that one could use the mmap() primitive as an alternative to fork(), but we felt fork() provided the most elegant solution. 3.3
Sparse Binding Arrays
Supporting the SBA requires changes to both the engine and the sharing mechanism. The main changes to the engine affect pointer comparison, and variable and binding representation. As regards pointer comparison, the Yap system assumes pointers in the stacks follow a well-defined ordering: the local stack is above the global stack, the local stack grows downwards, and the global stack grows upwards. These invariants allows one to easily calculate variable age and are useful for trailing and recovering space. Unfortunately, they are not valid in the SBA, as the cactus stack is fragmented. Aurora uses the BA offset as a means for calculating age, but has to pay the overhead of maintaining an extra counter. The Aurora/SBA implementation used an age counter that records the number of choice-points above the current choice-point [8]. The YAP SBA implementation does not maintain such counters, and instead follows the rule: 1. the sequential invariant is guaranteed to hold for private data; 2. shared data in the cactus-stack are protected as regards recovering space, and age follows the simple rule: smaller is older. To implement this rule, each worker manages the so-called frozen registers that separate its private from the shared parts of the tree. Moreover, an extra register, BB, replaces the WAM’s B register when detecting whether a binding is conditional. Note that these same problems must be addressed to support IAP. The second issue we had to address is variable representation. In the original WAM a variable is represented as a pointer to itself. This is unfortunate, because we would need to initialise the whole of the BA. BA based systems (with the exception of Andorra-I [11]) thus assume unbound variables are ultimately null cells. In Aurora, a new variable is initialised as a tagged pointer to the BA, itself null. In the SBA we do not need pointers to the BA, as it is sufficient to
750
V´ıtor Santos Costa, Ricardo Rocha, and Fernando Silva
calculate the offset we are at in the shared space, and add it to the SBA base. Aurora/SBA thus initialises a new variable as a tagged age field. We decided to optimise for the sequential overhead in the YapOr/SBA implementation. To do so, a new cell is initialised as a null field. Moreover, and in contrast to previous BA-based systems, conditional bindings will only be moved to the SBA when they are made public, and only then. This means that private execution in our scheme will not use the SBA at all. As bindings are made public they will be copied to the SBA. Moreover, the original cell will be made to point to the SBA. Thus the variable dereferencing mechanism is unaware of the existence of the SBA. Note that the pointer that is placed in the original cell is independent of workers, although it points at a private structure. The changes to the engine are therefore quite extensive. As regards the changes to p share work() and q share work(), the new algorithm is as follows: Busy Worker P Signals Idle Worker Q __________________________ ________________ __________________________ Compute stacks to share Wait sharing signal Share private nodes . . ----sharing----> . . Install conditionals Back to Prolog execution Fail to top shared node
4
Performance Evaluation
In order to compare the performance of these three models we experimented the 3 systems in two parallel architectures: a Sun SparcCenter2000 with 8 CPUs and 256MB of memory, running Solaris2.7, and a PC server with 4 PentiumPro CPUs/200MHz/256KB caches and 128MB of memory, running Linux2.2.5 from standard RedHat6.0. Each CPU in the PC server is about 4 times as fast as each CPU in the SparcCenter. All systems used the same compilation flags. We used a standard set of all-solutions benchmarks, widely used to compare ORP logic-programming systems [19]. We preferred all-solutions benchmarks because they are not susceptible to speculative execution, and our goal was to compare the models. The benchmarks include the n-queens problem, the puzzle and cubes problems from Evan Tick’s book, an hamiltonian graph problem and a na¨ıve sorting resolution. Table 1 shows the execution time, in seconds, for Yap Prolog and the overhead, in percentage over Yap Prolog, introduced by each or-parallel model when executing with one worker. The overhead for copying (YapOr) confirmed previous results, and is of the order of 2% or 13% on PC/Linux and Sparc/Solaris, respectively. The overhead obtained for αCOWL is equivalent to copying. The results are consistent, and the variations are quite above the noise in our measures. We expected performance to be about the same, as for a single processor we execute quite the same code: the systems only differ in their scheduling code, and this is never activated. The overhead for SBA is, as expected, higher but not very much so, only of 15% or 32%. We believe this good result stems from the optimisations discussed
Novel Models for Or-Parallel Logic Programs: A Performance Analysis Yap Prolog YapOr Programs PC Sparc PC Sparc cubes5 0.216 0.753 2% 13% cubes7 2.505 9.042 1% 7% ham 0.435 1.537 4% 20% nsort 34.810 142.161 1% 14% puzzle 2.145 8.411 2% 22% queens10 0.703 2.809 3% 8% queens12 20.921 84.600 2% 4% Average 2% 13%
αCOWL PC Sparc 4% 10% 3% 5% 4% 24% 1% 21% 2% 18% 4% 7% 3% 4% 3% 13%
751
SBA PC Sparc 9% 21% 7% 17% 25% 57% 22% 55% 22% 39% 12% 17% 10% 15% 15% 32%
Table 1. Overheads Yap Prolog/Or-Parallel Models with one worker. in the previous section. In fact, SBA vs. YapOr performs relatively better than Aurora vs. Muse. We believe this result supports continuing research on the SBA. Table 2 shows speedups for the PC Server, and Table 3 for the SparcCenter. We use Copy for copying and COW for the αCOWL. The results show that the best speedups are obtained with copying. The SBA follows quite closely, but the speedups are not as good for higher number of workers. We believe this is partly a problem with the SBA optimisations. As work becomes more finegrained, more bindings need to be stored in the Binding Array. Execution thus slows down as the system needs to follow longer memory references and touches more cache-lines and pages.
2 workers Programs cubes5 cubes7 ham nsort puzzle queens10 queens12 Average
3 workers
4 workers
Copy COW SBA Copy COW SBA Copy COW SBA
1.99 1.99 1.97 2.03 1.97 1.99 2.00 1.99
1.88 1.98 1.90 2.00 1.93 1.88 1.99 1.94
1.98 1.99 1.99 1.96 1.95 1.99 1.99 1.98
2.97 2.99 2.93 3.06 2.96 2.96 3.00 2.98
2.60 2.92 2.64 2.98 2.41 2.12 2.86 2.65
2.97 2.99 2.97 2.93 2.92 2.97 2.99 2.96
3.95 3.99 3.82 4.08 3.94 3.92 4.00 3.96
2.69 3.83 2.79 3.90 3.20 2.42 3.82 3.24
3.98 3.98 3.87 3.90 3.88 3.92 3.98 3.93
Table 2. Speedups for the three models on the PC Server. The results for the αCOWL are quite good, considering the very simple approach we use to share work. The αCOWL performs well for smaller number of processors and for coarse-grained applications. As granularity decreases the overhead of the fork() operation becomes more costly, and in general system performance decreases versus other systems. As implemented, the αCOWL is therefore of interest for parallel workstations or for applications with large running times, which are indeed the ultimate goal for our work.
752
V´ıtor Santos Costa, Ricardo Rocha, and Fernando Silva 2 workers
Programs cubes5 cubes7 ham nsort puzzle queens10 queens12 Average
4 workers
6 workers
8 workers
Copy COW SBA Copy COW SBA Copy COW SBA Copy COW SBA
2.00 2.01 1.98 1.94 2.02 2.03 2.01 2.00
1.83 1.97 1.78 1.97 1.92 1.85 1.95 1.90
1.96 1.99 1.90 2.02 1.91 1.93 1.97 1.95
3.95 3.98 3.79 3.83 3.94 3.97 4.01 3.92
2.72 3.79 2.54 3.77 3.08 2.36 3.74 3.14
3.70 3.87 3.98 4.01 3.64 3.82 3.92 3.85
5.79 5.97 5.57 5.69 5.91 5.83 5.97 5.82
2.87 5.03 3.00 5.42 3.79 2.73 5.13 4.00
4.88 5.53 5.33 5.88 5.12 5.48 5.89 5.44
7.35 7.74 6.97 7.42 7.68 7.47 7.77 7.49
2.32 5.87 2.15 6.03 3.79 2.43 5.81 4.06
6.02 7.40 7.29 7.77 7.08 6.95 7.66 7.14
Table 3. Speedups for the three models on the SparcCenter.
5
Conclusions
We have discussed the performance of 3 models for the exploitation of ORP in logic programs. Our results show that copying has a somewhat better performance for all-solution search problems. The results confirm the relatively low overheads of copying for ORP systems. Our results confirm that the SBA is a valid alternative to copying. Although the SBA is slightly slower than copying and cannot achieve as good speedups, it is an interesting alternative for the applications where copying does not work so well. As an example we are using the SBA to implement IAP. Our implementation of the αCOWL shows good base performance, but suffers heavily as parallelism becomes more fine-grained. Still, we see the αCOWL as a valid alternative for the good reason that the applications that interest us the most have very good parallelism. The αCOWL has two interesting advantages for such applications: it facilitates support of extensions to Prolog, such as sophisticated constraint systems, and it largely simplifies the implementation of garbage collection, that in this model can be performed independently by each worker. The next major challenge for the αCOWL will be the support of suspension, required for single-solution applications. We would like to perform low-level simulation in order to better quantify how the memory footprints and miss-rates differs between models. Work on these models is progressing apace. We are working on better application support for constraint and inductive logic programming systems. Moreover, we are using copying as the basis for parallelising tabling [17], useful say for model-checking, and the SBA as the basis for IAP [6], that has been used in natural language applications. Acknowledgments The authors would like to acknowledge and thank the contribution and support from Eduardo Correia. The work has also benefitted from discussions with Lu´ıs Fernando Castro, Inˆes de Castro Dutra, Kish Shen, Gopal Gupta, and Enrico
Novel Models for Or-Parallel Logic Programs: A Performance Analysis
753
Pontelli. Our work has been partly supported by Funda¸c˜ao da Ciˆencia e Tecnologia and JNICT under the project Dolphin (PRAXIS/2/2.1/TIT/1577/95) and by Brazil’s CNPq under the NSF-CNPq project CLoP n .
References [1] A. Aggoun and et. al. ECLiPSe 3.5 User Manual. ECRC, December 1995. [2] K. A. M. Ali and R. Karlsson. The Muse Approach to OR-Parallel Prolog. International Journal of Parallel Programming, 19(2):129–162, April 1990. [3] K. A. M. Ali and R. Karlsson. The Muse Or-parallel Prolog Model and its Performance. In NACLP’90, pages 757–776. MIT Press, October 1990. [4] A. Beaumont, S. M. Raman, P. Szeredi, and D. H. D. Warren. Flexible Scheduling of OR-Parallelism in Aurora: The Bristol Scheduler. In PARLE’91, volume 2, pages 403–420. Springer Verlag, June 1991. [5] M. Carlsson and J. Widen. SICStus Prolog User’s Manual. SICS Research Report R88007B, Swedish Institute of Computer Science, October 1988. [6] L. F. Castro, V. S. Costa, C. Geyer, F. Silva, P. Kayser, and M. E. Correia. DAOS: Distributed And-Or in Scalable Systems. In EuroPar’99. Springer-Verlag, LNCS, August 1999. [7] M. E. Correia, F. Silva, and V. S. Costa. Aurora vs. Muse; A Performance Study of Two Or-Parallel Prolog Systems. Computing Systems in Engineering, 6(4/5):345– 349, 1995. [8] M. E. Correia, F. Silva, and V. S. Costa. The SBA: Exploiting Orthogonality in AND-OR Parallel Systems. In ILPS’97, pages 117–131. The MIT Press, 1997. [9] V. S. Costa. COWL: Copy-On-Write for Logic Programs. In IPPS’99. IEEE Press, May 1999. [10] V. S. Costa. Optimising Bytecode Emulation for Prolog. In PPDP’99. SpringerVerlag, LNCS, September 1999. [11] V. S. Costa, D. H. D. Warren, and R. Yang. The Andorra-I Engine: A Parallel Implementation of the Basic Andorra Model. In ICLP’91, 1991. [12] L. Damas, V. S. Costa, R. Reis, and R. Azevedo. YAP User’s Guide and Reference Manual, 1998. http://www.ncc.up.pt/~vsc/Yap. [13] G. Gupta and V. S. Costa. And-Or Parallelism in Full Prolog with Paged Binding Arrays. In PARLE’92, pages 617–632. Springer-Verlag, LNCS 605, June 1992. [14] G. Gupta and B. Jayaraman. Analysis of or-parallel execution models. ACM TOPLAS, 15(4):659–680, 1993. [15] Gopal Gupta, M. Hermenegildo, E. Pontelli, and V´ıtor Santos Costa. ACE: And/Or-parallel Copying-based Execution of Logic Programs. In Proc. ICLP’94, pages 93–109. MIT Press, 1994. [16] E. Lusk and et. al. The Aurora Or-parallel Prolog System. New Generation Computing, 7(2,3):243–271, 1990. [17] R. Rocha, F. Silva, and V. S. Costa. Or-Parallelism within Tabling. In PADL’99, pages 137–151. Springer-Verlag, LNCS 1551, January 1999. [18] R. Rocha, F. Silva, and V. S. Costa. YapOr: an Or-Parallel Prolog System based on Environment Copying. In EPIA’99, pages 178–192. Springer-Verlag, LNAI 1695, September 1999. [19] P. Szeredi. Performance Analysis of the Aurora Or-parallel Prolog System. In NACLP’89, pages 713–732. MIT Press, October 1989. [20] D. H. D. Warren. The SRI Model for Or-Parallel Execution of Prolog—Abstract Design and Implementation Issues. In SLP’87, pages 92–102, 1987.
Executable Specification Language for Parallel Symbolic Computation Alexander B. Godlevsky and Ladislav Hluch´ y Institute of Informatics, Slovak Academy of Sciences D´ ubravsk´ a cesta 9, 842 37 Bratislava, Slovakia {godlevsky.ui,hluchy.ui}@savba.sk
1
Introduction
Two goals, simplicity of program designing, and efficiency of its computation, always remain topical in programming, and more than anything this is true about parallel programming systems. The former goal is usually achieved for declarative programming languages, the latter - by embedding of coordination level operators. One of the earliest such extensions, future annotation, isa proposed in [3]. Their use allows to start a function computation before the moment when computation of its annotated arguments will be completed. Another advance to increase program parallelization was using of nondeterministic operators in pseudo-functional languages [5]. One more resource widely used in logic programming to program parallelization is speculative computation of alternative branches. In this paper we propose SL specification language that combines all of these features: 1) future annotating; 2) speculative annotating to point out conditional operators whose alternatives can be started to compute before the condition value will be determined; 3) sets as data structures and a nondeterministic choice operator with erratic choice semantics [6]. Such nondeterminism allows to choose elements from a set even if there are other set’s elements that are in progress. Another pecularity of our approach is SL program transformations during compile time stage illustrated below by an example.
2
SL Language, Its Sequential and Parallel Semantics
The syntax of SL language is presented below in a normalized let-form. Symbols ? and ! are used for future and speculative annotations, unite operation for sets union, scar(A) for nondeterministic choice element from A set, and scdr(A) defines the complement to scar(A) in A set. Though the language is not a functional one, we use the λ symbol to connect variables, similar to λ-calculi. T ::= x | (let (x V ) T ) | (let (x (? T )) T ) | (let (x (car y)) T ) | (let (x (cdr y)) T ) | (let (x (scar y)) T ) | (let (x (scdr y)) T ) | (let (x (unite V V )) T ) | (let (x (if T T T )) T ) | (let (x (if (! T ) T T ))) T ) | (let (x (apply y z)) T )
This work was supported by the Slovak Scientific Grant Agency within Research Project No.2/7186/20
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 754–757, 2000. c Springer-Verlag Berlin Heidelberg 2000
Executable Specification Language for Parallel Symbolic Computation V ∈ V alue ::= c | x | ( λx.T ) | (cons x y) | S c ∈ Const ::= true, false, ∅ . . .
755
x ∈ V ars ::= x, y, z, . . . S ∈ Set ::= | {V } | (∪ S S )
The sequential semantics of SL is the relation evalseq which corresponds a set of permissible results to a closed program where the result is either a closed value whose all λ-expressions are replaced by procedures, or error indicating that an operation was misapplied, or ⊥ if program is not terminated. This semantics is call-by-value, and it is defined by sequential SSL-machine which cannot be decribed here because of lack space. Transition rules for ? and ! annotations define them as identity operations requiring that the expression which they are applied to will be reduced first to a value and only then the expression’s name will be replaced by the value. All transition rules correspond to common knowledge of the language constructors. The parallel semantics is defined by parallel PSL-machine whose transition rules are partially presented in Fig.1. With reference to the SSL-machine, its state space uses an additional class of values - placeholder variables. Likewise to functional language with future annotations, these variables represent computation results still undeterminated. When one of parallel processes requires the value of an expression and this value is a placeholder corresponding to computation in another process, then the former process will be suspended. When the latter process is terminated, the placeholder will be replaced by the value computed and computation of the former process can go on. Specification Q ∈ Statep ::= T | error | (f-let (p Q) Q) | (if-let (p (p Q Q)) Q) (States) (Placeholders) p ∈ Ph-Vars ::= {p1 , p2 , p3 , . . . } Transition Rules E[ let (x (if !(T0 ) T1 T2 )) T ] →par (if-let T0 (let (x T1 ) E(T )) (let (x T2 ) E(T ))) (spec) ∃p. p ∈ FP(T0 ) (if-let T0 Q1 Q2 ) →par (if-let T1 Q3 Q4 ) if T0 →par T1 , (par spec) Q1 →par Q3 , Q2 →par Q4 Q1 if V = true error if V = error (end spec) (if-let (V Q1 Q2 )) →par if V =true, p Q2 E[ let (x (? T1 )) T ] →par (f-let (p T1 ) E[ T [x ← p] ]), p ∈FP(T ) (future) (join) (f-let (p V ) Q) →par Q[p ← V ] (error) (f-let (p error) Q) →par error (f-let (p1 Q1 ) Q2 ) →par (f-let (p1 Q3 ) Q4 ) if Q1 →par Q3 , (par future) Q2 →par Q4 (lift) (f-let (p2 f-let (p1 Q1 ) Q2 )) Q3 ) →par (f-let (p1 Q1 )(f-let (p2 Q2 ) Q3 )) p1 ∈FP(Q3 )
Fig. 1. Nondeterministic parallel PSL-machine
f-let and if-let are states of PSL-machine added to the SSL-machine states to support its parallel computation. Q1 from (f-let (p Q1 ) Q2 ), the right hand side expression of f uture rule, is the body of ?-expression, and Q2 is its evaluation context, p placeholder represents the result of Q1 computation in Q2 . FP function returns the set of all placeholders of the expression to which it is applied. According to sequential semantics of SL the computation of Q1 is mandatory, and Q2 computation is speculative since the result of the latter
756
Alexander B. Godlevsky and Ladislav Hluch´ y
can be unnecessary if the former remained nonterminated or its result is error. (if-let T0 Q1 Q2 ) state is generated by (spec) rule for speculative computation along both if-operator branches. Q1 and Q2 expressions represent then- and elsebranches, T0 corresponds to the operator’s condition that can be computed in parallel with Q1 , Q2 expressions, but the only mandatory computation among them is running for the Q0 condition. if-let state will be reduced according to (end spec) rule after Q0 will be replaced by a proper value. (spec) and (f uture) rules initiate new parallel computations, (par spec) and (par f uture) run them, (lif t) rule restructures of nested parallel states to increase the level of parallelism. We define the correctness of parallel semantics with respect to sequential one as each result possible for PSL-machine is also possible for SL-machine, that is as the inclusion evalpar ⊆ evalseq . Its proof is based on comparative behaviours of SSL- and PSL-machines.
3
Compile-Time Transformations
Together with the language and its semantics declaration, the key element of the paper are program transformations for compile-time stage. They allows modification of original equations with reduction of some intermediately generated speculative branches to a new compact form. For instance, let us consider an SL-program Gr(A, B) that (being without its annotations) defines the outer recursion of Buchberger’s algorithm [1] of polynomial bases constructing. The condition to stop computation of Gr is Gr(∅, B) = B and its recursion is defined as follows Gr(A, B) = (let f ?Red(scar(A),B)) if !(f =0) then Gr(scdr(A), B) else Gr(unite(?Create(f, B), F ilter(!(Divf ), scdr(A))), unite({f }, B)). According to (f uture) and (spec) rules of PSL-machine the right-hand side of this recursive equation can be transformed to (f-let ph1 Red(scar(A),B)) (if-let (ph1 =0) [then] Gr(scdr(A), B) [else] Gr(unite(?Create(ph1 , B), F ilter(!(Div ph1 ), scdr(A))), unite({ph1 },B))). Here then and else are comments for branches which can be computed in parallel. Our trick is to unify these branches using some identities and redefining (generating) some equations. In this case we use the identities unite(∅, A)=A and F ilter(true,A)=A, and also generate the operators Create1(x, y) = ( if x=0 then ∅ else Create(x, y) ), zero(x) = ( if x=0 then ∅ else {x} ). After such unification, both branches are equal to Gr(unite(?Create1(ph1 , B), F ilter(!(Div ph1 ), scdr(A))), unite(?zero(ph1 ), B)). Now, eliminating the if-let branching and applying (f uture) rule for ? annotation, the right-hand side of the Gr equation can be transformed as follows:
Executable Specification Language for Parallel Symbolic Computation
757
(f-let ph1 Red(...))(f-let ph2 Create1(ph1 , B))(f-let ph3 zero(ph1 )) Gr(unite(ph2 , F ilter(!(Div ph1 ), scdr(A))), unite(ph3 , B)). To apply this equation on-line, it is necessary to replace the fixed names ph1 , ph2 , ph3 by calls of a generator of such new names. When the first parameter of Gr equation is not a set but an exrpession similar to unite(ph2 , F ilter(!(Div ph1 ),A)), the modification of this equation for next applications are more complicated. In this case we use the identities scar(unite(ph, A)) = scar(A), scdr(unite(ph, A)) = unite(ph, scdr(A)) and unfolding for F ilter operator. However, here we are limited to demonstrate these transformations.
4
Conclusion and Future Works
We consider annotations as a tool to coordinate computation in declarative programs. Nondeterministic choice operators, on the one hand, allow to balance the tendence to advance parallel computation even when some other processes are in progress, and, on the other hand, the tendence to be close to some optimal heuristics when choosing elements from sets. One of the key elements of our approach are compile-time program transformations. These transformations are similar to ones from the theory of partial evaluations [4] in the sense that we consider placeholder variables as dynamical ones. Then, we use identities a part of which contain high-order functions (filter, f-let,...), and here we have elements close to program parallelization used in the skeleton theory [2]. Finally, some of our transformations (for example, unfolding) have to be applied in a lazy manner, and here we are interested in a parallelization experience of programs written in lazy functional languages [7].
References 1. B.Buchberger. An Algorithm for Finding a Basis for the Residue Class Ring of Zero-Dimensional Polynomial Ideal. Ph.D.Thesis, Math.Inst., Univ. of Innsbruck, Austria, 1965. 2. M.Cole. Algorithmic skeletons: structured Management of Parallel of Parallel Computation. MIT Press, Cambridge, Mass., 1989. 3. R. Halstead. Multilisp: A language for concurrent symbolic computation, ACM Transaction on Programming Languages and Systems, 7, 4 (1985), pp.501-538 4. N.D. Jones, C.K. Gomard, and P. Sestoft, Partial Evaluation and Automatic Program Generation, Prentice Hall International, June 1993. xii + 415 pages. 5. Wolfgang Schreiner: A Para-Functional Programming Interface for a Parallel Computer Algebra Package, J. Symbolic Computation (1996), 21, No 4, pp.593-614 6. H. Sondergard and P. Sestoft.Non-determinism in functional languages, The Computer Journal, 35(5): 514–523, 1992. 7. Trinder , P., Hammond, K., Loidl, H.-W., Peyton Jones, S. Algorithm + Strategy = Parallelism, Journal of Functional Programming, 8(1): 23-60, Jan. 1998.
Efficient Parallelisation of Recursive Problems Using Constructive Recursion Magne Haveraaen Institutt for Informatikk, Universitetet i Bergen, HiB, N-5020 Bergen, Norway http://www.ii.uib.no/∼magne
Abstract. Many problems from science and engineering may be nicely formulated recursively. As a problem solving technique this is very efficient. Due to the high cost of executing recursively formulated programs, typically an exponential growth in time complexity, this is rarely done in practice. Constructive recursion promises to deliver a space and time optimal compilation of recursive programs. The constructive recursive scheme also straight forwardly generates parallel code on high performance computing architectures.
1
Introduction
Recursive formulation of algorithms has for decades been considered an efficient problem solving method in computer science. The high run-time costs for most recursive programs has been hindering its general applicability. The standard operational interpretation of a recursive program forms a tree of recursive calls spreading out from the node of the initial invocation, i.e., the root node, and this is where the result of the computation is gathered. The shape of this tree is defined by the data dependency pattern of the recursive calls. The tree often has many nodes with identical subtrees. The efficiency of the computation, and its parallelisation, crucially depends on identical subtrees being merged into one, thus forming a compact data dependency graph. Also, the traversal order of this graph is important: if we trace the data dependency graph from the root, we need to instantiate all the nodes of the data dependency graph before any computation takes place. If we instead proceed in a data-driven fashion, starting at the leaves, we will only instantiate nodes when the computation reaches them. A node may be deleted as soon as the execution has passed on the value computed at it to the next node. Of the three computational models, operational (Turing machines, imperative programs), syntactic (general grammars, term rewriting, λ-calculus and related functional programming), and semantic (µ-recursion), the semantic model readily generalises to recursion on data dependency graphs (from its traditional
This investigation has been carried out with the support of the European Union, ESPRIT project 21871 SAGA (Scientific Computing and Algebraic Abstractions) and a grant of computing resources from the Norwegian Supercomputer Committee.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 758–761, 2000. c Springer-Verlag Berlin Heidelberg 2000
Efficient Parallelisation of Recursive Problems Using Constructive Recursion
759
recursion on natural numbers). This generalisation is called constructive recursion [3, 2]. It includes the definition of the data dependency graph as an explicit component of a recursive program. A compiler may then use this information to create data-driven code distributed on the processors of a parallel machine. We have developed a programming language, Sapphire, embodying these ideas. A Sapphire program consists of three parts: the definition of the data dependency, the definition of the recursive function, and the initiation of the computation. The next section sketches the principles of constructive recursion and associated compilation technique. Section 3 presents an example with timings. The conclusion gives some references to related work.
2
Constructive Recursion
Somewhat simplistically, a recursively defined function f can be abstracted to the form fi1 ,...,im = φ(fδ1 (i1 ,...,im ) , . . . , fδk (i1 ,...,im ) ), where the tuple (i1 , . . . , im ) of variables range over some subset D ⊆ N m , the index domain, of the m-tuple of natural numbers. The functions δ : D → N m define the dependency pattern or stencil of the recursion1 . The recursion is well defined if the δ define a well-funded partial order on the m-tuple of natural numbers N m , i.e., all paths generated by the δ are finite, and there is a set of initial values i1 ,...,im for every index-tuple (i1 , . . . , im ) ∈ (N m \ D). To get a constructive recursive definition of f , we also need to know the opposite of the stencil δ . The opposite does not mean the inverse of each δ . Instead we need a set of functions δ for ∈ {1, 2, . . . , k }, such that for every (i1 , . . . , im ) ∈ N m for which there exists a δ and (i1 , . . . , im ) such that δ (i1 , . . . , im ) = (i1 , . . . , im ), then there exists a δ such that δ (i1 , . . . , im ) = (i1 , . . . , im ). (The number k of opposite functions may be larger or smaller than k.) Knowing the stencil and its opposite makes us able to trace out the compact data dependency graph from the initial values. Following [5], parallelising a program is to embed the program’s data dependency pattern into the space-time of the target computer’s communication structure. This information can be provided by a mapping of the nodes of the dependency graph to space-time coordinates of the target computer. Allowing the programmer to define this mapping, gives explicit control over the parallelisation at a very high abstraction level. The Sapphire compiler uses this information to generate an imperative program which traces the compact data dependency graph from the initial values. The imperative program generates the nodes of the graph when they are first activated in the computation, and reclaims the nodes as soon as they have been computed and their values passed on in the graph. Parallel code is generated using MPI. 1
If we limit the form of the δ , we get a recurrence relation.
760
3
Magne Haveraaen
Example: Heatflow in One Dimension 2
u(x,t) The one-dimensional heat flow equation, α2 ∂ ∂x = ∂u(x,t) 2 ∂t , describes the heat distribution u(x, t) in a one-dimensional rod along the x-axis at time t. Here α is a physical constant describing the heat transfusion properties of the rod. To turn the equation into a recursion we may use the finite difference method both in time and space, giving
u(x, t) = κu(x + ∆x, t − ∆t) + (1 − 2κ)u(x, t − ∆t) + κu(x − ∆x, t − ∆t), where κ = α2 ∆t/∆x2 . This simple program has 3 recursive calls per timestep. Its stencil is given by the δ functions. (x, t) HH δ δ HHH δ Hj t − ∆t) ? (x + ∆x, (x − ∆x, t − ∆t) (x, t − ∆t) 3
2
1
As opposites we get the δ functions.
HHH
HHH j (x,?t) δ
(x − ∆x, t + ∆t) δ1
(x, t + ∆t) δ2
(x + ∆x, t + ∆t) 3
(By convention we always draw the arrows in the δ direction.) The Sapphire compiler translates this code to non-recursive ANSI C, which traverses the compact graph of the stencils bottom-up. A series of tests were run sequentially on one processor of a SGI Cray Origin 2000 machine. At each timestep the heat distribution for all discrete x-values is computed. The running time of the generated C program clearly grows linearly with the number of timesteps. No. of No. of Execution Measured Ideal Grid Points Timesteps Time(sec) growth growth 4000 1000 7.83 1 1 4000 2000 15.64 2.0 2 4000 4000 31.24 4.0 4 4000 8000 62.46 8.0 8 Tests of the MPI version on the SGI Cray Origin 2000 shows good parallel efficiency (1000 timesteps and 80 000 grid points). No. of Execution Measured Ideal Efficiency Processors Time(sec) speed-up speed-up Measured/Ideal 2 298 1 1 100% 4 150 2.0 2 100% 8 75 4.0 4 100% 16 38 7.8 8 98% 32 20.5 14.5 16 91% All tests were run with the machine in ordinary production mode.
Efficient Parallelisation of Recursive Problems Using Constructive Recursion
4
761
Conclusion
Recursive programming is a very efficient problem solving technique, but with traditional compiler technology this is often prohibitively expensive to compute. We have suggested supplying the stencil and its opposite as an explicit part of the program. This makes it possible for a compiler to generate code that traces the compact data dependency graph in an efficient data-driven fashion. We can also augment the data dependency graph with information on the distribution of the computation on the processors of a parallel computer. This yields full control over the parallel execution of the code, without the need for the programmer to get involved in the parallel code itself. Similar approaches have been investigated by other researchers, but in more restricted contexts. In [6] many approaches based on functional programming are reported. Some of these utilise the dependency in order to generate data driven code and parallelise programs, but they do not give the user explicit control over the opposite dependency. This limits the efficiency of the approaches to cases where the dependency may be automatically inverted. In an imperative context ˇ Cyras & al. [1] have used dependencies to provide a modular decomposition of the recursive function (with stencils) from code tracing out the graph (the opposite stencils). A detailed comparison with our technique is presented in [2]. See [4] for more detailed examples of the constructive recursion technique. Acknowledgements Thanks to Steinar Søreide who implemented the Sapphire compiler and did the timings.
References ˇ [1] Vytautas Cyras. Loop synthesis over data structures in program packages. Computer Programming, 7:27–50, 1983. (in Russian). ˇ [2] Vytautas Cyras and Magne Haveraaen. Modular programming of recurrencies: a comparison of two approaches. Informatica, 6(4):397–444, 1995. [3] Magne Haveraaen. How to create parallel programs without knowing it. In Proceedings of the 4th Nordic Workshop on Program Correctness – NWPC4, number 78 in Reports in Informatics, pages 165–176, P.O.Box 7800, N-5020 Bergen, Norway, April 1993. [4] Magne Haveraaen and Steinar Søreide. Solving recursive problems in linear time using constructive recursion. In Sverre Storøy, Said Hadjerrouit, et al., editors, Norsk Informatikk Konferanse – NIK’98, pages 310–321. Tapir, Trondheim, Norway, 1998. [5] W.L. Miranker and A. Winkler. Spacetime representations of computational structures. Computing, 32:93–114, 1983. [6] B.K. Szymanski, editor. Parallel Functional Languages and Compilers. ACM Press; New York / Addison Wesley; Reading, Mass., 1991.
Development of Parallel Algorithms in Data Field Haskell Jonas Holmerin1 and Bj¨ orn Lisper2 1
2
Department of Numerical Analysis and Computing Science, Royal Institute of Technology, S-100 44 Stockholm, SWEDEN, [email protected] Dept. of Computer Engineering, M¨ alardalen University, P.O. Box 883, S-721 23 V¨ aster˚ as, SWEDEN, [email protected]
Abstract. Data fields provide a flexible and highly general model for indexed collections of data. Data Field Haskell is a dialect of the functional language Haskell which provides an instance of data fields. We describe Data Field Haskell and exemplify how it can be used in the early phase of parallel program design.
1
Introduction
Many computing applications require indexed data structures. The canonical indexed data structure is the array. However, for sparse, distributed applications, other, more dynamic indexed data structures are needed. It is desirable to develop such algorithms on a high level first, in order to get them right, since the low level data representations can be intricate. Data Field Haskell provides an instance of data fields – a data type for general indexed structures. This Haskell dialect can be used for rapid prototyping of parallel computational algorithms which may involve sparse structures. Various versions of the data field model have been described elsewere [1, 2, 3, 4]. The contribution of this paper is a description of an implementation and an example of how it can be used in parallel program design.
2
The Data Field Model
Data fields are based on the more abstract model of indexed data structures as functions with finite domain [1, 2]. This model is simple and powerful, but for real implementations explicit information about the function domains is needed. Data fields are thus entities (f, b) where f is a function and and b is a bound, a set representation which provides an upper approximation of the domain of the corresponding function. We require that the following operations are defined for bounds:
This work was supported by The Swedish Research Council for Engineering Sciences (TFR), grant no. 98-653.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 762–766, 2000. c Springer-Verlag Berlin Heidelberg 2000
Development of Parallel Algorithms in Data Field Haskell
763
– An interpretation of every bound as a set. – A predicate classifying each bound as either finite or infinite, depending on whether its set is surely finite or possibly infinite. – For every bound b defining a finite set, size(b) which yields the size of the set and enum(b) which is a function enumerating its elements. – Binary operations , on bounds (“intersection”, “union”). – The bounds all and nothing representing the universal and empty set, respectively. These operations are chosen to support the usually assumed set of collectionoriented operations [7] without revealing the inner structure of the bounds. An important derived operation is explicit restriction: (f, b) ↓ b = (f, b b). The theory of data fields also defines ϕ-abstraction, a syntax for convenient definition of data fields which parallels λ-abstraction for functions. See [3, 4].
3
Data Field Haskell
Data Field Haskell is a Haskell dialect where the arrays have been replaced by an instance of data fields, a variation of the sparse/dense arrays of [3, 4]. Our implementation of Data Field Haskell is based on the NHC compiler [6] for Haskell v. 1.3. The implementation is sequential and we have not implemented any advanced optimizations.
datafield ! bounds outofBounds predicate join, meet <\> inBounds foldlDf
defines data field from function and bound data field indexing the bound of a data field an out-of-bounds error value forms predicate bound from predicate “union” and “intersection” of bounds explicit restriction of data field with bound checks if an element belongs to the set defined by a bound folds (reduces) finite data field w.r.t. binary operation
Table 1. Some operations on data fields and bounds.
Data Field Haskell has data types Datafield a b for datafields and Bounds a for the corresponding bounds. Table 1 lists some functions for data fields and bounds. It has a rich variety of finite and infinite bounds: dense bounds, i.e., traditional array bounds, sparse finite bounds, which represent general finite sets, predicate bounds, which are classified as infinite, universe which represents the universal set, and empty which represents the empty set. Product bounds represent Cartesian products and generalise multidimensional array bounds.
764
3.1
Jonas Holmerin and Bj¨ orn Lisper
Forall- and For-Abstraction
Data Field Haskell provides ϕ-abstraction, with syntax similar to λ-abstraction in Haskell [5]: forall apat1 . . . apatn -> exp The semantics of forall x -> t is datafield (\x -> t) b, where b is computed from bounds of data fields in t as to give an upper approximation of the domain of \x -> t. It can be thought of as an implicitly parallel, functional forall statement where first b is computed and then, if needed, \x -> t is computed for all x in b. for-abstraction provides a convenient syntax to define a data field by cases over different parts of its domain. The syntax is for pat in { e1 -> e1 ; . . . ; en -> en } where the ei are bounds and the ei are data field values. The semantics is a data field whose bound is restricted by the union of e1 , . . . , en and whose value for each x is ei ! x, for the lowest i such that x belongs to ei .
4
An Example
We exemplify the use of Data Field Haskell with simulation of a system of particles over a range of time. Each particle i has a state (¯ ri , v¯i ) with a position r¯i and a velocity v¯i . The state transition function (1) gives the new state after time ∆t. These equations are parameterized w.r.t. fi , which yields the acceleration ai for each particle i from the spatial distribution of the particles. They are more or less directly expressed1 in Data Field Haskell in Fig. 1. The result is an executable, dimension-polymorphic specification of particle simulation with time step dt. (¯ ri + v¯i · ∆t, v¯i + a ¯i · ∆t), i ∈ Particles (¯ ri , v¯i ) → a ¯i = fi ( rj | j ∈ Particles ), i ∈ Particles
(1)
This specification can be manually refined into a more explicitly parallel algorithm for particle simulation by distributing the particle state onto a set of processors, and localizing the parallel algorithm by neglecting long-range interactions. The resulting parallel algorithm has a distributed state which consists of a predicate, a sparse particle datafield, and a neighbourhood bound for each processor. A processor “owns” each particle in the area defined by its predicate. After each iteration every processor tests which particles it now owns. This test is localized to particles originating from the neighbours only. This approximation is correct only if long-range interactions can be ignored, for instance if we 1
The explicit restriction in p newstate is necessitated by a flaw in the current derivation of bounds for forall-abstraction. We expect to rectify it in later releases of Data Field Haskell.
Development of Parallel Algorithms in Data Field Haskell
765
simulate molecules which interact through collisions only. The set of neighbour processors is described, for each processor, by its neighbourhood bound. Fig. 1 also gives the data type and transition function d_newstate for the distributed particle simulation. d_newstate first computes, for each processor, the union of the particles over the neighbours, then a local state transition for these, and finally it “masks out” the particles which now belong to it. (Since data fields are not sets we must instead of set union use “prunion”, which gives priority to its first argument for indices where both arguments are defined.) d_newstate does not modify the predicate field, but this is possible and could be used to model adaptive methods where areas for different processors are changed to balance the load. data Pstate = PS (Datafield Int Float) (Datafield Int Float) pos (PS r _) = r vel (PS _ v) = v p_newstate dt f s = forall i -> let ai = f i s in PS ((forall k-> ((pos (s!i))!k) + ((vel (s!i))!k)*dt) <\> bounds (pos (s!i))) (forall k-> ((vel (s!i))!k) + (ai!k)*dt)
data Dstate proc_id part_id = DS ((Datafield Int Float) -> Bool) (Datafield part_id Pstate) (Bounds proc_id) pred (DS p _ _) = p state (DS _ s _) = s neigh (DS _ _ b) = b prunion d1 d2 = forall x -> if (inBounds x (bounds d1)) then d1!x else d2!x prUnion = foldlDf prunion (datafield (\x -> outofBounds) empty) d_newstate dt f dstate = forall p -> DS (pred (dstate!p)) (let { neighstate = prUnion (for pn in neigh (dstate!p) -> state (dstate!pn)); newstate = (p_newstate dt f neighstate) } in newstate <\> predicate(\j -> pred (dstate!p)(pos(newstate!j)))) (neigh (dstate!p))
Fig. 1. Data Field Haskell solutions for the simulation problem: executable specification, and parallelised version.
766
5
Jonas Holmerin and Bj¨ orn Lisper
Conclusions
We have presented Data Field Haskell, a Haskell dialect with data fields which supports a highly flexible form of collection-oriented programming. A possible application is for rapid prototyping in the early specification phase of parallel algorithms. We exemplified with the specification and initial development of a simple parallel particle simulation algorithm. The resulting algorithm specification is dimension-polymorphic and easily adaptable to regular and irregular grids, static or dynamic mappings, and different target architectures.
References [1] P. Hammarlund and B. Lisper. On the relation between functional and data parallel programming languages. In Proc. Sixth Conference on Functional Programming Languages and Computer Architecture, pages 210–222. ACM Press, June 1993. [2] B. Lisper. Data parallelism and functional programming. In G.-R. Perrin and A. Darte, editors, The Data Parallel Programming Model: Foundations, HPF Realization, and Scientific Applications, Vol. 1132 of Lecture Notes in Comput. Sci., pages 220–251, Les M´enuires, France, Mar. 1996. Springer-Verlag. [3] B. Lisper. Data fields. In Proc. Workshop on Generic Programming, Marstrand, Sweden, June 1998. http://wsinwp01.win.tue.nl:1234/WGPProceedings/. [4] B. Lisper and P. Hammarlund. The data field model. Technical Report TRITA-IT R 99:02, Dept. of Teleinformatics, KTH, Stockholm, Mar. 1999. ftp://ftp.it.kth.se/Reports/TELEINFORMATICS/TRITA-IT-9902.ps.gz. [5] J. Peterson, K. Hammond, L. Augustsson, B. Boutel, W. Burton, J. Fasel, A. D. Gordon, J. Hughes, P. Hudak, T. Johnsson, M. Jones, E. Meijer, S. L. Peyton Jones, A. Reid, and P. Wadler. Report on the programming language Haskell: A non-strict purely functional language, version 1.4, Apr. 1997. http://www.haskell.org/definition/. [6] N. R¨ ojemo. Garbage Collection, and Memory Efficiency, in Lazy Functional Languages. PhD thesis, Department of Computing Science, Chalmers University of Technology, Gothenburg, Sweden, 1995. [7] J. M. Sipelstein and G. E. Blelloch. Collection-oriented languages. Proc. IEEE, 79(4):504–523, Apr. 1991.
The ParCeL-2 Programming Language Paul-Jean Cagnard Swiss Federal Institute of Technology, CH-1015 Lausanne, Switzerland [email protected]
Abstract. A semi-synchronous parallel programming language and the corresponding computing model are presented. The language is primarily designed for massively parallel fine grained applications such as cellular automata, finite element methods, partial differential equations or systolic algorithms. In this language it is assumed that the number of parallel processes in a program is much larger than the number of processors of the machine on which it is to be run. The computational model and the communication model of the language are presented. Finally, the language syntax and some performance measurements are presented.
1
Introduction
Many parallel programming models have been proposed in the past years as alternatives to both message passing and data parallel programming. Among the most prolific class of alternate models there is the family of languages based on concurrent objects or actors [1, 2]. In actor- or object-based models, parallelism of execution is expressed explicitly. These models allow to express the distribution of data explicitly and in a natural way. This opens interesting opportunities for automated data- and process-mapping. Nearly all models based on concurrent objects or actors use asynchronous execution models, i.e., execution schemes where computation and message sending are not inherently automatically globally synchronised. This entails that communication is not easy to implement efficiently and concurrent object systems are often restricted to coarse grain parallelism for performance reasons. In the following a parallel programming language whose computation model can be seen as an extension of the BSP [3] model and other synchronous objectoriented models is presented.
2
The ParCeL-2 Programming Model
A ParCeL-2, program is typically composed of a large number of fine grained processes executing concurrently. Each process execution consists of a sequence of supersteps. A superstep is composed of two phases: a computation phase and a communication phase. Supersteps are separated by synchronisation barriers.
Work supported by the Swiss National Science Foundation under grant 21-49519.96.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 767–770, 2000. c Springer-Verlag Berlin Heidelberg 2000
768
Paul-Jean Cagnard
During the computation phase of a superstep, a process executes computations which only manipulate data local to this process. A process can send data to other processes in the course of a computation phase. The effective transmission of data between processes happens at the end of each superstep. This means that no data are exchanged during computation phases. It can be observed that this computing model is an intrinsically distributed memory MIMD model. Aggregates of processes can be constructed in order to build abstract processes whose behaviour is more complex than an elementary process but the rest of the program does not need to know how this behaviour is implemented. Aggregates of processes offer means of creating arbitrarily complex process topologies and hierarchical parallel program designs. In ParCeL-2, unlike in BSP, every process, or cell, can execute supersteps with a slower frequency than that of the execution environment’s global clock. This is why ParCeL-2 is called semi-synchronous. This is useful since complex cells containing other cells may need more time to produce a meaningful result than an elementary cell. This is a way of saying that if a cell has a synchronisation period n, data will only send along or read from its channels every n elementary supersteps. In ParCeL-2, processes can only communicate through predeclared channels. Channels have four main properties. They are typed: a channel, like a local variable, is of a certain type, and only data of this type can be sent through this channel. They are directed: channels are unidirectional. If bidirectional communications are needed, two channels of opposite directions must be created between processes. They have a period which is the same as the period of the cell at the start of the channel. They are static: this means that processes connected at the other end of a channel can’t change during execution, and that channels can’t be created nor destroyed at runtime. Processes are also static, they can’t be created nor destroyed at runtime. In the case of complex cells, there is a special type of connection allowed, called a shunt which is useful when the complex cell is only a structural cell, i.e. has no body, or when some of the internal cells need to pass data to the exterior of the complex directly. Channels do not necessarily connect only one process to another single process, they may connect many processes together. Thus three kinds of channels may be declared in a program: 1-1 channels, 1-n channels, and n-1 channels in which case an operator can be attached to the channel to reduce n values into a single one. One of the goals of ParCeL-2 and one of its main features is to provide a topology description language in order to be able to describe easily, elegantly and concisely complex topologies. Such topology description languages have been developed, see [4, 5], but neither of them permits to construct topologies with all the features of ParCeL-2 channels.
The ParCeL-2 Programming Language
3
769
The ParCeL-2 Syntax
The declaration of a type of cells consists of three main parts: an interface, a body or behaviour, and a topology. 3.1
Interface Declarations
In the interface part of a cell type, one can put type declarations, constant declarations and channel declarations. Type and constant declarations are written according to the syntax of the language chosen for the body of the cell type. An out channel is always declared the same way, whether it will be connected as a 1-1, 1-n or n-1 channel: OUT
An in channel can be declared in three ways according to the chosen kind of utilisation, respectively 1-1 or 1-n, n-1 mailbox , and n-1 with reduction. IN
The import
Body Declarations
In the body part of a cell type, anything that can be written in the chosen programming language can be written. With the exception that some keywords are reserved by ParCeL-2. That code will be executed at each superstep. 3.3
Topology Declarations
This is in that part of a cell type that internal cells are instantiated and connected according to the topology desired by the programmer. This is what differentiates elementary cells from complex cells. Complex cells embed several instances of other types of cells, which are called internal cells, and the topology describes how these instances are connected together and with the embedding cell. The topology description language is currently still under development. Work is underway to provide construction operators like the cartesian product of two graphs and the joining of two sub-topologies, like in Candela [6]. The language will be shown through two examples which illustrate a highly regular and a highly irregular topology. The cell types used are supposed to be declared somewhere else. ring(n) of TA { aux = array(1..n) of TA; for i in 1..n
770
Paul-Jean Cagnard
connect(aux[i].out1,aux[(i mod n) + 1].in1); connect(aux[i].out2,aux[(i mod n) + 1].in2); connect(aux[(i mod n) + 1].out3, aux[i].in3); end for; } ring(3) of TA; A = new T1; B = new T2; C = new T3; connect(A.out1,B.in1); connect(B.out1,C.in1); connect(C.out1,A.in1); connect(A.g,g); connect(h,B.h); shunt(C.e,c); shunt(d,e.f);
4
Conclusion and Future Work
A programming language particularly well suited to the design of massively parallel fine grained applications has been presented. Complex computations using cellular automata, partial differential equations, finite element methods, systolic arrays or even static neural networks can benefit much from such a language, notably in portability and readability. A first version of the runtime environment has been implemented and used to run a program implementing the game of life with a grid of 1000 × 1000 cells on a Fast Ethernet network of 333 MHz Sun Ultra 10 workstations using MPI as a communication library. For 50 iterations, the best sequential program took 3.3 s. The parallel version was slower, but scaled quite well, from 22.5 s to 3.89 s, using 1 to 16 processors, which is very satisfying considering that our runtime environment is not optimized at all yet. Now its behaviour must be studied with real world applications like wave propagation [7] for example. Work on a full compiler remains to be done. It exists only in parts at the moment.
References [1] C. Hewitt, P. Bishop, and R. Steiger. A universal modular actor formalism for artificial intelligence. In IJCAI-73, 1973. [2] G. Agha. ACTORS, a model of concurrent computation in distributed systems. MIT Press, 1986. [3] Leslie G Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103, August 1990. [4] Alexey Lastovetsky. mpC - a multi-paradigm programming language for massively parallel com puters. In ACM SIGPLAN Notices, volume 31(2), pages 13–20. February 1996. [5] Andr´ as Varga. Parametrized topologies for simulation programs. In Western Multiconference on Simulation (WMC’98) / Communication Network s and Distributed Systems (CNDS’98), San Diego, CA, January 1998. [6] Herbert Kuchen and Holger Stoltze. Candela – a topology description language. Journal of Computers and Artificial Intelligence, 13(6):557–676, 1994. [7] F. Guidec, P. Cal´egari, and P. Kuonen. Parallel irregular software for wave propagation simulation. Future Generation Computer Systems (FGCS), N.H. Elsevier, 13(4-5):279–289, March 1998. ISSN 0167-739X.
Topic 11 Numerical Algorithms for Linear and Nonlinear Algebra Ulrich R¨ ude and Hans-Joachim Bungartz Topic Chairmen
Since the early days of supercomputing, numerical algorithms have been the application with the highest demand for computing power anywhere. Many of today’s fastest computers in the world, as given in the TOP 500 list1 , are mostly used for the solution of huge systems of equations as they arise in the simulation of complex large scale problems in engineering and science. In contrast to the situation a decade ago when vector processors still dominated the supercomputer market, now, all high-performance computers are parallel architectures. Furthermore, the number of processors in current state-of-theart supercomputer systems ranges from 100 to 10000, corresponding to moderate to massive parallelism. The most powerful of these high-end computers currently perform beyond one Teraflop, and expected future Petaflop systems will even have to further increase parallelism, thus increasing the demand for efficient parallel numerical algorithms, too. The answer to the question whether it will be possible to exploit the full potential of such massively parallel systems will, therefore, primarily depend on future progress in the development of parallel algorithms for large scale algebraic systems and related numerical problems. This crucial importance of parallel numerical algorithms certainly justifies devoting a special workshop to this topic at a conference on parallel systems and algorithms like Euro-Par, in addition to the discussion of special aspects of high-performance computing in Topic 7 (Applications on High-Performance Computers). Recently, as numerical simulation has become more and more a standard technology in academic and industrial research and development, parallel numerical algorithms have also gained in importance outside the field of classical supercomputers. Clusters of networked personal computers or workstations, as they are in the focus of Topic 18 (Cluster Computing), do no longer suffer from the image of being the poor man’s supercomputer, only. Among other arguments, the better availability as well as an often more favourable cost effectiveness of the cluster approach have obviously helped to raise industry’s interest in parallel or distributed solutions. Hence, today, we face an increasing demand for parallel engineering applications on clustered systems. Overall, fifteen papers were submitted to our Topic. With authors from Croatia, Germany, Greece, Italy, Russia, Spain, and the United Kingdom, Europe is the dominant male (as expected at Euro-Par), but four papers are authored 1
www.top500.org
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 771–773, 2000. c Springer-Verlag Berlin Heidelberg 2000
772
Ulrich R¨ ude and Hans-Joachim Bungartz
or co-authored by scientists from the United States and from China. Out of these fifteen submissions, nine were accepted (six as regular papers and three as research notes). Both devising new numerical algorithms and adapting existing ones to the current parallel systems are vigorously flourishing and still developing fields of research. Hence, it is no surprise that the ten research articles presented in this section cover a wide range of topics arising in the various subdomains of parallel numerical algorithms. At the conference, the presentations were arranged into three sessions on Numerical Linear Algebra, Algorithms for Partial Differential Equations, and Ordinary Differential Equations and other Topics. This structure also reflects in the following part of the conference’s proceedings. Many if not most large scale algebraic systems arise out of the discretization of (partial) differential equations. Hence, the resulting algebraic systems are sparse. Exploiting the sparsity structure is the key to an efficient solution. However, this requires approaches tailored to the respective situation. Here, many of the leading algorithms are iterative schemes. Thus, numerical methods for partial differential equations and numerical linear algebra are closely related, which may let appear the subdivision of the talks in the first two sections somewhat arbitrary. In the Numerical Linear Algebra section, David S. Wise discusses the Ahnentafel indexing and Morton-ordered arrays, a technique for improving the locality of multi-dimensional arrays (like sparse sets of points or sparse matrices) in memory. This is of importance for both improving the cache performance on a single processor and for partitioning matrices across processors without entailing large communication during matrix operations. Peter Gottschling and Wolfgang E. Nagel deal with practical experience with the cascadic conjugate gradient method, an algorithm with an asymptotic performance similar to multigrid methods, but with a simpler access to parallelization. The contributions in the session on Algorithms for Partial Differential Equations deal with the fast solution of large linear systems and with grid generation. For an efficient parallelization of PDE solvers, already the grid generation and partitioning must be performed in parallel. The presentation given by J¨ orn Behrens and Jens Zimmermann is concerned with the parallel generation of unstructured grids based on the idea of space-filling curves, recently used for parallelization purposes. The talk of Michael Bader and Christoph Zenger deals with a robust multigrid solver for systems stemming from convection-diffusion equations, based on the idea of the iterative nested dissection. The third talk, given by Marcus Mohr, reports on a variant of multigrid where special features of the algorithm are exploited to reduce the amount of communication between subdomains. The final session gathers various topics. Peter Benner, Rafael Mayo, Enrique S. Quintana-Orti, and Vicente Hernandez deal with a cluster solver for the discrete-time periodic Riccati equations. Furthermore, there are two contributions on optimization problems. In the presentation by Torsten Butz, Oskar von Stryk, Thieß-Magnus Wolter, and Cornelius Chucholowski, the optimization
Topic 11: Numerical Algorithms for Linear and Nonlinear Algebra
773
arises out of a parameter estimation problem in vehicle dynamics as a genuinely industrial application on a cluster of PCs. The talk by Marco D’Apuzzo, Marina Marino, Panos M. Pardalos, and Gerardo Toraldo deals with box-constrained quadratic programming. Finally, Christos Kaklamanis, Charalampos Konstantopoulos, and Andreas Svolos discuss a parallel implementation of a slidingwindow compression algorithm for hypercube processor topologies. Altogether, the contributions to Topic 11 at the millennium Euro-Par in Munich show once more the great variety of interesting, challenging, and important issues in the field of parallel numerical algorithms. Thus, we are already looking forward to the results presented at next year’s Euro-Par conference.
AHNENTAFEL INDEXING INTO MORTON-ORDERED ARRAYS, or MATRIX LOCALITY FOR FREE? David S. Wise?? Indiana University Abstract. Definitions for the uniform representation of d-dimensional matrices serially in Morton-order (or Z-order) support both their use with cartesian indices, and their divide-and-conquer manipulation as quaternary trees. In the latter case, d-dimensional arrays are accessed as 2d -ary trees. This data structure is important because, at once, it relaxes serious problems of locality and latency, and the tree helps schedule multiprocessing. It enables algorithms that avoid cache misses and page faults at all levels in hierarchical memory, independently of a specific runtime environment. This paper gathers the properties of Morton order and its mappings to other indexings, and outlines for compiler support of it. Statistics elsewhere show that the new ordering and block algorithms achieve high flop rates and, indirectly, parallelism without any low-level tuning. CCS Categories and subject descriptors: E.1 [Data Structures]: Arrays; D.3.2 [Programming Languages]: Language Classifications–concurrent, distributed and parallel languages; applicative languages; D.4.2 [Operating Systems]: Storage management–storage hierarchies; E.2 [Data Storage Representations]: contiguous representations. General Term: Design. Additional Key Words and Phrases: caching, paging, compilers, quadtree matrices.
1 INTRODUCTION Maybe we’ve not been representing them efficiently for some time. Matrix problems have been fodder for higher-level languages from the beginning [1], and row- or column-major representations for matrices are universally assumed. Both use the same space, both still survive. But maybe both are archaic perspectives on matrix structure, which might best be represented using a third convention. Architecture has developed quite a bit since we had to pack scalars into small memory. With hierarchical—rather than flat—memory, only the faster memory is precious; with distributed processing instructions on local registers are far faster than those touching remote memory. And, of course, multiprocessing on many cheap processors demands less crosstalk than code for single threading on uniprocessors with unshared ? Copyright c 2000 to Springer-Verlag, with rights reserved for anyone to make digital or hard copies of part
or all of this work for personal or classroom use, provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full Springer citation on the first page. Rights are similarly reserved for any library to share a hard copy through interlibrary loan. ?? Supported, in part, by the National Science Foundation under grants numbered CDA93–03189 and CCR9711269. Author’s address: Computer Science Dept, 215 Lindley Hall, Bloomington, IN 47405–7104, USA. [email protected]
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 774-783, 2000. © Springer-Verlag Berlin Heidelberg 2000
Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free 16
0
0
1
4
5
2
3
6
7
8
9
12
13
10 11 14
15
64
31
15 32
1
2
3
4
5
6
7
8
9
10 11
12 13
14 15
128
143
175
k
191
5
7
8
9
10
255
239
5
3
2
223 240
7
1
6
207
4 matrix, and analogous Morton indexing.
4k+1 4k+2 4k+3 4k+4
∑ 4i i=0
127 208
224
0
l-1
111
159 176
95 112
192
144
160
Figure 1. Row-major indexing of a 4
79
63
47
0
80
96
48
775
4
4 l (4 -1) 3
6
18
9
10
2
11 12
04
13 14
3 15 16
17 18 19 20
11 12 13 14 15 16 17 18 19 20
Figure 2. Level-order indexing of the order-4 quadtree and its submatrices.
memory. Address space, itself, has grown so big that chunks of it are extremely cheap, when mapped to virtual memory and never touched. (Intel’s IA-64 processor has three levels of cache, two on board.) But fast cache, local to each processor, remains dear, and, perhaps, row-major and column-major are exactly wrong for it. This paper enhances Morton-order (also called Z-order) storage for matrices. Consistent with the conventional sequential storage of vectors, it also provides for the usual cartesian indexing into matrices (row, column indices). It extends to higher dimensional matrices. That is, we can provide cartesian indexing for “dusty decks” while we write new divide-and-conquer and recursivedescent codes for block algorithms on the same structures. H ASKELL and ML could share arrays with F ORTRAN and C. Thus, parallel processing becomes accessible at a very high level in decomposing a problem. Best of all, the content of any blocks is addressed sequentially and blocks’ sizes vary naturally (they undulate) to fit the chunks transferred between levels of the memory hierarchy.
2 BASIC DEFINITIONS Morton presented his ordering in 1966 to index frames in a geodetic data base [11]. He defines the indexing of the “units” in a two-dimensional array much as in Figure 1, and he points out the truncated indices available for enveloping blocks (subtrees), similar to Figure 3. Finally, he points out the conversion to and from cartesian indexing available through bit interleaving.
776
David S. Wise k
0
4k+0 4k+1 4k+2 4k+3
0
0
Base ten
0
Base four Base two
1
00
2 01
0 0 0 1
0 0 0 0
i= 0 j= 0
0 1
2
1
3
02 0 1 0 0
1 0
4
03 0 1 0 1
1 1
5
6
7
8
9
3
4l-1
10 11 12 13 14 15
10
11
12
13
20
21
22
23
30
31
0 0 1 0
0 0 1 1
0 1 1 0
0 1 1 1
1 0 0 0
1 0 0 1
1 1 0 0
1 1 0 1
1 0 1 0
1 0 1 1
1 1 1 0
1 2
1 3
2 1
3 0
3 1
2 2
2 3
3 2
0 2
0 3
2 0
32
33 1 1 1 1
3 3
Figure 3. Morton indexing of the order-4 quadtree. k
3
4k+0 4k+1 4k+2 4k+3
12
3•4l
Base ten
14
13
15
4l+1-1
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Base four 300 301 302 303 Base two 110000 110001 110010 110011
i= 0 j= 0
0 1
1 0
1 1
310
311
312
1 0 0 1 0 0 1 0 1 1 1 0 1 1 1 1 1 0
0 2
0 3
1 2
313
320
1 0 1 1 1 1
1 1 0 1 1 0 1 0 0 1 0 1
1 3
2 0
321
2 1
322
323
1 1 1 1 0 0
1 1 0 1 1 1 1 1 0 1 0 1
3 0
3 1
330
2 2
331
332
1 1 0 1 1 1
1 1 1 1 1 0
2 3
3 2
333 1 1 1 1 1 1
3 3
Figure 4. Ahnentafel indexing of the order-4 quadtree.
The definitions later focus on two-dimensional arrays: matrices. The early ones are general: a d-dimensional array is decomposed as a 2d -ary tree. 2.1 ARRAYS Definition 1 In the following m = 2d is the degree of the tree appropriate to the dimension d. If the maximal order in any dimension is n then the tree has maximal level dlgne. Use Figures 2, 3, and 4 in reading the following definitions. Definition 2 A complete array has level-order index 0. A subarray (block) at levelorder index i is either a scalar, or it is composed of m subarrays, with level-order indices mi + 1 mi + 2 : : : mi + m. Definition 3 The root of an array has Morton-order index 0. A subarray (block) at Morton-order index i is either a unit (scalar), or it is composed of m subarrays, with indices mi + 0 mi + 1 : : : mi + (m 1) at the next level.
Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free
777
Theorem 1. The difference between the level-order index of a block at level array and its Morton-order index is (ml 1)=(m 1):
l in an
The difference is the number of nonterminal nodes above level l. Since each level is indexed by its zero-based scheme, it is necessary also to know the level and a Morton index, to identify a specific node. Ahnentafel indices are immensely useful for identifying blocks at all levels [17]. Algorithms that use recursive-descent (divide-and-conquer) to descend to a block of arbitrary size, or to return the index of a selected block, need only this single index to identify any subtree. Treat them only as identification numbers because there are gaps in the sequence between levels. Conversions among Ahnentafel indices, cartesian indices, Morton order, and level order are easy. Ahnentafel indices come to us from genealogists who invented them for encoding one’s pedigree as a binary, family tree. This generalization to m-ary trees is new. Definition 4 [4] A complete array has Ahnentafel index m 1. A subarray (block) at Ahnentafel index a is either a scalar, or it is composed of m subarrays, with indices ma + 0 ma + 1 : : : ma + (m 1). Theorem 2. The nodes at level l have Ahnentafel indices from (m 1)ml to ml+1 The gap in indices between level l 1 and level l is ml (m 2) + 1.
1.
Theorem 3. The level of a node with Ahentafel index a is l = blogm ac = blogm ma 1 c. The difference between the Ahnentafel index and the level-order index is (ml+1 (m 2) + 1)=(m 1): The difference between the Ahnentafel index and the Morton-order index is (m 1)ml : The gaps between levels in Ahnentafel indexing of quadtrees do not arise in binary trees. The strong similarity between level-order and Ahnentafel indexing in this common case perhaps explains why the latter has been often overlooked. For instance, Knuth’s level-order indexing, based at one, is off-byone relative to Definition 2 [10, p. 401], but coincides with Ahnentafel indexing on binary trees. 2.2 MATRICES Hereafter, we assume d = 2 for matrices; so m = 4. In all the figures, the cartesian indices of the leaves appear in outlined font below the tree. Corollary 1. The difference between level-order index of a matrix block at level l and its Morton-order index is (4l 1)=3.
Corollary 2. The gap in Ahnentafel indices between level l
1
and l is 22l+1 + 1.
Corollary 3. The difference between the Ahnentafel index of a submatrix at level l and its level-order index is (22l+3 + 1)=3. The difference between its Ahnentafel and its Morton-order index is 3 4l :
Definition 5 Let w be the number of bits in a (short) word. Each qk is a modulo-4 digit (or quat). Each qk is alternatively expressed as qk = 2ik + jk where ik and jk are bits.
778
David S. Wise
Cartesian indices have w bits; Morton indices (and, later, dilated integers) have 2w bits.
P
P
w 1
P
P
w 1
P
w 1
Theorem 4. [11] The Morton index k=0 qk 4k = 2 k=0 ik 4k + k=0 jk 4k w 1 w 1 corresponds to the cartesian indices: row k=0 ik 2k and column k=0 jk 2k The proof is by simple induction on w; doubling the order of a matrix introduces two high-order bits. The quats, read in order of descending subscripts, select a path from the root to the node, as in Figure 4. For example, if i = 4 = 104 and j = 8 = 204 in Figure 1, the Morton index is 2004 + 10004 = 30004 = 9610 . Corollary 4. Let l = blog4 ac. The Ahnentafel index, a = 1 k the Morton index lk=0 qk 4 .
P
P
l k k=0 qk 4 corresponds to
The bits, fik g, are the odd-numbered bits in the Morton index, and the fjk g are just the even-numbered bits; and, excluding Corollary 3’s two high-order 1 bits, coincident with those in an Ahnentafel index. This is Morton’s bit interleaving of cartesian indices. Code to convert from cartesian indices to a Morton index by shuffling bits— or the inverse conversion that deals out the bits—can be slow. Fortunately, as the next section shows, most conversions can be elided. ! w 1 Definition 6 The integer b = k=0 4k here labeled evenBits in C, and is the ! constant 0x55555555). Similarly, b = 2 b is called oddBits, 0xaaaaaaaa). ! In C code b is a very important constant available independently of w as ! ((unsigned int)-1)/3). Masking a Morton index with b or b extracts the bits of the column and row cartesian indices. Morton describes how to use these to obtain indices of neighbors. Their easy identification makes the indexing attractive for graphics in two dimensions and for spatial data bases in three. It is remarkable how often these basic properties of Morton ordering have been reintroduced in different contexts [3, 7, 9, 12, 16]. Samet gives an excellent history [13]. When first introduced, it might appear that Morton indexing is poor for large arrays that are not square or whose size is not a power of two, because in those cases its gaps seem to waste space. With the valid elements justified to the north and west, the “wasted” space lands in the south and east. It can be viewed as padding, perhaps all zero elements. However, the perceived waste is merely address space. In hierarchical memory only its margin will ever be swapped into cache. That is, the “wastage” exists only within the logical/physical addressing of swapping disk; little valuable, fast memory is lost. With these orders defined and the various index translations among them understood, we can summarize the programmer’s view of Morton order:
P
– The elements of a matrix (an array) are mapped onto memory in Morton order. – Row and column traversals can still be handled. See the next section. – If necessary with cartesian indexing, restrict blocking to submatrices implicit in the quadtree (those with Ahnentafel indices.) – If data is associated also with nonterminal blocks, store it in level order. – Use Ahnentafel indices to control recursive-descent algorithms [17]. – Ahnentafel indices are monotonic across any row or column, so bounds checking remains available via masking with evenBits or oddBits.
Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free
779
An early context for implementation of this design is H ASKELL whose higherdimension array aggregates (also, array comprehensions) do not imply any particular internal representation; that is, no code depends on a particular ordering. Moreover, asynchronous, parallel algorithms are implict in that style. Not accidentally, implementations of H ASKELL and other functional languages do a good job implementing recursion in preference to iteration. Nevertheless, the algebra of indices implemented in the strength reduction and loop unrolling of F ORTRAN and C compilers does not yet have an analog under recursion. Morton order, with its own algebra of indexing and recursion unfolding, offers this leverage and opens access to new scientific algorithms with functional style.
3 CARTESIAN INDEXING AND MORTON ORDERING The following techniques for cartesian indexing of Morton-order arrays seem to be hardly known, a fact that is unfortunate because they make the structure useful for blocking matrices even if only used with cartesian indexing. In particular, this section newly shows how to compile row and column traversals (of blocks at any level of the tree) with the reductions in operator strength associated with optimizing compilers. 3.1 DILATED INTEGERS The algebra of dilated integers is surprisingly old. Tocher outlined it in 1954 and under similar constraints to those again motivating us: non-flat memory with access time dependent on locality, and nearby information more rapidly accessible [15, p. 53–55]. But how the size of the memory has changed! Tocher needed fast access into a 32 32 32 boolean array stored on a drum (4kB!). Schrack shows how to effect efficient cartesian indexing from row i and column j indices into Morton-order matrices [14]. The trick is to represent i and j as dilated integers, with information stored only in every other bit. w 1 w 1 Definition 7 The even-dilated representation of j = k=0 jk 2k is k=0 jk 4k , w 1 !| : The odd-dilated representation of i = k=0 ik 2k is 2 !{ , denoted { : denoted The arrows suggest the justification of the meaningful bits in either dilated representation. For example, the right arrow suggests rightmost Bit 0 and even kin.
P
P
P
Theorem 5. A matrix of m rows and n columns should be allocated as a sequential ! block of m 1 + n 1 + 1 scalar addresses.
This value is, of course, the Morton index of the southeast-most element of the matrix, plus one for (the northwest-most) one whose Morton index is 0. Not all of that sequence need be active; undefined data at the idle addresses will remain resident in the lowest level of the memory hierarchy. Only data in the active addresses will ever migrate to cache. Theorem 6. [14] If “” is read as semantic equivalence and “=” denotes equality on integer representations, then for unsigned integers
!{ = !| ) (i = j ) ( { = | ) (
!{ < !| ) (i < j ) ( { < | ): (
780
David S. Wise
So comparison of dilated integers is effected by the same processor commands as those for ordinary integers. Definition 8 The following theorems apply to w-bit 2’s-complement integers. The infix operator ^ indicates bitwise conjunction; _ denotes bitwise disjunction. ! Instead of representing i and j internally, represent them as { and | .
! Theorem 7. The Morton index for the hi j ith element of a matrix is { _ | , ! equivalently { + | :
Often (normalized dilated integers [14]) addition will be used instead of disjunction to associate at compile time with an adjacent addition. Addition and subtraction of dilated integers can be performed with a couple of minor instructions. ! ! Definition 9 Addition (+ +) and subtraction ( ) of even- and odd-dilated integers:
! ! | ! ! n = j n !! ! ! | + n = j + n
{ n = i n: = { + n i + n:
Theorem 8. Subtraction, addition, constant addition, and shifts on dilated integers:
! ! | ! ! ! ! n = ( | n)^ b
! ! | + ! ! ! ! n = ( | + b + n)^ b ! ! ! !{ + ! ! c = { (c) ! ! b = (1) ! !{ <<(2k) i<
{ n = ( { n)^ b ! { +n = ( { + b + n)^ b { + c = { (c)
[14] [14]
b = 1 i<
Theorem 8 and 6 suggest that the C loop for (int i=0; i
be compiled to int nn= n and for (int ii=0; ii
So, if i and j are not represented literally as integers, but translated by the ! { and | , the resulting object code can be just a simple compiler to their images, homomorphic image of what the programmer expected. Code like the following source might be demanded from the programmer, but it would be better introduced via a transformation by the helpful compiler: #define evenBits ((unsigned int) -1)/3) #define oddBits (evenBits <<1) #define oddIncrement(i) (i= ((i - oddBits) & oddBits)) #define evenIncrement(j) (j= ((j - evenBits) & evenBits)) ... for (i = 0; i< rowCountOdd ; oddIncrement(i)) for (j = 0; j< colCountEven; evenIncrement(j)) for (k = 0; k< pCountEven ; evenIncrement(k)) c[i + j] += a[i + k] * b[j + 2*k];
Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free
!
781
The index computations of k and k in the innermost loop above reduce to three RISC instructions, plus two to sum the matrix-element addresses. Although fast and constant time, this is still half-again what is available from column-major representation. With integer/floating processors tuned to columnmajor this difference may once have mattered but now it doesn’t, with register operations so fast relative to memory access. It becomes important, though, for compilers to provide this arithmetic as loop control. For three and higher dimensions, it seems best to implement arithmetic only for one type of dilated integer, and to translate other representations from it. Dilated integers simplify input and output of Morton-ordered matrices in human-readable raster order [3]. Such convenience is not computationally significant because I/O delays dominate that indexing, but it may be politically important just to make this matrix representation accessible to the programmers who need to experiment with it. 3.2 SPACE AND BOUNDS Theorem 5 tells how much address space an m n matrix occupies, usually more than the mn positions containing data. As mentioned above and observed ! in experiments, if the difference, m 1 + n 1 + 1 mn is large then most of it will never move into faster memory. Moreover, the larger the difference, the more remote (and cheaper) will be the bulk of the addresses allocated. The size of that excess space for a rectangular matrix depends the number of elements and also on the aspect ratio between the census of rows and columns. In block algorithms, cache hits make it better to iterate through Mortonorder sequentially. For instance, if b is the initial index of a block from matrix A of size n=4p that is to be zeroed, it is better to initialize it with a single, localfor (int i=0; i
For Ahnentafel indices especially, the compiler can do even more. It will precompute two vectors of bounds on a row or column index at each level of the quadtree. The first, rowBound[], is a bound on the perimeter of the matrix, to preclude access to southern and eastern padding. The second, rowDense[], contains bounds on interior Ahnentafel indices that are north and west of this perimeter which, once passed, obviate the need for further bounds checking at any subtree/subblock. Allocate two vectors of size h to contain row bounds on the Ahnentafel indices at each level, where h is the height of the tree. For both, the hth entry is r where r is the number of rows. Thereafter,
1 ) >>2;
rowBound[level-1] = rowBound[level]>>2; rowDense[level-1] = (rowDense[level]
With c columns the column bounds are computed from ! c similarly, as even dilated integers; we usually test row and column limits simultaneously.
782
David S. Wise
One might not be overly precise on Ahnentafel bounds at the perimeter. Because extra space is often allocated to the south and east anyway, it has been observed to be cheaper to round the actual bounds on the matrix up to, say, the next multiple of eight and then to treat the margins as “active” padding. For a recursive block algorithm, the alternative is to detect blocks of size four, two, and one where there will be too few operations; with the smallest block at order eight, operations on it can be dispatched as unconditional, straight-line, superscalar code. That is, we choose to treat padding in small marginal blocks as sentinels, initialized so each can participate in the gross algorithm without affecting its net result. Typically this is zero to the south or east, and the identity matrix to the southeast: for (int jj=0; jj
The compiler also needs to use the algebra of dilated integers from Section 3.1 to unfold recursions and to unroll loops effectively; it becomes another kind of reduction in the strength of indexing. Finally, symmetric matrices and matrix transpose are also easy with Morton or Ahnentafel indices. If m is either kind of index, then the index of its reflected element or (untransposed) block is quickly computed by exchanging its even and odd bits in a dosido: ( (m &evenBits) <<1) + ( (m &oddBits) >>1)
4 CONCLUSION We have already demonstrated vast improvements using both the cartesian and Ahentafel indexing schemes described here. A good quadtree algorithm uses recursive descent on Ahnentafel indices, so there is no need to preselect a block size to fit one—or any—level of cache; the algorithm simply accelerates when a block fits [8]. None of the indices, themselves, need be stacked. Each can be shifted right to effect a stack pop or incremented to refer to a sibling. Still, innermost recursions need to be unfolded [2], just as the C compiler unrolls loops, in order to obtain straight-line code for superscalar processors. See [18] for more details; all the speed improvements are due to good locality [5]. The formalism presented here shows how to implement Morton-order matrices, with efficient algorithms for the group of dilated integers for the program that uses cartesian indexing, or with Ahnentafel indexing for the one that uses recursive descent on quadtrees. We have built a propotype compiler to translate C programs using row-major matrices and cartesian indices to Morton-order using dilated indices. We had already demonstrated the ease of tree-wise scheduling parallel processors in [7], and we continue to search for similar quadtree algorithms [17, 6]. It remains to close the gap between these efforts: on the one hand to improve compilers to optimize indexing on Morton-ordered matrices (e.g. unfolding Ahnentafel–controlled recursions and unrolling dilated-integer–controlled loops). And, on the other hand, to exercise Morton-order matrices under both styles, comparing performance of both families of algorithms by comingling them in a single environment as two libraries of interchangeable modules. Our underlying goal remains to uncover better algorithms that use more locality and balanced parallelism to solve matrix problems faster.
Ahnentafel Indexing into Morton-Ordered Arrays, or Matrix Locality for Free
783
Acknowledgement: Thanks to Christian Weiss for helpful comments and special thanks to Hanan Samet who provided the early references.
References 1. J. Backus. The history of F ORTRAN I, II, and III. In R. L. Wexelblat (ed.), History of Program. Languages, New York, Academic Press (1981), 25–45. Also preprinted in SIGPLAN Not. 13, 8 (1978 August), 165–180. 2. R. M. Burstall & J. Darlington. A transformation system for developing recursive programs. J.ACM 24, 1 (1997 January), 44–67. 3. F. W. Burton, V. J. Kollias, J. G. Kollias. Real-time raster-to-quadtree and quadtree-toraster conversion algorithms with modest storage requirements. Angew. Informatik 4 (1986), 170–174. 4. H. G. Cragon A historical note on binary tree. SIGARCH Comput. Archit. News 18, 4 (1990 December), 3. 5. D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, & T. von Eicken. LogP: a practical model of parallel computation. Commun. ACM 39, 11 (1996 November), 78–85. 6. J. Frens. Matrix Factorization Using a Block-Recursive Structure and Block-Recursive Algorithms. PhD dissertation, Indiana University, Bloomington (in progress). 7. J. Frens & D. S. Wise. Auto-blocking matrix multiplication, or tracking BLAS3 performance from source code. Proc. 1997 ACM Symp. on Principles and Practice of Parallel Program., SIGPLAN Not. 32, 7 (1997 July), 206–216. 8. M. Frigo, C. E. Leiserson, H. Prokop, & S. Ramachandran. Cache-oblivious algorithms, Extended abstract. Lab for Computer Science, M.I.T., Cambridge, MA (1999 May). http://supertech.lcs.mit.edu/cilk/papers/abstracts/FrigoLePr99.html 9. Y. C. Hu, S. L. Johnsson & S.H. Teng. High performance Fortran for highly irregular problems. Proc. 6th ACM SIGPLAN Symp. on Principles and Practice of Parallel Program., SIGPLAN Not. 32, 7 (1977 July), 13–24. 10. D. E. Knuth. The Art of Computer Programming I, Fundamental Algorithms (3rd ed.), Reading, MA: Addison-Wesley, (1997). 11. G. M. Morton. A computer oriented geodetic data base and a new technique in file sequencing. Ottawa, Ontario: IBM Ltd. (1966 March 1). 12. J. A. Orenstein & T. H. Merrett. A class of data structures for associative searching. Proc. 3rd ACM SIGACT–SIGMOD Symp. on Princ. of Database Systems (1984), 181–190. 13. H. Samet. The Design and Analysis of Spatial Data Structures Reading, MA: AddisonWesley, (1990), x2.7. 14. G. Schrack. Finding neighbors of equal size in linear quadtrees and octrees in constant time. CVGIP: Image Underst. 55, 3 (1992 May), 221-230. 15. K. D. Tocher. The application of automatic computers to sampling experiments. J. Roy. Statist. Soc. Ser. B 16, 1 (1954), 39–61. 16. M. S. Warren. & J. K. Salmon. A parallel hashed oct-tree N-body problem. Proc. Supercomputing ’93. Los Alamitos, CA: IEEE Computer Society Press (1993), 12–21. 17. D. S. Wise. Undulant block elimination and integer-preserving matrix inversion. Sci. Comput. Program. 33, 1 (1999 January), 29–85. http://www.cs.indiana.edu/ftp/techreports/TR418.html
18. D. S. Wise & J. Frens. Morton-order matrices deserve compilers’ support. Technical Report 533, Computer Science Dept, Indiana University (1999 November). http://www.cs.indiana.edu/ftp/techreports/TR533.html
An Efficient Parallel Linear Solver with a Cascadic Conjugate Gradient Method: Experience with Reality Peter Gottschling1 and Wolfgang E. Nagel2 1
2
GMD FIRST, Berlin, Germany, [email protected] Center for High Performance Computing (ZHR), Dresden University of Technology, Germany, [email protected]
Abstract. To solve large systems of linear equations with sparse matrices in parallel, there are three factors that contribute to the computing time: the numerical efficiency, the floating point performance, and the scalability. In this paper, we mainly consider the floating point performance. For large linear systems, multi-level techniques, like the cascadic conjugate gradient method (CCG), require significantly less operations than single-level methods. On the other hand, they are considered less efficient with regard to performance and limited in parallelization. Therefore, to achieve an efficient, massively parallel multi-level solver, we used the fastest available communication and revised the whole computation. The performance improvements led to a parallel solver which is able to solve a linear system with more than 16 million unknowns in 0.77 seconds on 256 PEs of Cray T3E. This corresponds to an overall performance of 10.34 GFLOPS. Keywords. Floating Point Performance, RISC Processors, Matrix Sparsity Pattern, Cascadic Conjugate Gradient Method
1
Introduction
The solution of large systems of linear equations with sparse matrices plays an important role in many simulation applications and in some of the so-called ‘grand challenge’ problems. Three components can be identified which determine the necessary time to solve a linear system in parallel: the numerical efficiency, the floating point performance, and the scalability. First of all, one should examine whether it is possible to apply a multi-level solver – like multigrid or cascadic conjugate gradient method – to the investigated problem. Single-level solvers – like Gauss-Seidel – most often converge poorly for large linear systems. Though the floating point performance and the
Part of the work was done while the author was a research associate at ZHR in 1999.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 784–794, 2000. c Springer-Verlag Berlin Heidelberg 2000
An Efficient Parallel Linear Solver with a CCG Method
785
parallel efficiency is sometimes better, this cannot compensate for the numerical deficiencies. The usually better parallelism is explained by the larger ratio between the number of operations and the size of transferred data between the processors because multi-level methods work on several linear systems with different dimensions. Multi-level techniques use single-level methods on the individual levels. Therefore, the floating point performance on several levels is similar to that of the single-level techniques used. On the smaller linear systems, the performance is sometimes even higher because of better cache reuse. On the other hand, the additional operations on multi-level techniques – the transfer operations between the grids – perform poorly due to indirect and irregular memory accesses. Nevertheless, the computing time of the transfer operations is usually quite short. For that reason, a multi-level technique can sometimes perform similarly as the containing single-level method. The systems of linear equations, which we considered in our investigations, originate from the discretization of the ground-water flow equation. The strong variation of the parameters of this partial differential equation causes strongly varying coefficients in the matrix of the linear system. Despite the variation of the coefficients and the largeness of the linear system, the cascadic conjugate gradient method with an algebraically generated hierarchy of linear systems enables good convergence [5]. The paper is organized as follows. In section 2, we consider different types of sparse matrices. For these matrix types, the counter-movement of the applicability of discretization schemes and the possibilities of performance tuning is shown. The communication expense is covered in the subsequent section. Section 4 presents the optimization targets for the arithmetic part used in the parallel solver. Different implementations of the matrix vector multiplication are compared in section 5. The last section describes the optimization of the conjugate gradient method.
2
Sparsity Patterns of Matrices
Often, systems of linear equations with sparse matrices originate from discretized partial differential equations. The type of discretization determines the sparsity pattern of the matrix. In this paper, we distinguish three types of matrices. Structured Matrices: These matrices (fig. 1a) are characterized by a set of constants C = {c1 , c2 , . . . cm } so that j − i ∈ / C → aij = 0. This means that only matrix elements with a certain distance from the diagonal can be non-zero. Matrices of this form arise in equidistant discretizations of rectangular or cuboid domains. Matrix vector products with this type can be programmed with simple loops using constant offsets. Therefore, different optimizations like loop unrolling and blocking (cf. [4]), are applicable. Locally Structured Matrices: For the second type of sparsity, the expression ‘locally structured’ matrices (fig. 1b) is introduced. Here, several sets
786
Peter Gottschling and Wolfgang E. Nagel
of differences C1 , C2 , . . . can be defined where each set is valid for a certain interval of lines. The sparsity pattern can be expressed in an implementation by structural specifications that correspond with these intervals of matrix lines. This representation at least allows floating point optimizations within the intervals. Locally structured matrices originate, for instance, from the equidistant discretization of domains with irregular borders. Unstructured Matrices: The most general kind of sparse matrices are unstructured matrices (fig. 1c, from [6]). No assumptions about the sparsity pattern are made here. Thus, arbitrary discretizations are permitted. On the other hand, each matrix element must be treated separately in a matrix vector multiplication and the performance is lower for that reason. Nevertheless, discretizations that are adapted to the problem are often necessary and the lower speed is justified by a significant reduction of the equation size. 0
0
0
50
50
50
100 100
100 150
150 150 200
200 200
250
0
50
100 nz = 1072
150
(a) structured
200
0
50
100
150 nz = 1282
200
250
(b) locally struct.
0
50
100 nz = 8003
150
200
(c) unstructured
Fig. 1. Types of sparsity patterns In our work, we consider locally structured matrices because of their importance in ground-water flow simulations. The two other matrix types are used in numerous other projects (structured matrices e.g. in [1] and unstructured ones e.g. in [10]).
3
Communication Expense
The computation of one step of the conjugate gradient method involves one exchange of the inner borders in the matrix vector multiplication and two reductions in dot products. Further exchanges of the inner borders are necessary in some preconditionings (e.g. incomplete Cholesky factorization). Moreover, the implementation of the termination criterion requires an additional reduction unless the values of the conjugate gradient method (CG) are used (cf. [2]). The data dependencies in the conjugate gradient method allow the simultaneous reduction of the termination criterion and of the first dot product in the next iteration step of the CG method. Since the communication latency is
An Efficient Parallel Linear Solver with a CCG Method
787
rather large compared to the bandwidth on every parallel computer and computer network the global reduction of two values takes roughly the same time as the reduction of one value. To minimize the expense of the partitioning, the domain (on the fine grid) was decomposed by coordinate section in Px × Py subdomains on P = Px · Py processors. The decomposition was passed to the coarser grid where the boundaries were slightly adapted to conserve the load balance (cf. [5]). A significant decrease of the communication time can sometimes be established by replacing portable communication procedures with proprietary ones. Figure 2 shows the time line, visualized by VAMPIR [7], of two iterations of the CG-method on a linear system with about 16,000 unknowns on Cray T3E. In this implementation based on the MPI library, the interprocessor communication, represented by lines, is the dominant part. Implementing the communication with equivalent shmem functions (shmem double get and shmem double sum to all) clearly shortens the communication time so that the execution time of one iteration is reduced from 1.437 ms to 0.632 ms (figure 3).
Fig. 2. Two iteration steps with MPI communication Although solving a linear system with 16,000 unknowns is not very interesting to parallel computing, there are two reasons for accelerating the communication. Firstly, the communication becomes more important for increasing numbers of processors, even for large problems, and secondly, multilevel solvers spend most of their time on communication (delay mainly due to latency issues) while solving the coarse grid equations. Since we are interested in multilevel solvers on many processors, an optimized communication is very important.
4
Optimization Targets to Improve the Floating Point Performance on RISC Processors
To increase the floating point performance of a RISC processor for our application, we followed four goals. We focused on the DEC Alpha 21164 of Cray T3E,
788
Peter Gottschling and Wolfgang E. Nagel
Fig. 3. Two corresponding iterations with shmem communication nevertheless, the optimization steps should be helpful on other superscalar RISC processors, too. Decrease of the Memory Accesses by Cache Reuse: The main bottleneck of fast RISC processors is the rather slow main memory access. On Cray T3E, for instance, processors with 300-600 MHz and 600-1200 MFLOPS face main memories with 75 MHz. To yield floating point performance near peak performance, it is necessary (but not sufficient) to reuse data in registers or in the primary cache as often as possible so that memory accesses are not relevant to the execution time. Seidl [9] has shown by examples that the acceleration of algorithms can be predetermined from the reduced probability of memory accesses. Consecutive Memory Accesses in Increasing Order: Loading a data item into the cache is realized by loading a certain amount of memory, called cache line, which is typically larger than the data item itself. If referred subsequently, the next data items are already in the cache with some probability. On Cray T3E, additional benefit can result from the stream buffers. This is a mechanism that looks for increasing memory accesses and, if recognized, loads successive cache lines into a special register where they can be rapidly loaded into the second level cache (cf. [8]). Operations on large vectors are, from the authors’ experience, typically computed twice as fast with stream buffers. Independent Operations: Implementations with many independent operations allow the efficient use of multiple pipelines. The number of independent operations is very often increased by loop unrolling. Reduction of Divide Operations: The divisions of floating point numbers are not as fast as additions and multiplications on most processors. In addition, they cause pipeline blocking on the DEC Alpha 21164.
An Efficient Parallel Linear Solver with a CCG Method
5
789
Matrix Vector Multiplication
The importance of the matrix vector multiplication is based on the fact that in our solver half of the floating point operations are performed in this section. In the original implementation, the multiplication was calculated in two steps. At first, the result vector was initialized with the product of the input vector and the diagonal of the matrix. Then, the components of the result vector were incremented by non-zero matrix entries outside the diagonal multiplied with the elements of the input vector. In this way, sub-vectors of maximal length were used and the loop overhead was minimized. As an example, we considered a multiplication with a vector containing about 126,000 elements and an appropriate matrix, so that about 1,134,000 floating point operations had to be executed. The original implementation required 34.9 ms to compute the multiplication on the 600 MHz processor applying stream buffers. This corresponds to 32.5 MFLOPS. While in the second section approximately 8 floating operations per unknown were computed in relation to 4 to 6 memory accesses (depending on the extension and form of the domain and the size of the second level cache), there are three memory accesses per unknown with only one operation performed in the first section. This unfavorable ratio between operations and memory accesses resulted in a very slow computation. Once more looking at Cray T3E, on the 600 MHz processor the initialization step performed with 8.4 MFLOPS without stream buffers where the performance was only augmented to 18.3 MFLOPS by using the stream buffers. To emphasize the importance of the memory bandwidth, the same operation was performed with 7.5 MFLOPS and 15 MFLOPS respectively on a 300 MHz processor. To avoid this slow computation, the initialization of the output vector was included in the incrementing step. The product of the diagonal and the input vector was then only computed for those elements where the output vector was incremented in the next moment. This modified implementation saved 2 memory accesses per unknown for loading the vectors (unless the sub-vectors involved in the incrementing section were too large for the cache) but additional overhead was produced to control which components of the result vector must be initialized and due to dividing the initialization into several sections. On the example problem, this implementation took 26.9 ms for the multiplication (42.2 MFLOPS). Experiments with structured matrices have shown that matrix vector multiplications were significantly faster if the elements of the result vector were computed explicitly (v[i]= a[i][i]*q[i] + a[i][j]*q[j] + · · ·) instead of by incrementing as described above. For locally structured matrices, it is more complicated to apply the explicit calculation. In comparison with the former implementation, the loops are shorter and they need more preparations. Altogether, the loop overhead is more than doubled, although the computing time is decreased by 59.5 per cent to 14.1 ms on the example problem (80.2 MFLOPS). Another performance optimization is applicable if the multiplication is part of the conjugate gradient method. There, the dot product of the input and the output vector is used. Since two memory accesses are necessary per floating point
790
Peter Gottschling and Wolfgang E. Nagel
operation this calculation is slow (there is a bottleneck on the scalar value, too). For the considered vector size, the dot product requires 5.1 ms. Including this dot product in the matrix vector multiplication, as proposed in [3], saves the memory accesses. Consequently, most of the computing time for the dot product is saved (the scalar value is less critical here because several floating point operations are performed between two increments). In a parallel implementation, attention has to be paid to the inner boundaries. To enable a multiplication with a symmetric matrix, vector components q[j] that are assigned to other processors are considered in an extra computation (v[i]+= a[i][j]*q[j]). In this section, the dot product is calculated as dot+= a[i][j]*q[j]*q[i] while the computation with the symmetric part of the matrix is dot+= v[i]*q[i]. Altogether, the execution time of the combined computation was 14.7 ms which corresponds to 93.9 MFLOPS.
6
Iteration Steps of the Conjugate Gradient Method
The conjugate gradient method itself only consists of vector operations. Since vector operations are characterized by a poor ratio between the number of floating point operations and the number of memory accesses, the performance is rather low. To improve this ratio, the vector operations have to be combined to compute as many floating point operations as possible on a vector component while it resides in the cache. Whereas calculations depending on a vector can start as soon as a part of the vector is available, calculations depending on a scalar value resulting from a vector reduction must wait until the reduction is finished. Although the conjugate gradient method is well known, we present the iteration step as a C++ program in table 1 for a better illustration. The startup phase is omitted because it does not provide further optimization opportunities. The extension of the CG method to the cascadic conjugate gradient method is quite simple. Starting on the coarsest grid, the CG method is computed on every grid. When the termination criterion is fulfilled on a certain grid the vector x is interpolated to the next finest grid and is used as an initial guess for the CG method on that grid. The program can be implemented in this form by using templates in C++. The use of templates is critical because each operator is computed separately. For this reason, temporary vectors are required, which produce overhead for additional memory allocations and memory accesses, unless advanced numerical template libraries like Blitz++ [11] are used. In the calculation of dot products, the scalar value represents a bottleneck for superscalar processors. Since the addition is associative (in exact arithmetic), the products of the vector components can be summed into several values within the loop and added at the end. For the vector size considered in the previous section, the computing time was reduced from 5.1 ms to 3.1 ms, which corresponds to an increase of the performance from 49.2 MFLOPS to 77.8 MFLOPS. In our
An Efficient Parallel Linear Solver with a CCG Method vector<double> double
791
x, v, q, r, w; alpha, delta, gamma, gamma old;
int cg iteration () { double delta local, gamma local;
}
q= (gamma/gamma old) ∗ q + w; exchange inner borders (q); v= a mult (q); delta local= dot (v, q); delta= reduce (delta local); alpha= gamma / delta; x+= alpha ∗ q; r-= alpha ∗ v; w= thepc→f (r); gamma local= dot (w, r); gamma= reduce (gamma local); return thepc→f ();
// requires communication
// requires communication
// chosen preconditioning (possibly requires comm.) // requires communication // chosen termination criterion (usually requires comm.)
Table 1. Iteration of the conjugate gradient method
investigations, changing the order of the summation did not noticeably influence the exactness of the floating point arithmetic. Inlining a particular preconditioning and a particular termination criterion saves the function calls and allows further optimization. For the investigated systems of linear equations, the diagonal preconditioning w = D−1 r represented the best compromise between the numerical properties and the computational and communicational expense. Among different termination criteria, it has been shown that for equations with strongly varying coefficients, the diagonally preconditioned residual D−1 r enables the best error estimation. So, the commitment to this combination permits the elimination of redundant calculations. Since the element-wise division is rather slow and the vector components are divided by constant values, it is worthwhile to store the inverse of the diagonal matrix. An element-wise multiplication then replaces the element-wise division at the price of an extra vector and some additional computation before starting the linear solver. To reduce the number of memory accesses, the calculations of r-= alpha ∗ v, w= adinv ∗ r, dot (w, r) and dot (w, w) can be combined in one loop, where adinv is a vector with adinv[i] = 1/aii . Of course, this loop can be unrolled, too. With the simultaneous reduction of the local computations of dot (w, r) and dot (w, w) and the modifications described above, the iteration of the conjugate gradient method looks as shown in table 2. As an example, a linear system with more than 16 million unknowns was solved on 32 processors. Here, the execution time of a single iteration step on the finest level was decreased from 432 ms in the original version to 206 ms in the accelerated one. Furthermore, it has been shown that the variations of the computing time between the different processors gain in significance due to the performance tuning. Although the number of operations are equally distributed among the processors, the computing time varies noticeably. As a consequence,
792 int
Peter Gottschling and Wolfgang E. Nagel nupo;
// number of points corresponds to vector size
int cg iteration jacobi () { double delta s, delta a, delta local, stop gamma local [2], stop gamma [2], s0, s1, s2, s3, g0, g1, g2, g3, tmp0, tmp1, tmp2, tmp3; int i, nupo4= nupo 2 2;
}
scadd (gamma/gamma old, q, w); // requires communication exchange inner borders (q); delta local= a mult (v, q); // requires communication delta= reduce (delta local); alpha= gamma / delta; scadd (x, alpha, q); s0= s1= s2= s3= g0= g1= g2= g3= 0.0; for (i= 0; i < nupo4; i+= 4) {tmp0= (r [i]-= alpha ∗ v [i]) ∗ adinv [i]; s0+= tmp0 ∗ tmp0; g0+= r [i] ∗ tmp0; tmp1= (r [i+1]-= alpha ∗ v [i+1]) ∗ adinv [i+1]; s1+= tmp1 ∗ tmp1; g1+= r [i+1] ∗ tmp1; tmp2= (r [i+2]-= alpha ∗ v [i+2]) ∗ adinv [i+2]; s2+= tmp2 ∗ tmp2; g2+= r [i+2] ∗ tmp2; tmp3= (r [i+3]-= alpha ∗ v [i+3]) ∗ adinv [i+3]; s3+= tmp3 ∗ tmp3; g3+= r [i+3] ∗ tmp3; w [i]= tmp0; w [i+1]= tmp1; w [i+2]= tmp2; w [i+3]= tmp3;} for (i= nupo4; i < nupo; i++) {w [i]= tmp0= (r [i]-= alpha ∗ v [i]) ∗ adinv [i]; s0+= tmp0 ∗ tmp0; g0+= r [i] ∗ tmp0;} stop gamma local [0]= s0 + s1 + s2 + s3; stop gamma local [1]= g0 + g1 + g2 + g3; // requires communication reduce2 (stop gamma local, stop gamma); gamma old= gamma; gamma= stop gamma [1]; return sqrt (stop gamma [0]) < thetc→epsilon;
Table 2. Iteration of the specialized and accelerated version the parallel execution time was affected more by the loss of synchronism than by the communication expense on Cray T3E. On two processors, where the balance of the computing time and the communication are less significant, the accelerated implementation achieved a performance of 66.2 MFLOPS per processor. On 256 processors,1 the performance per processor was more than 40 MFLOPS, leading to an overall performance of 10.34 GFLOPS. To solve the linear system with 16 million unknowns, the solver based on the cascadic conjugate gradient method and on algebraically generated coarse grid equations required 9 iterations of the CG method on the finest grid and 15 iterations on the other grids [5]. So, the linear system was solved in 0.77 seconds. Since many simulations of physical processes – like ground-water flow – are based on the solution of many large linear systems, a fast parallel multi-level solver can save a lot of computing time. On the other hand, the acceleration of the solver may also be used to increase the simulation complexity in order to improve the simulation results.
7
Conclusion
Although on modern computer architectures the memory bandwidth is still by far too low with regard to the processor speed, this relation is expected to get worse in the near future (cf. [4, p. 34]). For this reason, the primary performance optimization target is reducing the number of main memory accesses. Therefore, 1
The authors would like to thank Prof. Hoßfeld from John von Neumann-Institute for Computing (NIC-ZAM) for providing capacity on Cray T3E.
An Efficient Parallel Linear Solver with a CCG Method
793
the computations have to be reordered so that as many operations as possible are performed on a data item while it resides in cache. Fortunately, there is a relatively small kernel in many numerical applications where most of the computing time is spent. In this case, the performance tuning efforts can be restricted to this kernel so that the expense to improve the performance is usually low compared with the expense for the program development. In scientific applications described by partial differential equations most of the execution time is usually spent on the solution of linear systems. Therefore, the acceleration of the linear solver can yield a great profit. First of all, it should be examined whether a multi-level method can be applied to the respective type of the linear system. In this case, the number of operations can often be decreased by several orders of magnitude by using a different algorithm. For the considered linear systems, which originate from the ground-water flow equation, the difficulty lies in the strong variation of the coefficients (up to 108 ). Nevertheless, the cascadic conjugate gradient method with algebraically generated coarse grid equations converged well for the examined linear systems. To implement the parallel CCG solver efficiently, the whole iteration step of the CG method was optimized with regard to the performance, including the preconditioning and the termination criterion. A special matrix type is introduced which allows the demanded applicability of discretization schemes and provides more performance optimization possibilities than unstructured matrices. In addition, the communication time was significantly shortened by changing from the portable MPI library to the proprietary shmem library. So, even multilevel solvers can work efficiently on up to several hundred PEs on Cray T3E. Altogether, the fast communication, the high floating point performance and the good convergence enabled the solution of a linear system with over 16 million unknowns in less than a second.
References [1] Manfred Alef. Implementation of a multigrid algorithm on suprenum and other systems. Parallel Computing, 20:1547–1557, 1994. [2] Peter Deuflhard. Cascadic conjugate gradient methods for elliptic partial differential equations. algorithm and numerical results. In D. Keyes and J. Xu, editors, Proc. of the 7th Int. Conf. on Domain Decomp. Methods 1993, pages 29–42, 1994. [3] Jack J. Dongarra, Iain S. Duff, Danny C. Sorensen, and Henk A. van der Vorst. Numerical Linear Algebra for High-Performance Computers. SIAM, Philadelphia, 1998. [4] Kevin Dowd and Charles R. Severance. High Performance Computing. O’Reilly, Sebastopol, second edition, 1998. [5] Peter Gottschling. Efficient parallel computation of the discretized ground-water flow equation with algebraic multigrid methods (in german). Phd thesis, TU Dresden, in preparation. [6] Uwe Lehmann. Festigkeitsuntersuchung von Eisenbahnfahrwegen auf Parallelrechnern. Diplom, TU Dresden, 1999. [7] W.E. Nagel and A. Arnold. Performance visualization of environment. Technical report, Forschungszentrum J¨ ulich, 1995. http://www.fz-juelich.de/zam/PT/ReDec/SoftTools/PARtools/PARvis.html.
794
Peter Gottschling and Wolfgang E. Nagel
[8] Wolfgang E. Nagel. The new massively parallel computer cray t3e in spring 1996: Experience with virginity (in german). In Hans-Werner Meuer, editor, Supercomputer 1996, pages 92–107. K. G. Saur, 1996. [9] Stephan Seidl. Code crumpling: A straight technique to improve loop performance on cache based systems. In Proc. 5th Eur. SGI/Cray MPP Workshop, 1999. [10] J. Stiller, K. Boryczko, and W.E. Nagel. A new approach for parallel multigrid adaption. In Proc. 9th SIAM Conf. on Par. Proc. for Sci. Comp., 1999. [11] Todd Veldhuizen et al. Blitz++. www://oonumerics.org/blitz/.
A Fast Solver for Convection Diffusion Equations Based on Nested Dissection with Incomplete Elimination Michael Bader and Christoph Zenger TU M¨ unchen, Lehrstuhl f¨ ur Informatik V, D-80290 M¨ unchen, Germany {bader,zenger}@in.tum.de http://www5.in.tum.de/
Abstract. We present an approach for the efficient parallel solution of convection diffusion equations. Based on iterative nested dissection techniques [1] we extended these existing iterative algorithms to a solver based on nested dissection with incomplete elimination of the unknowns. Our elimination strategy is derived from physical properties of the convection diffusion equation, but is independent of the actual discretized operator. The resulting algorithm has a memory requirement that grows linearly with the number of unknowns. This also holds for the computational cost of the setup of the nested dissection structure and the individual relaxation cycles. We present numerical examples that indicate that the number of iterations needed to solve a convection diffusion equation grows only slightly with the number of unknowns, but is widely independent of the type and strength of the convection field.
1
Introduction
We are searching for an efficient parallel solver for the linear systems of equations arising from the discretization of the convection diffusion equation − u + a(x, y) · ∇u = f.
(1)
In this paper, we focus on standard finite difference or finite element discretizations on rectangular Cartesian grids resulting in the standard five or nine point discretization stencils. To obtain a truly efficient solver for convection diffusion equations one has to overcome three main problems. The performance of the solver should be independent of the convection field a(x, y). The solver should be able to treat arbitrary geometries of the computational domain with equal efficiency. Finally, it should be possible to produce efficient parallel implementations of the solver to take optimal advantage of high performance computers. Solvers based on geometric multigrid methods are usually quite easy to parallelize due to their structured coarse grids. On the negative side they often have difficulties in treating complicated geometries. Moreover, they often require a A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 795–805, 2000. c Springer-Verlag Berlin Heidelberg 2000
796
Michael Bader and Christoph Zenger
special treatment of curved or circular flow fields, which sometimes seems to impair the parallelization properties. Algebraic multigrid (AMG) methods are usually very robust with respect to the strength or type of the convection or even the geometry. As AMG chooses its coarse grids according to the operator and the geometry, it often produces excellent convergence results. On the other hand, the automatic grid generation often makes it difficult to produce efficient parallel implementations of the solvers. This is especially true for the construction of the different coarse grids itself, because the most commonly used strategy [5] for the coarse grid selection is an inherently sequential process. Our goal is therefore to find a fast (i.e. with multigrid or multigrid-like performance) method that is easy to parallelize and allows the treatment of an arbitrary geometry of the computational domain. We base our approach on previous work by H¨ uttl [1] and Ebner [2], whose algorithms combine ideas from domain decomposition and nested dissection techniques to iterative methods based on recursive substructuring of the computational domain. After giving a short comprehension of these techniques in section 2, we will, in section 3, present our extension of these approaches by an incomplete elimination of the most significant couplings in the system matrices. Finally, in section 4 we will show some numerical examples that indicate the promising potential of our approach.
2 2.1
The Nested Dissection Approach Nested Dissection as a Direct Solver
The nested dissection method was introduced by George [4] as a direct solver for the sparse linear systems of equations resulting from the discretization of PDEs with finite elements. It it based on the recursive substructuring of the computational domain by which it minimizes the fill-in that usually occurs during the solution process. Throughout this paper we will divide the nested dissection algorithm into three passes: the recursive substructuring, the static condensation, and the solution. Pass 1: Recursive Substructuring In the first (top-down) pass the computational domain is recursively divided into two or more subdomains that are connected via a separator. As shown in figure 1, we will use a separation by alternate bisection throughout this paper. Compared to substructuring in four (or even more) subdomains the alternate bisection produces linear systems of equations with a slightly smaller number of unknowns, which gives advantages in parallelization. On each subdomain, we classify the unknowns into the set I of the inner unknowns and the set E of the outer unknowns. The inner unknowns are those unknowns on the separator that do not belong to the separator of a parent
A Fast Solver for Convection Diffusion Equations
797
Fig. 1. Recursive substructuring by alternate bisection domain. The outer unknowns lie on the border of the subdomain and are exactly those unknowns that will become separator unknowns on a parent domain. The other unknowns inside the subdomain can be ignored, as their couplings with inner and outer unknowns are eliminated by the static condensation (see pass 2) on the lower levels. Figure 1 illustrates this classification by painting the inner unknowns as white circles and the outer unknowns as black circles. Pass 2: Static Condensation The static condensation pass is a bottom-up process and computes the local systems of equations for each subdomain. On the finest level — i.e. when the subdomain has only 2 × 2 or 3 × 3 unknowns and is not further divided — the system of equations can be taken directly from the discretization. On the higher levels the system is computed from the local systems of the two subdomains. The first step is to assemble and renumber the system matrix: K1 0 KEE KEI VT V =: . (2) 0 K2 KIE KII The operator V combines the system matrices K1 and K2 that are taken from the two subdomains. The unknowns on the separator belong to both subdomains, thus the corresponding matrix lines are simply added up. The renumbering is such that the outer and inner unknowns form separate matrix blocks KEE and KII to enable the following block elimination. In this second step the couplings between the inner and the outer unknowns are eliminated by computing the Schur complement −1 EE 0 Id 0 KEE KEI Id −KEI KII K = (3) −1 KIE KII −KII 0 Id KIE Id 0 KII EE = KEE − KEI · K −1 · KIE . where K II Of course, the right-hand sides have to be treated accordingly. Pass 3: Solution The solution itself is again a top-down process. Starting with the separator of the whole computational domain, the values of the separator unknowns are computed recursively from the local system of equations on each subdomain.
798
2.2
Michael Bader and Christoph Zenger
Iterative Versions of the Nested Dissection Method
In the case of a two-dimensional PDE using standard discretization with a fiveor nine-point stencil, computing the direct solution of the resulting linear system 3 of equations (with N = n × n unknowns) needs O(N 2 ) operations and requires 3 O(N log N ) memory [4]. While one can sometimes put up with the O(N 2 ) operation count and the resulting increased computing time, shortness of memory often inhibits simulations altogether as the required resolution exceeds the memory capacity. However, the nested dissection method is often used in industrial codes for reason of reliability and robustness. For Poisson type equations, H¨ uttl [1] and Ebner [2] introduced iterative versions of the nested dissection method that have a memory requirement that grows only linearly with the number of unknowns. The main differences between an iterative and a direct solver occur in the condensation and the solution pass. While the assembly part of the condensation pass mainly stays the same, the elimination of the separator couplings is left out entirely or at least greatly reduced in an iterative solver. The single top-down solution pass is replaced by a series of iteration cycles. Each of those cycles consists of a bottom-up pass that transports the current residual, like the righthand side in the condensation pass, from the finest level subdomains to the higher levels. The second part of the iteration cycle is the actual solution, where Gauss-Seidel- or Jacobi-relaxation is used for the computation of the separator unknowns. Figure 2 illustrates the complete sequence of the different passes during the iterative nested dissection.
solution
relaxation ... more iterations ...
recursive substructuring
static condensation with elimination
distribution of residual
direct solver iterative solver
Fig. 2. The different passes of the iterative nested dissection algorithm
To get an acceptable speed of convergence, the systems of equations are preconditioned, for example by means of hierarchical bases [6]. The resulting algorithms reduce the number of operations to O(N log N ) and the memory requirements back to O(N ). The convergence factor of such an iterative nested dissection method can be further improved by eliminating at least some of the couplings between the hierarchically highest unknowns in the local systems of equations [2]. Figure 3 shows the general structure of the resulting algorithm, which will be the basis for the algorithm introduced in section 3.
A Fast Solver for Convection Diffusion Equations
799
Pass 2: static condensation (set-up phase)
(S1) (S2) (S3)
T
K1 0
K =V V 0 K2 ¯ = H T K H K ¯ −1 K = L−1 KR
assemble system matrix from subdomains hierarchical transformation partial elimination of the separator couplings, the elimination matrices L and R have to be stored
Pass 3: iteration cycle (bottom-up part)
T
r
(U1) r = V (U2) r¯ = H T r (U3) r = L−1 r¯
1 r2
assemble local residual from subdomains hierarchical transformation right-hand side of elimination Pass 3: iteration cycle (top-down part)
[it] u ˆS
(D1) = ωdiag(K)−1 r ˆ[it] (D2) u ¯ = R−1 u (D3) u =H u¯ (D4) uu12 = V T u
relaxation step (here: weighted Jacobi) revert elimination hierarchical transformation distribute solution to subdomains
Fig. 3. Iterative Nested Dissection Algorithm: steps (S3), (U3) and (D2) are only needed if partial elimination is used
2.3
Parallel, Iterative Nested Dissection
The parallelization of iterative nested dissection algorithms like the one described in figure 3 was analysed in depth by H¨ uttl [1] and Ebner [3]. Figure 4 shows a typical distribution of the subdomains to different processors. Obviously, the tree structure of the subdomains and the simple bottom-up/top-down structure of the several computational cycles enable the parallel processing of all subdomains that are on the same level. However, the different levels have to be executed sequentially.
0 0
8
0 0 0
2 1
2 3
4 4
6 5
12
8
4
6
8 7
8
10
12
14
9 10 11 12 13 14 15
Fig. 4. Distribution of subdomains to 16 processors
800
Michael Bader and Christoph Zenger
The computations on each subdomain are usually not parallelized, so the size of these sequential subproblems, which is mainly determined by the size of the separator, should be kept small. We already mentioned that, in this context, the alternate bisection used in the recursive substructuring pass gives some advantage over other substructuring approaches. When using Jacobi relaxation for solution, the memory requirement on each subdomain is only linearly dependent on the number of unknowns on the separator. Therefore, these methods are particularly well suited for architectures with distributed memory. Compared to other types of solvers (geometric multigrid, AMG, etc.), one should also emphasize that all three passes of the algorithm have equally good parallelization properties.
3
Nested Dissection with Incomplete Elimination
While the algorithms discussed in the previous section give good results for the solution of diffusion type equations, this no longer holds when they are applied to convection diffusion equations. In this section, we want to propose an approach which extends the existing iterative algorithms, such that convection diffusion equations can be treated efficiently as well. When solving simple diffusion type equations, the elimination of the couplings between inner and outer unknowns can well be left out entirely, yet the convergence rates can be improved by eliminating just a small, fixed number of only the strongest couplings. For example, it is sufficient to eliminate the couplings between the unknown in the middle of the separator and those four unknowns that lie on the corners of the subdomain. However, for the convection diffusion case, eliminating a fixed number of couplings on each subdomain is no longer sufficient to achieve fast convergence. In general, we can expect that the more couplings we eliminate the better the convergence rates will become, because finally, when all couplings between inner and outer unknowns are eliminated, the algorithm becomes a direct nested dissection solver again, and ”converges” in one step. Of course we pay for this 3 rapid convergence with the O(N 2 ) computing time for the eliminations and O(N log N ) memory requirement due to the generated fill-in. We therefore have to look for a compromise between eliminating enough couplings to achieve good convergence rates and keeping the number of eliminations small enough to maintain the O(N ) complexity with respect to computing time and memory requirement. A suitable heuristics for choosing the ”correct” number of eliminated1 unknowns can be found by analysing the underlying physical effects. Figure 5 shows a certain subdomain where a heat source on the left border is transported via a constant convection field towards the domain separator. Due to diffusion the former peak will extend over several mesh size steps after it has been transported to the separator. It is clear that using only one point on the separator is not enough to resolve the resulting heat profile (left picture). 1
to ”eliminate” an unknown in this context means that all couplings between ”eliminated” unknowns are eliminated.
A Fast Solver for Convection Diffusion Equations
801
h
H
Fig. 5. Comparison of different elimination strategies: only the couplings between the black nodes are eliminated
Therefore, eliminating only the couplings between the black nodes of the left picture would not be sufficient to achieve good convergence. However, as we can see in the right picture, it is not necessary to eliminate all separator nodes like it is done in nested dissection. The question is now how to choose the distance h between two eliminated nodes. From the underlying physics it is known that, for convection diffusion equations, the streamlines of the transported heat have a parabolic profile. Therefore, if the typical distance H between the unknowns grows by a factor of four, we can expect the heat profile to become twice as wide. Thus we can also double √the distance h between the eliminated nodes. In other words, we choose h ∝ H and the number of eliminated separator unknowns proportional to the square root of the total number of unknowns on the separator. In our algorithm, we implement this square root dependency by doubling the number of eliminated separator unknowns after every four bisection steps. This strategy is illustrated in figure 6. The overall algorithm remains the same as in figure 3, but now the elimination matrices L and R no longer have only a small, fixed number of non-diagonal entries. If we have a subdomain with m separator √ unknowns, we eliminate the √ couplings between O( m) inner unknowns and O( m) outer unknowns which produces O(m) entries in L and R. As a result of these extra eliminations, the
Fig. 6. Bisection with incomplete elimination, only the couplings between the grey nodes are eliminated
802
Michael Bader and Christoph Zenger
fill-in created in the system matrix K is now much bigger compared to the algorithms of section 2.2. Therefore, one can no longer afford to store the entire system matrix K for each subdomain as this would result in an O(N log N ) memory requirement. However, if we choose the (weighted) Jacobi-method for relaxation, we only have to store the main diagonal of K as we can compute the residual separately (see steps U1-U3 of the algorithm in figure 3). This puts both the memory requirement and the number of operations needed for the set up of the system matrices back to O(N ). Of course the Jacobi-relaxation gives less satisfying convergence factors than the Gauss-Seidel-relaxation, but, as we will show in the next section, it is still possible to achieve reasonable performance.
4
Numerical Results
We tested our algorithm on the three different benchmark problems outlined in figure 7. Problem (a) indicates a flow with constant convection a(x, y) = a, problem (b) an entering flow that contains a 180 degree curve of the flow, and finally problem (c) a circular flow problem. Each problem was discretized on a square using standard second order finite difference discretization with a five point stencil. As the elimination points are chosen independently of the flow field it is clear that the computation time and the memory used are independent of the type of the test problem. Figure 8 indicates that both computation time and memory requirement rise linearly with the number of unknowns. This can also be deduced by theoretical means. Table 1 shows the convergence rates for test problem (c), but nearly identical results are obtained for test problems (a) and (b). Table 2 shows the average convergence rates for test problem (c) if the algorithm is used as a preconditioner for the BiCGStab method [7]. For the Poisson equation it is known that preconditioning with hierarchical bases leads to a logarithmic increase of the convergence rates, which corresponds well with the behaviour we observe in tables 1 and 2. We can see that the convergence rates are largely independent on the strength of convection as long as a certain ratio between convection strength and mesh
(a) straight convection
(b) bent pipe convection
(c) circular convection
Fig. 7. Convection fields of the three test problems
A Fast Solver for Convection Diffusion Equations
803
256M
128s 64M
O(N)
solution
32s
16M
setup
8.0s
memory
4M
2.0s
O(N) 32x32
64x 64
128x128
256x256 512x512
1M
32x32
Costs for setup and complete solution
64x 64
128x128
256x256 512x512
Memory requirement
Fig. 8. Computation time and memory requirement
size is not exceeded. For stronger convection the heuristics of our elimination strategy no longer holds and convergence begins to break down.
32×32 64×64 128×128 256×256 512×512
a=0 0.635 0.700 0.721 0.746 0.783
a=1 0.635 0.700 0.722 0.746 0.783
a=2 0.636 0.700 0.722 0.746 0.783
a=4 0.637 0.699 0.722 0.746 0.783
a=8 0.644 0.697 0.724 0.746 0.783
a = 16 0.661 0.693 0.734 0.746 0.783
a = 32 0.743 0.694 0.756 0.744 0.784
a = 64 a = 128 a = 256 — — — 0.753 — — 0.823 0.853 — 0.757 0.818 0.848 0.795 0.859 div
Table 1. Convergence rates for test problem c (circular convection), a denotes the strength of the convection. (Jacobi-relaxation with ω = 23 ) 32×32 64×64 128×128 256×256 512×512
a=0 0.079 0.151 0.156 0.208 0.226
a=1 0.083 0.153 0.156 0.208 0.226
a=2 0.085 0.154 0.157 0.207 0.226
a=4 0.084 0.155 0.157 0.205 0.227
a=8 0.083 0.156 0.159 0.197 0.227
a = 16 0.127 0.153 0.175 0.214 0.229
a = 32 0.147 0.161 0.216 0.221 0.231
a = 64 a = 128 a = 256 0.210 0.315 0.467 0.227 0.401 0.404 0.351 0.433 0.545 0.238 0.357 0.462 0.271 0.376 0.463
Table 2. Average convergence rates for test problem c (circular convection) using preconditioned BiCGStab and upwind discretization
A parallel implementation of our iterative nested dissection code was realized using MPI as message passing protocol. The distribution of the subdomains to the processors was done as was shown in figure 4. Figure 9 illustrates speedup and parallel efficiency of the parallel implementation for a problem with 512×512 unknowns on a cluster of SUN Ultra 60 workstations.
804
Michael Bader and Christoph Zenger 16 512x512 unknowns
100%
speedup
4
80% 70% 60%
2 processors
1
parallel efficiency
90%
8
1
2
4
8
processors
50% 16
1
2
4
8
16
Fig. 9. Speedup and parallel efficiency on a cluster of SUN Ultra 60 Workstations
5
Present and Future Work
We are currently working on two topics which are still missing in the presented algorithm to become a suitable solver for practical convection type problems. The first topic is the treatment of complicated computational domains and of inner boundaries and obstacles, the other one is the removal of the O(log n) dependency in the convergence rates and to make them independent of the resolution of the mesh. It seems that both topics can be successfully treated if we replace the hierarchical basis preconditioning by the usage of generating systems which turns the algorithm into a multigrid type method (see Griebel [8]). First numerical experiments indicate that the convergence rates seem to indeed become independent of mesh size, geometry and strength of convection (as long as the diffusion still dominates the convection on the finest mesh). The efficient treatment of strongly convection dominated flow leads our list of future work. In the case of strong convection our elimination strategy is no longer appropriate as it depends on the existence of a certain amount of real diffusion. However, an efficient elimination strategy should be achievable either by eliminating couplings that are aligned with the direction of the flow or by choosing the eliminated couplings in an algebraic manner which leads to an AMG like method with fixed coarse grid selection.
References 1. H¨ uttl, R., Schneider, M.: Parallel Adaptive Numerical Simulation. Institut f¨ ur Informatik, TU M¨ unchen, SFB-Bericht 342/01/94 A (1994) 2. Ebner, R.: Funktionale Programmierkonzepte f¨ ur die verteilte numerische Simulation. PhD thesis, TU M¨ unchen (1999) 3. Ebner, R., Zenger, C.: A distributed functional framework for recursive finite element simulation. Parallel Computing 25 (1999) 813-826 4. George, A.: Nested Dissection of a Regular Finite Element Mesh. SIAM Journal on Numerical Analysis 10 (1973)
A Fast Solver for Convection Diffusion Equations
805
5. Ruge, J.W., St¨ uben, K.: Algebraic multigrid. In: McCormick, S.F. (ed.): Multigrid Methods. SIAM (1987) 73-130 6. Yserentant, H.: Hierarchical basis give conjugate gradient methods a multigrid type speed of convergence. Appl. Math. and Comput. 19 (1986) 347-358 7. van der Vorst, H.: Bi-CGSTAB: A fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems. SIAM J. Sci. Statist. Comput., 13 (1992) 631-644 8. Griebel, M.: Multilevel algorithms considered as iterative methods on semidefinite systems. SIAM Int. J. Sci. Stat. Comput., 15/3 (1994) 547-565
Low Communication Parallel Multigrid A Fine Level Approach Marcus Mohr System Simulation Group of the Computer Science Department, Friedrich-Alexander-University Erlangen-Nuremberg [email protected]
Abstract. The most common technique for the parallelization of multigrid methods is grid partitioning. For such methods Brandt and Diskin have suggested the use of a variant of segmental refinement in order to reduce the amount of inter–processor communication. A parallel multigrid method with this technique avoids all communication on the finest grid levels. This article will examine some features of this class of algorithms as compared to standard parallel multigrid methods. In particular, the communication pattern will be analyzed in detail. Keywords: elliptic pde, parallel multigrid, domain decomposition, communication cost
1
Introduction
There exists a great variety of different parallel architectures, ranging from clusters of workstations to massively parallel machines. In this paper we will consider the case that the number of processors is significantly smaller than the number of grid points and that memory is distributed. With this background the most common approach to the parallelization of numerical algorithms is grid partitioning. In the special case of multigrid methods nested levels of grids are employed. This approach introduces the need to communicate values related to the points on or near the inner boundaries. The cost of communication naturally limits the possible speedup of a parallel method. This cost can be alleviated by sophisticated programming techniques, e.g. by overlapping communication with calculation. Generally the communication cost is determined by the number of messages that must be exchanged and the number of bytes that have to be transmitted. In multigrid the number of messages is proportional to the number of domains and therefore independent of the grid level. The number of bytes on the other hand is strongly related to the coarseness of the respective level since it is coupled to the number of interface points. This leads to another aspect, namely the parallel efficiency of the algorithm, determined by the ratio between communication and computation. This ratio is directly proportional to the ratio of volume and surface of the subgrids and therefore becomes worse on the coarser grid levels. This is the starting point for A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 806–814, 2000. c Springer-Verlag Berlin Heidelberg 2000
Low Communication Parallel Multigrid
807
many approaches to improve parallel multigrid, like e.g. coarse grid agglomeration, multiple coarse grid and concurrent algorithms, see e.g. [6], or methods that employ different cycle schemes like e.g. the U–cycle [9]. A radical approach to reduce fine grid communication has been suggested by Brandt and Diskin in [3]. Here an algorithm that completely eliminates the need for inter–processor communication on several of the finest grids is presented. In the following we will examine a variant of this algorithm. We want to depict some of its characteristics and problems and compare its communication requirements to that of a “conventional” parallel multigrid algorithm. The algorithm will be introduced in Sect. 2. We will compare its communication requirements to that of a “conventional” parallel multigrid algorithm in Sect. 3 and analyze the communication pattern in Sect. 4.
2
Algorithm of Brandt & Diskin
In [3] Brandt and Diskin introduced a parallel multigrid algorithm completely without interprocessor communication on several of the finest levels. This was achieved by the use of segmental–refinement–type procedures, which were originally proposed as out–of–core techniques on sequential computers, cf. e.g. [1]. Since the bulk of communication takes place on the fine grids, it may be especially attractive to save this cost. The effectiveness of such a technique depends on specific machine and problem parameters. The potential benefits are most impressive when communication is slow and expensive with respect to computation. Their algorithm can be described as follows. Starting from a hierarchy of grids, in a preliminary step all levels, except the coarsest one, are decomposed into as many overlapping subdomains as processors are available. We get sequences of nested subdomains, where a subdomain on a coarser grid occupies a larger area than the corresponding subdomain on the next finer grid. Each such sequence is then assigned to a processor, which starts on it a standard V–cycle, descending through its grid hierarchy. In this process it does not exchange data with its neighbours. When the second coarsest level is reached, the local (approximate) solutions from all processors are used to formulate a global coarse grid problem. This is then solved exactly by some unspecified algorithm, possibly of course a standard parallel multigrid method. The solution of the global coarsest grid problem is used by each processor to correct its local solution approximation on the second coarsest level. As in the following correction steps on the finer levels the values at the interfaces are included into the correction. After this step each processor finishes its V–cycle, again without communication with its neighbours. The algorithm uses the following basic principles to reduce communication: – Clearly some exchange of information between the processors is inevitable to solve the problem. In the algorithm by Brandt and Diskin however, this data exchange is restricted to the coarse grid correction from the common coarsest grid. So there is no communication on the finer grid levels.
808
Marcus Mohr
– To compensate errors introduced by the missing exchange of information, each subdomain has a buffer area around it, that fulfills two purposes. On the one hand, if an appropriate relaxation scheme, like e.g. red–black Gauß– Seidel is chosen, the buffer slows the propagation of errors due to wrong values at the interfaces. On the other hand, as in standard multigrid, the coarse grid correction introduces some high–frequency error components on the fine grid. Since the values at the interfaces cannot be smoothed, the algorithm cannot eliminate these components. But in elliptic problems high– frequency components decay quickly. So the buffer area keeps these errors from affecting the inner values too much. – Nevertheless the algorithm will in general not be able to produce an exact solution of the discrete problem. While this may seem prohibitive at first glance, one should remember that, when solving a PDE, it is the continuous solution one is really interested in. Since the latter is represented by the discrete solution only up to the discretization error, the result of the algorithm is still valuable, as long as it can be guaranteed, that its algebraic error remains at least smaller than the discretization error. – Since the overlap areas are included in the V–cycle, the algorithm trades communication for calculation. Thus savings in communication time may be partly compensated by the extra computation. This depends on several factors, such as size of the problem, number and arrangement of subgrids, extension of the overlap areas, and the MFLOP– and transfer–rates of the applied hard– and software. It will be further examined in the next section. A crucial aspect of the above algorithm is of course the choice of an appropriate size for the overlap between subdomains. Until now there exists no general a priori error estimate that would allow to give a bound for the algebraic error depending on the overlap parameter J, but the experiments in [3,7,8] indicate that already small overlaps can produce reasonable results.
3
Efficiency Analysis
In this section we will explore how much overall computation time can be saved by the trade–off between communication and additional computation in the overlap areas. As mentioned above this depends on several architecture and problem parameters. To examine this question we compare a “standard” parallel V–cycle for the coarse grid correction (CGC) to an identical V–cycle that does not exchange data between processors, but instead employs overlapping subdomains. We model the times spent with computation and communication within the algorithm from the following simplifying assumptions: – Data exchange between processors is performed by message passing. – Communication and calculation are sequential and cannot be overlapped. – A processor can send and receive messages simultaneously.
Low Communication Parallel Multigrid
809
We now define a relative time saving TRS per grid level in the following way TRS :=
T (stand) − T (nocom) . T (stand)
(1)
Here T (nocom) is the time for the variant without communication and T (stand) for the standard variant with communication. The times include all the work that has to be done for the specific grid level within one cycle of the CGC– scheme, that is smoothing, calculation of the defect, restriction of the defect, prolongation of the coarse grid solution and adding of the correction. They can be split in the following way T (stand) = Tcalc (stand) + Tcomm
(2)
T (nocom) = Tcalc (stand) + Tcalc (buffer) .
(3)
To determine the times for communication and calculation we use the following two models γ Tcalc = νFsmooth N1 + Fmulti N2 (4) np 1 αM +βW (5) Tcomm = np with the parameters α β γ np ν W
latency Fsmooth # of bandwidth Fmulti # of time per FLOP N1 # of # of processors N2 # of sum of smoothing steps M # of # of words (double precision values)
FLOPs per point for smoothing FLOPs per point for CGC points that are smoothed points included in the CGC messages to be exchanged that must be exchanged
As model problem we consider a 2D elliptic boundary value problem discretized with a 5–point stencil on a logically rectangular grid. We assume the use of a 9–point stencil for restriction, bilinear interpolation as prolongation, and red–black Gauß–Seidel as smoother. For the 3D analogue, we use a 7–point discretization, a 27–point stencil for restriction, and trilinear interpolation. We assume that our global finest grid is quadratic/cubic with N dim points (N − 1 being a power of 2), and that we have a logical processor grid. Assuming furthermore that a non–overlapping domain decomposition approach is employed for the grid partitioning [5,8], we can, for given values of N , px , py and eventually pz , and for a given overlap parameter J, derive the corresponding values of np , N1 , N2 , M , and W . We estimate the number of numerical operations per grid point as Fsmooth ≈ 9 (2D) / 11 (3D) and Fmulti ≈ 7.5 (2D) / 9 (3D). Now we choose two different processor types, i.e. two different values for γ, one with 200 and the other one with 50 MFLOP. We compare the relative time saving TRS per grid level for different grid sizes N and different communication
Low Communication Parallel Multigrid
811
speeds β. Latency effects are taken into account by assuming that α = const · β. The results are shown in Fig. 1 and 2. Communication hardware may have widely varying performance characteristics. For orientation we present typical parameters for current implementations of the message passing interface (MPI). The following values have been taken from [4]: Myrinet Fast Ethernet Ethernet α 70µs 630µs 1150µs β 0.36µs 2µs 11.4µs α/β 194 315 100 The results in Fig. 1 and 2 confirm the following properties of the approach. The value TRS of the relative time saving grows with β. This was to be expected, since the more expensive communication, the greater the gain by replacing it with computation. However, as the grids become larger TRS is reduced. Although the margin between communication time Tcomm and additional calculation time Tcalc (buffer) becomes more and more favorable as N grows, this is to some extent compensated by the growing influence of the regular computational work Tcalc (stand) on the overall time. This can clearly be seen by comparing the figures for the 200 and the 50 MFLOP case. This compensation effect is significantly smaller in 3D than in 2D, since here the number of points and the amount of computation is of order O(N 3 ), whereas the number of interface points, responsible for communication, grows like N 2 . In 2D, on the other hand, these values are N 2 and N , respectively. As far as the remaining problem parameters are concerned, some properties can easily be derived. If we consider the overlap parameter J, it can be said that, as long as N J, the additional computation grows almost linearly with J. So we have that TRS = C1 − C2 J, with constants Ck depending on the other parameters. We found that the parameter ν had little influence on TRS , but the number of processors np does. The relative time saving is more favorable, when there are more processors, because then there are fewer points on each subgrid and therefore the above mentioned compensation effect is reduced.
4
The Two Level Brandt–Diskin–Algorithm
In this section we will take a closer look at the two level version of the multilevel cycle in order to analyze in detail the communication pattern of the algorithm and show some of its properties. We do this by means of the model problem on the unit square Ω and restrict us to the case of two subdomains, which will suffice to explain the basic concept. We discretize the problem on a fine grid Ω h := {xij | 0 6 i, j 6 N } and a coarse grid Ω H := {x2i2j | 0 6 i, j 6 N/2}. Here xij = (ih, jh) denotes a grid point on the ith vertical and the j th horizontal line. We decompose the fine grid into two equally large parts Ω h,1 and Ω h,2 with the grid line j = N/2 as common
812
Marcus Mohr
boundary. Now we augment each subgrid with a buffer zone of J > 0 grid lines and get the extended subdomains ˆ h,1 := xij j 6 N/2 + J ˆ h,2 := xij j > N/2 − J . Ω and Ω (6) The quantity J > 0 describes the amount of overlap between the two subdomains. It is required that J is a multiple of 2 to ensure that there are coarse grid points that coincide with interface points on the fine grid. Each subgrid is now assigned to one processor p1 and p2 . Starting from an approximation uhS to the discrete solution of the fine grid problem the algorithm is defined by iteratively performing the following seven steps: 1. 2. 3. 4.
Both processors perform separate pre–smoothing steps. A global approximate solution is composed from the local solutions. A global defect function is composed. With the global solution and defect the right hand side of the coarse grid equation is calculated according to the FAS–scheme. 5. The coarse grid equation is solved exactly (by whatever means). 6. Each processor uses the coarse grid solution to correct its local approximate. 7. Both processors perform separate post–smoothing steps. In step 1 and 7 the iteration method used is red–black Gauß–Seidel and there is no communication between the processors to update values along the interfaces. As a consequence the local solutions u2,1 and u2,2 will start to differ. Steps 2 and 3 are of virtual character in the sense that no processor knows the global functions completely. Formally the global solution uh is computed by taking the values of the local solutions in the interior of the subdomains Ω 2,p and their average along their common boundary. The same is done for the global defect function, so that we get rh,1 (xi,j ) if j < N/2 h h,1 h,2 (7) r (xi,j ) := r (xi,j )/2 + r (xi,j )/2 if j = N/2 rh,2 (xi,j ) if j > N/2 Here rh,p = f h − Lh uh,p denotes the local defect functions and f h and Lh are ˆ h,p . We point to be interpreted as restrictions on the extended subdomains Ω h out that in general the so defined global defect function r is different from the defect f h − Lh uh of the global solution function at points near the interface. We will return to this fact later on. In a normal FAS–scheme the correction in step 6 would be performed as
h uhnew = uhold + IH uH − IhH uhold (8) where uH is the solution of the coarse grid problem obtained in step 5, and h IH and IhH are a prolongation and a restriction operator. In the two level cycle the same is done, but since the global solution is not locally available, the local
Low Communication Parallel Multigrid
813
solution is corrected instead. In order to understand how this correction leads to an update of information between the processors, we re-write the correction. We define a prolongation operator h,p ˆ h,p JH : ΩH → Ω
(9)
ˆ h,p from the respective values in Ω H ∩ Ω ˆ h,p by bilinear calculating the values in Ω interpolation and a restriction operator H ˆ h,p → Ω H Jh,p :Ω
(10)
ˆ h,p through injection and sets the values which computes the values in Ω H ∩ Ω H ˆ h,p p ˆ h,p , we get to zero. Denoting by E the identity operator on Ω in Ω \Ω h,p H h,p H p h,p u (P ) . (11) uh,p (P ) = E − J J (P ) + J u new old h,p H H ˆ h,1 ∩ Ω ˆ h,2 . Consider now a point P = xij that lies in the overlap area Ω For such a point both processors store a local solution value. Due to lack of communication during relaxation these values will in general differ prior to the correction step. The second term in the correction formula will lead to the same value on both processors, while the first term in (11 will vanish for points that can be found on the coarse grid. So in this case the values of the two processors at the point will coincide after the correction. For points not on the coarse grid, the first term will in general not vanish. This is due to interpolation errors. So the values of the two processors will be composed of a common part, namely h,p H u )(xij ), and a disturbance, which is a remainder of the old function value (JH at each processor. In this way an information update between the processors is embedded into the correction. In the following, we want to point out some of the features of the two level cycle. First of all, since the algorithm represents an iteration on the pair of local h,2 solution functions (uh,1 i , ui ), we have to analyze its convergence. The experiments in [3] as well as the analysis for the special case of exact solvers as smoothers in [7] indicate that the algorithm will converge for every choice of initial values. But contrary to standard multigrid the result depends on the initial values. These can be grouped into equivalence classes according to their hierarchical offset, with every class converging to the same limit function. The reason for this is that the first term in (11) will always reproduce the hierarchical offset of the solution, while the second term represents a bilinear interpolation and therefore has no hierarchical offset at all. As a consequence the hierarchical offset of the initial values along the inner boundaries is never changed by the algorithm. As another property we require that for a proper choice of the overlap parameter J the error with respect to the true discrete solution remains at least of the same order of magnitude as the discretization error. In [3,7] it was shown by experiment that this can already be achieved with an overlap of 2 to 4 grid lines. The accuracy can be improved, by a different way to compose a global
814
Marcus Mohr
defect function rh . As already mentioned, the global defect function when set up according to (7) does not match the defect r¯h := f h − Lh uh of the global solution approximation. If we use r¯h as the global defect, the algebraic error will be smaller than with the use of rh and it will strongly be restricted to the vicinity of the common boundary of the subdomains. In this case also smaller overlaps are sufficient to keep the algebraic error smaller than the discretization error and this will often be achieved in a smaller number of cycles, [7,8]. This approach does not cause extra cost. Since the calculation of the defect f h − Lh uh as well as the prolongation by full weighting and the application of the coarse grid operator LH are linear operations the amount of communication needed to set up the coarse grid equation is the same independent of whether we use r¯h or rh .
5
Conclusions
In this paper we have shown that the approach by Brandt and Diskin to parallelize multigrid methods can be used to reduce overall computation time. This remains valid even for high–speed communication infrastructures as long as the processor speeds are fast enough. We have presented an analysis of the communication pattern of the two–level–version of their algorithm and have shown a way to improve its performance. Questions that remain open are a priori estimates for the algebraic error of the algorithm as well as convergence rates.
References 1. A. Brandt, Multi–level adaptive solutions to boundary value problems, Mathematics of Computation 31 (1977) 333 - 390. 2. A. Brandt, B. Diskin, Multigrid Solvers on Decomposed Domains, Contemporary Mathematics 157 (1994) 135 - 155. 3. B. Diskin, Multigrid Solvers on Decomposed Domains, M. Sc. Thesis, Department of Applied Mathematics and Computer Science, The Weizmann Institute of Science, 1993. 4. M. Griebel and G. Zumbusch, Parnass: Porting gigabit–LAN components to a workstation cluster, in: W. Rehm, ed., Proceedings des 1. Workshop ClusterComputing, 6.-7. November 1997, in Chemnitz, (Chemnitzer Informatik Berichte, CSR-97-05) 101-124. 5. M. Jung, On the parallelization of multi–grid methods using a non–overlapping domain decomposition data structure, Appl. numer. Math. 1 (1997) 119-138. 6. L. Matheson, R. Tarjan, Parallelism in Multigrid Methods: How much is too much?, Int. J. Parallel Programming 5 (1996) 397 - 432. 7. M. Mohr, Kommunikationsarme parallele Mehrgitteralgorithmen, Diplomarbeit, Institut f¨ ur Mathematik, TU M¨ unchen, 1997. 8. M. Mohr, U. R¨ ude, Communication Reduced Parallel Multigrid: Analysis and Experiments, Technical Report No. 394, University of Augsburg, 1998. 9. D. Xie, L. Scott, The Parallel U–Cycle Multigrid Method, Virtual Proceedings of the 8th Copper Mountain Conference on Multigrid Methods, MGNET, 1997, (http://casper.cs.yale.edu/mgnet/www/mgnet.html).
Parallelizing an Unstructured Grid Generator with a Space-Filling Curve Approach J¨ orn Behrens1 and Jens Zimmermann2 1
Munich University of Technology, D-80290 M¨ unchen, Germany, [email protected] www-m3.ma.tum.de/m3/behrens/, 2 Ludwig-Maximilians-Universit¨ at, D-80539 M¨ unchen, Germany, [email protected]
Abstract. A new parallel partitioning algorithm for unstructured parallel grid generation is presented. This new approach is based on a spacefilling curve. The space-filling curve’s indices are calculated recursively and in parallel, thus leading to a very efficient and fast load distribution. The resulting partitions have good edge-cut and load balancing characteristics.
1
Introduction
Adaptive unstructured grid generation poses a nontrivial problem to parallelization approaches. To give an example: The adaptive simulation of atmospheric transport has been parallelized in [1]. However, grid generation has been used in its serial form, because at that time no adequate parallel grid generator was available. A new hierarchical adaptive grid generator (amatos: Adaptive mesh Generator for Atmospheric and Oceanic Simulations) has been implemented, based on modular software techniques. pamatos is the parallel implementation of amatos. pamatos can generate (i.e. refine, coarsen, and adapt corresponding to a given error criterion) dynamically changing grids in two space dimensions. The grid generator’s behavior can be controlled by a simple programming interface (API). Parallelization is based on the message passing paradigm. Besides the nontrivial problems associated to the data management in unstructured grid generation on parallel computers, one major problem is the mesh partitioning. Several authors have proposed different approaches, c.f. [4,7,11,12]. However, most of these methods require rather complicated calculations on the graph of the mesh. Our approach follows a different line, as it is based on the geometry of the mesh. We propose a new construction scheme for a space-filling curve (SFC). Hilbert and Peano originally introduced SFCs for proving a set theoretical problem [6]. A SFC can be constructed such that it passes through all points of a multi-dimensional domain. This induces a mapping which allows to find a unique A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 815–823, 2000. c Springer-Verlag Berlin Heidelberg 2000
816
J¨ orn Behrens and Jens Zimmermann
Fig. 1. A Hilbert-type space-filling curve for an adaptively refined rectangular grid (left). New space-filling curve in a locally refined triangular grid (left).
enumeration for all elements of the unstructured mesh. Consecutively numbered elements can then be distributed with almost optimal load balancing. Roberts et al. [10] proposed several space-filling curve mechanisms for key generation in a hash table based mesh. Their bit-manipulation based algorithm also leads to a natural decomposition of the mesh. Griebel and Zumbusch [5] introduced space-filling curves to sparse grid adaptive methods. However, both approaches do not easily extend to non-rectangular domains. In the following section we give a detailed description of our recursive algorithm for the SFC for non-rectangular domains. In section 3 we describe the grid generator which has been tailored for advection problems in atmospheric and oceanic simulation applications. Section 4 provides some numerical results and a conclusion is given in section 5.
2
Recursive Calculation of the Space-Filling Curve for Triangle Bisection
For two-dimensional rectangular domains Hilbert-type SFCs have the shape shown in Fig. 1. A recursive construction mechanism is given, for example, by Breinholt and Schierz [3]. There are a lot of different forms of space-filling curves for different purposes. Lafruit and Cornelis [9] use so called dove-tail space-filling curves for the parallelization of fast wavelet transforms. The SFC is characterized by its corners. Each corner’s index can be calculated recursively. For partitioning a mesh, a SFC has to be constructed such that each mesh cell holds at least one SFC corner. Each cell is indexed by the corresponding SFC index. Now, partitioning is easy: each processor receives an equally sized chunk of ascending indices. Nearly optimal load balancing can be achieved. Griebel and Zumbusch [5] as well as Roberts et al. [10] showed that the partitions obtained by space-filling curve approaches are well behaved with respect to other measures (e.g. edge-cut). Our results in section 4 support this observation. The algorithm for calculating a triangle index depends on three basic properties of our grid. First, the procedure mimics the bisection of triangles. Second,
Parallelizing an Unstructured Grid Generator
817
C
b D
A
a
c
d B
Fig. 2. Denotation of a triangle for the space-filling curve algorithm (see text).
the algorithm uses the center coordinates of each triangle as corners for the space-filling curve. Finally, in order to find new indices, a tabulated indexing scheme is used (see Table 1). There are three variables describing the triangle’s state. Each triangle of a given macro triangulation has state 1 by definition. Table 1. Table of states for the space-filling curve algorithm. state description variables resulting states original bisection direction of direction of new state i new state ii vector marked edge SFC segment state 0 d b + 5, add 3 d b 2 4, add 1 a c + 7, add 1 2 a c 0 6, add 3 c a + 1, add 6 4 c a 7 0, add 5 b d + 3, add 4 6 b d 5 2, add 7
To find the index for a refined triangle, we first have to describe the coarse triangle’s state in detail. Figure 2 shows a triangle ABC in state 1, with an edge marked for bisection CA. Vectors a, b, c, and d, denote different directions, used to describe the state. The triangle is divided by a line defined by d (dashed line in Fig. 2). The arrow inside the triangle indicates the direction of the SFC. The algorithm for calculating the SFC index reads as follows (examples referring to Fig. 2 are given in brackets): Algorithm 1 (Recursive space-filling curve) 1. Determine original state from bisection, marked edge, and SFC direction (The SFC’s direction is oriented from A to C, thus vector b with negative sign
818
J¨ orn Behrens and Jens Zimmermann 3 2.5 2 1.5 1 0.5 0 0.5 1 1.5 2
2
1.5
1
0.5
0
0.5
1
1.5
2
2.5
3
Fig. 3. Six partitions of a locally refined triangulation over a non-rectangular domain. defines the direction of the SFC, and vector d bisects the triangle. Therefore, triangle ABC is in state 1). 2. Determine position of refined triangle (triangle DBC’s center is in the opposite direction as the marked edge’s orientation). 3. Chose corresponding new state from Table 1 (we have new state ii for triangle DBC). 4. Add offset to the index value, if required (as DBC’s new state is 3 without the “add” attribute in the table nothing has to be done here). Table 1 contains all possible states. Applying Algorithm 1 recursively, yields a unique index for each triangle on the finest level of triangulation. When using a hierarchical grid generator (as in our case), we start with a coarse macro triangulation. Utilizing Algorithm 1 for each triangle of the macro triangulation, adding an offset to the indices of each macro triangle, and supposing that the children of each macro triangle will not be deformed too much, this space-filling curve can be used for non-rectangular domains. An example is given in Fig. 3. Each step of the following SFC partitioning algorithm can be executed in parallel. The recursive index calculation is embarrassingly parallel. Sorting can be done in parallel and moving elements is a parallel unstructured communication. To conclude, the whole SFC partitioning can be parallelized. Algorithm 2 (SFC partitioning) 1. 2. 3. 4.
3
Calculate SFC index for each triangle Sort global index set Cut global index set into chunks of equal size Move all triangles which do not belong to the own chunk
The Parallel Grid Generator
This section gives a brief description of the parallel mesh generator pamatos, using the new partitioning method. pamatos is a two-dimensional grid generator
Parallelizing an Unstructured Grid Generator
819
for atmospheric and oceanic simulations. It implements a bisection refinement technique for the elements. pamatos is implemented in Fortran 90 utilizing a modular object oriented programming paradigm. A relatively small number of interface routines controls the behavior of the grid. There is a serial sister of pamatos, called amatos, that has been used in atmospheric simulations in [2]. In principle, the programming interface for the serial and the parallel mesh generator are similar, so the application programmer does not need to care about parallelization aspects. It is the task of pamatos, to distribute the work load evenly among the processors, to provide the necessary communication, etc. The internal structure of pamatos is characterized by a hierarchy of different software layers. Communication primitives are built on top of a standard message passing library (at present MPI). The communication primitives layer hides the system dependent software parts away from the actual grid generator. On top of these, high level communication subroutines do the work required for movement and information update in an unstructured grid. High level communication subroutines are engaged both by the grid generation layer and by the list layer. The list layer facilitates programming in an unstructured grid generation process. An abstract procedure can be given as follows: Algorithm 3 (Abstract list oriented procedure) 1. Do the normal serial work until a data item is encountered that resides on a remote processor 2. Skip the part of work, depending on a non-available data item 3. Put the pointer to the required data item into the collection list and continue with serial work 4. If no serial work is left, communicate all data items in the collection list 5. resume work with (now available) data items With this mechanism, the methods developed for the serial grid generator have to be extended only by calls to the collection list and a communication step. This reduces the number of messages and increases the message length. A typical adaptation step is given in Algorithm 4. Note that all the steps can be performed in parallel, however, there are several synchronization points required. Efficiency can be achieved only, if the grid is distributed evenly among the processors. Algorithm 4 (Adaptation of the grid) 1. [in parallel] Refine those elements, flagged for refinement 2. [synchronization point] Exchange edge information required for creation of an admissible triangulation 3. [in parallel] Coarse those elements, flagged for coarsening 4. [synchronization point] Exchange edge information required for creation of an admissible triangulation 5. [in parallel] Calculate new distribution 6. [synchronization point] Move data items to achieve load balancing, update connectivity information
820
J¨ orn Behrens and Jens Zimmermann
Fig. 4. Locally refined mesh, used in the test case described in the text. The mesh consists of approx. 4500 elements.
4
Numerical Examples
In order to show some numerical results a test case has been chosen that contains two regions of local refinement (see Fig. 4). Execution time and speedup for up to eight processors are given in Table 2. The computing environment consists of two SGI Origin 200 systems with four processors each (MIPS R10K, 225 MHz), connected by a dedicated Gigabit network connection. Table 2. Execution times and relative speedup of the space-filling curve indexing. no. of processors 2 4 8 time [ms] 68.7 26.5 14.9 1 2.6 4.6 speedup
Load balancing and relative edge-cut are given in Table 3. The load balancing parameter l is given by l = emax /emin, where emax and emin is the maximum and minimum number of elements on a single processor respectively. The edge-cut is the number of edges, that belong to more than one processor. This corresponds to the number of edges cut by a partition boundary in the dual graph (see [11]). The relative edge-cut is the edge-cut as a percentage of total edges. The SFC partitioning scheme is intended for partitioning adaptive meshes in time-dependent simulations. Therefore, we compare results of the (serial) SFC scheme in an adaptive trace-gas simulation with Metis (version 4.0) [8]. Figure 5 gives load balancing and relative edge-cut for both schemes as a function of time. Metis’s edge-cut is smaller than that of the SFC scheme. This should result in less communication in numerical calculations on the partitioned mesh. On the other
Parallelizing an Unstructured Grid Generator
821
Table 3. Load balancing and relative edge-cut for the SFC-based algorithm on the model problem. no. of processors 2 4 8 load balancing parameter 1.000 1.002 1.004 0.03 0.04 0.06 relative edge cut
1.15
0.14
SFC Metis
SFC Metis
relative edge cut
balancing parameter
0.12
1.1
1.05
0.1
0.08
0.06
1 0
200
400
time
600
800
0.04 0
200
400 time
600
800
Fig. 5. Load balancing parameter (left) and relative edge-cut (right) for SFC and Metis in an adaptive time-dependent simulation.
hand, the SFC algorithm performs better with respect to the load balancing. This is not surprising, because (as noted before) optimal load balancing can always be achieved with SFCs. Note that the SFC-based algorithm yields less data movement after remeshing. This is due to the geometry based distribution induced by the SFC, whereas Metis operates on the (dual) graph of the mesh and cannot take into consideration the location of each element. Figure 6 shows two consecutive meshes distributed with the SFC-based and the Metis algorithm respectively. In Table 4 we compare computation times for a trace gas simulation on a mesh of 20,000 elements and 20026 edges. The given time for adaptation includes 12 adaptation steps and the partitioning in each step. Again, edge-cut and load balancing parameter is given. Note that the SFC-based algorithm is considerably faster. Table 4. Load balancing, relative edge-cut, and timing of Metis versus SFCbased partitioning on the real-life problem. Metis SFC-based load balancing parameter 1.063 1.047 relative edge cut 0.026 0.042 time for adaptation [s] 16.5 6.5
822
J¨ orn Behrens and Jens Zimmermann
Fig. 6. Two consecutive meshes from a trace gas transport application. Metis redistributes almost every single element (upper row) while the SFC-based algorithm leaves most elements on the original processor (lower row).
5
Conclusions
In this article we have introduced a new recursive space-filling curve algorithm for the dynamic distribution of adaptively refined triangular meshes to many processors. The recursive indexing algorithm proves to be fast and easily parallelizable. The partitions resulting from our SFC have good load balancing and edge-cut characteristics. The SFC does not require rectangular domains. However, the SFC-based algorithm is not as versatile as graph partitioning algorithms, because it depends on the geometry of the mesh. The proposed SFC algorithm is tailored for 2D bisection triangulations. It is supposedly easily extensible to 3D tetrahedral triangulations. Our algorithm is not suited for regular refinements yet.
Parallelizing an Unstructured Grid Generator
823
We intend to use the new parallel grid generator pamatos in our atmospheric simulations. However, to be of practical use, a lot of work has still to be done. The results presented here are motivating to proceed further on this way.
References 1. J. Behrens. An adaptive semi-Lagrangian advection scheme and its parallelization. Mon. Wea. Rev., 124(10):2386–2395, 1996. 2. J. Behrens, K. Dethloff, W. Hiller, and A. Rinke. Evolution of small-scale filaments in an adaptive advection model for idealized tracer transport. Mon. Wea. Rev., 128, 2000. in press. 3. G. Breinholt and C. Schierz. Algorithm 781: Generating hilbert’s space-filling curve by recursion. AMS Trans. Math. Softw., 24:184–189, 1998. 4. R. Diekmann, D. Meyer, and B. Monien. Parallel decomposition of unstructured FEM-meshes. In Proceedings of IRREGULAR 95, volume 980 of Lecture Notes in Computer Science, pages 199–215. Springer-Verlag, 1995. 5. M. Griebel and G. Zumbusch. Hash-storage techniques for adaptive mulitlevel solvers and their domain decomposition parallelization. Contemporary Mathematics, 218:279–286, 1998. ¨ 6. D. Hilbert. Uber die stetige Abbildung einer Linie auf ein Fl¨ achenst¨ uck. Math. Ann., 38:459–460, 1891. 7. M. T. Jones and P. E. Plassmann. Parallel algorithms for the adaptive refinement and partitioning of unstructured meshes. In Proceedings of the Scalable High Performance Computing Conference, pages 478–485. IEEE Computer Society Press, 1994. 8. G. Karypis and V. Kumar. Metis – A Software Package for Partitioning Unstructured Graphs, Partitioning Meshes, and Computing Fill-Reducing Orderings of Sparse Matrices. University of Minesota, Dept. of Computer Science/ Army HPC Research Center, Mineapolis, MN 55455, 1998. Version 4.0. 9. G. Lafruit and J. Cornelis. A space-filling curve image-scan for the parallelization of the two-dimensional fast wavelet transform. In Proceedings of the 1995 IEEE Workshop on Nonlinear Signal and Image Processing, http://poseidon.csd.auth.gr/Workshop/, 1995. Aristotle University of Thessaloniki, Aristotle University of Thessaloniki, Thessaloniki 540 06, P.O Box 451, Greece. 10. S. Roberts, S. Kalyanasundaram, M. Cardew-Hall, and W. Clarke. A key based parallel adaptive refinement technique for finite element methods. Technical report, Australian National University, Canberra, ACT 0200, Australia, 1997. 11. K. Schloegel, G. Karypis, and V. Kumar. Multilevel diffusion schemes for repartitioning of adaptive meshes. J. Par. Distr. Comp., 47:109–124, 1997. 12. C. Walshaw, M. Cross, M. Everett, and S. Johnson. A parallelizable algorithm for partitioning unstructured meshes. In A. Ferreira and J. D. P. Rolim, editors, Parallel Algorithms for Irregular Problems: State of the Art, pages 25–46, Dordrecht, 1995. Kluwer Academic Publishers.
Solving Discrete-Time Periodic Riccati Equations on a Cluster Peter Benner1 , Rafael Mayo2 , Enrique S. Quintana-Ort´ı2, and Vicente Hern´andez3 1
3
Zentrum f¨ ur Technomathematik, Fachbereich 3/Mathematik und Informatik, Universit¨ at Bremen, D-28334 Bremen, Germany; [email protected] 2 Departamento de Inform´ atica, Universidad Jaume I, 12080–Castell´ on, Spain; {mayo,quintana}@inf.uji.es Departamento de Sistemas Inform´ aticos y Computaci´ on, Universidad Polit´ecnica de Valencia, 46071–Valencia, Spain; [email protected]
Abstract. This paper analyzes the performance of a parallel solver for discrete-time periodic Riccati equations based on a sequence of orthogonal reordering transformations of the monodromy matrices associated with the equations. A coarse-grain parallel algorithm is investigated on a Myrinet cluster.
Key words: Discrete-time periodic Riccati equations, periodic linear control systems, parallel algorithms, multicomputers, cluster computing.
1
Introduction
Consider the discrete-time periodic Riccati equation (DPRE) Xk = Qk + ATk Xk+1 Ak − ATk Xk+1 Bk (Rk + BkT Xk+1 Bk )−1 BkT Xk+1 Ak , (1) where Ak ∈ IRn×n , Bk ∈ IRn×m , Ck ∈ IRr×n , Qk ∈ IRn×n , Rk ∈ IRm×m , and p is the period of the system, i.e., Ak+p = Ak , Bk+p = Bk , Ck+p = Ck , Qk+p = Qk , and Rk+p = Rk . Under mild conditions, the periodic symmetric positive semidefinite solution Xk = Xk+p ∈ IRn×n of (1) is unique [3]. DPREs arise, e.g., in the solution of the periodic linear-quadratic optimal control problem, model reduction of periodic linear systems, etc. [3]. Consider now the periodic symplectic matrix pencil, associated with the DPRE (1), Ak 0 In Bk Rk−1 BkT Lk − λMk = (2) −λ ≡ Lk+p − λMk+p , −Qk In 0 ATk
Supported by the Conseller´ıa de Cultura, Educaci´ on y Ciencia de la Generalidad Valenciana GV99-59-1-14, the DAAD Programme Acciones Integradas HispanoAlemanas, and the Fundaci´ o Caixa-Castell´ o Bancaixa.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 824–828, 2000. c Springer-Verlag Berlin Heidelberg 2000
Solving Discrete-Time Periodic Riccati Equations on a Cluster
825
where In denotes the identity matrix of order n. If all the Ak are invertible, the solution of the DPRE is given by the d-stable invariant subspace of the periodic monodromy matrices [5,8] −1 Πk = Mk+p−1 Lk+p−1 · · · Mk−1 Lk ,
Πk = Πk+p .
(3)
Note that the monodromy relation still holds if (some of) the Ak are singular and the algorithm presented here can still be applied in this case [3,5,8]. A numerically sound DPRE solver relies on an extension of the generalized Schur vectors method [5,8]. However, the parallel implementation of this QR-like algorithm renders an efficiency and scalability far from those of traditional matrix factorizations [4]. In this paper we follow a different approach, described in [2], for the solution of DPREs based on a reliable swapping of the matrix products in (3). In section 2 we briefly review the “swapping” method for solving DPREs and present a coarse-grain DPRE solver. A medium-grain parallel DPRE solver was investigated in [9]. In section 3 we report the performance of our approach on a cluster of Intel Pentium-II processors.
2
Parallel Solution of DPREs
In [2] an algorithm is introduced for solving DPREs without explicitly forming the monodromy matrices. The approach relies on the following lemma. Lemma. Consider Z, Y ∈ IRn×n , with Y invertible, and let Q11 Q12 Y R = (4) −Z 0 Q21 Q22 −1 be a QR factorization of [Y T , −Z T ]T ; then Q−1 . 22 Q21 = ZY By (Y, Z) ← swap(Y, Z) we denote the application of the lemma to the matrix pair (Y, Z), where Y and Z are overwritten by Q22 and Q21 , respectively. Applying the swapping procedure to (3), we obtain reordered monodromy matrices of the form
ˆ k = (M ˆ −1 L ¯k · · · M ¯ k+p−1 )−1 (L ¯ k+p−1 · · · L ¯ k ), Πk = M k
(5)
without computing any explicit inverse. The solution of the corresponding DPRE is then computed by solving the discrete-time algebraic Riccati equation (DARE) ˆ k [10]. ˆ −1 L associated with the corresponding matrix pair M k The algorithm can be stated as follows [2]: Input: p matrix pairs (Lk , Mk ), k = 0, 1, . . . , p − 1 Output: Solution of the p DPREs associated with the matrix pairs for k = 0, 1, . . . , p − 1 ˆ (k+1) mod p = Mk ˆ k = Lk , M Set L end for t = 1, 2, . . . , p − 1
826
Peter Benner et al. for k = 0, 1, . . . , p − 1 (L(k+t) mod p , Mk ) ← swap(L(k+t) mod p , Mk ) ˆk ˆ k ← L(k+t) mod p L L ˆ (k+t+1) mod p Mk ˆ M(k+t+1) mod p ← M end end for k = 0, 1, . . . , p − 1 ˆk) ˆk, M Solve the DARE associated with (L end
The procedure is only composed of QR factorizations and matrix products. The computational cost of the reordering algorithm is O(p2 n3 ) flops (floatingpoint arithmetic operations) and O(pn2 ) for workspace. The cost of the solution of the p DAREs at the final stage is O(pn3 ) flops and O(pn2 ) for workspace. Consider a parallel distributed-memory architecture, composed of np processors, P0 , P1 ,. . . , Pnp −1 , and, for simplicity, assume that p = np . A coarse-grain parallel algorithm can be stated as follows: Input: p matrix pairs (Lk , Mk ), k = 0, 1, . . . , p − 1 Output: Solution of the p DPREs associated with the matrix pairs for k = 0, 1, . . . , p − 1 in parallel Assign (Lk , Mk ) to processor Pk ˆ k = Lk , M ˆ (k+1) mod p = Mk Set L Send Lk to P(k+p−1) mod p Receive L(k+1) mod p from P(k+1) mod p end for t = 1, 2, . . . , p − 1 for k = 0, 1, . . . , p − 1 in parallel (L(k+t) mod p , Mk ) ← swap(L(k+t) mod p , Mk ) ˆ (k+t) mod p to P(k+p−1) mod p Send M ˆ k ← L(k+t) mod p L ˆk L ˆ (k+t+1) mod p from P(k+1) mod p Receive M Send L(k+t) mod p to P(k+p−1) mod p ˆ (k+t+1) mod p Mk ˆ (k+t+1) mod p ← M M Receive L(k+t+1) mod p from P(k+1) mod p end end for k = 0, 1, . . . , p − 1 in parallel ˆk) ˆk, M Solve the DARE associated with (L end
This parallel algorithm only requires efficient serial kernels for the QR factorization and the matrix product, like those available in LAPACK and BLAS [1], and a serial DARE solver based, e.g., on the matrix disk function [2]. Moreover, the algorithm presents a highly regular and local communication pattern as, at each iteration of the outer loop t, the only communications neccesary are the left ˆ k . Computation and communication are overlapped in circular shifts of Lk and M the algorithm, and the computational load in the algorithm is perfectly balanced.
Solving Discrete-Time Periodic Riccati Equations on a Cluster
827
In case p > np , we can assign the matrix pairs (Lk , Mk ) cyclically to the processors, and proceed in the same manner.
3
Experimental Results
All the experiments were performed on a cluster of Intel Pentium-II processors connected via a Myrinet switch, using IEEE double precision floating-point arithmetic ( ≈ 2.2 × 10−16 ). BLAS and MPI implementations specially tuned for this architecture were employed [7]. Performance experiments with routine DGEMM achieved 200 Mflops (millions of flops per second) on one processor. Figure 1 evaluates the efficiency of our coarse-grain parallel algorithm for n=100 and 200, and np = p = 2, 4, . . . , 10 processors. We obtain efficiencies higher than 1 due to the better management of the memory achieved in the parallel algorithms, which have to deal with a smaller number of matrices. In this figure we also report the scalability of the parallel algorithm. For this purpose, we evaluate the Mflops per processor, with n2 p/np , p = np , fixed at 200. As the figure shows, the scalability is close to optimal.
150
Mflops per processor
1
Efficiency
0.8
0.6
0.4
100
50
0.2
0 0
2
4
6
Number of processors
8
10
0 0
5
10
15
20
Number of processors
25
30
Fig. 1. Efficiency (left) and scalability (right) of the parallel algorithm for n=100 (solid line) and 200 (dotted line).
References 1. E. Anderson et al. LAPACK Users’ Guide. SIAM, Philadelphia, PA, 1994. 2. P. Benner. Contributions to the Numerical Solution of Algebraic Riccati Equations and Related Eigenvalue Problems. Dissertation, Fak. f. Mathematik, TU Chemnitz–Zwickau, Chemnitz, FRG, 1997. 3. S. Bittanti, P. Colaneri, and G. De Nicolao. The periodic Riccati equation. In S. Bittanti, A.J. Laub, and J.C. Willems, editors, The Riccati Equation, pp. 127– 162. Springer-Verlag, Berlin, 1991. 4. L. S. Blackford et al. ScaLAPACK Users’ Guide. SIAM, Philadelphia, PA, 1997.
828
Peter Benner et al.
5. A. Bojanczyk, G.H. Golub, and P. Van Dooren. The periodic Schur decomposition; algorithms and applications. In Proc. SPIE Conference, 1770, pp. 31–42, 1992. 6. G.H. Golub and C.F. Van Loan. Matrix Computations. John Hopkins University Press, Baltimore, 1989. 7. W. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Programming with the Message-Passing Interface. MIT Press, 1994. 8. J.J. Hench and A.J. Laub. Numerical solution of the discrete-time periodic Riccati equation. IEEE Trans. Automat. Control, 39:1197–1210, 1994. 9. R. Mayo, E.S. Quintana-Ort´ı, E. Arias, and V. Hern´ andez. Parallel Solvers for Discrete-time Periodic Riccati Equations. Lecture Notes in Computer Science. Springer–Verlag, 2000. To appear. 10. V. Mehrmann. The Autonomous Linear Quadratic Control Problem, Theory and Numerical Solution. Number 163 in Lecture Notes in Control and Information Sciences. Springer-Verlag, Heidelberg, July 1991.
A Parallel Optimization Scheme for Parameter Estimation in Motor Vehicle Dynamics Torsten Butz1 , Oskar von Stryk1 , and Thieß-Magnus Wolter2 1
2
Chair of Numerical Analysis (M2), Zentrum Mathematik, Technische Universit¨ at M¨ unchen, D-80290 M¨ unchen, Germany http://www-m2.ma.tum.de TESIS DYNAware, Implerstraße 26, D-81371 M¨ unchen, Germany http://www.tesis.de
Abstract. For calibrating the vehicle model of a commercial vehicle dynamics program a parameter estimation tool has been developed which relies on observations obtained from driving tests. The associated nonlinear least-squares problem can be solved by means of mathematical optimization algorithms most of them making use of first-order derivative information. While the complexity of the investigated vehicle dynamics program only allows the objective gradients to be approximated by means of finite differences, this approach enables significant savings in computational time when performing the additionally required evaluations of the objective function in parallel. The employed low-cost parallel computing platform which consists of a heterogeneous PC cluster is well suited for the needs of the automotive suppliers and industries employing vehicle dynamics simulations.
1
Introduction
The numerical simulation of vehicle dynamics has gained considerable significance in automotive development, since it enables the thorough investigation of a novel vehicle in advance. Besides reducing the need for physical prototyping, real-time simulations may be used within hardware-in-the-loop test-benches which allow active control units, such as anti-lock braking systems and electronic stability programs, to be tested without danger for test driver and vehicle. The development of complex electronic devices requires the virtual car to reproduce the behavior of the real vehicle in detail. Therefore, we employ a sophisticated vehicle model which comprises a suitable multibody system, including force elements and kinematical connections, as well as a realistic tire model. The use of a tailored modeling technique enables the entire vehicle dynamics to be described by a large system of ordinary differential equations. Specifically for the use in a test-bench the calibration of the vehicle model need often be accomplished on the spot. For this purpose, nonlinear optimization algorithms and careful numerical differentiation can be combined to yield a parallel parameter estimation scheme which is suitable for low-cost computing platforms such as heterogeneous PC networks. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 829–834, 2000. c Springer-Verlag Berlin Heidelberg 2000
830
2
Torsten Butz, Oskar von Stryk, and Thieß-Magnus Wolter
Simulation of Full Motor Vehicle Dynamics
The vehicle dynamics program veDYNA [1] which has been employed for the following investigations is developed and commercially distributed by TESIS DYNAware, M¨ unchen. The vehicle model in veDYNA consists of a system of rigid bodies comprising the vehicle body, the axle suspensions and the wheels. In addition, partial models are employed to depict the characteristics of the drive train, the steering mechanism and the tires. The use of suitable minimum coordinates and generalized velocities avoids the need for algebraic constraints in the equations of motion [9]. Thus, the vehicle dynamics can fully be described by a system of 56 highly nonlinear ordinary differential equations. Due to the stiffness of the system its numerical integration is carried out by a semi-implicit Euler scheme. For a realistic implementation of virtual test drives on the computer also models for the driver and the road have been developed [3]. The numerical results obtained from veDYNA show good agreement with real vehicle behavior. Simulations with time steps in the range of milliseconds may be carried out in real-time on reasonable PC hardware.
3
Estimation of Vehicle Parameters
The equations of motion for the vehicle model in veDYNA are summarized by x(t) ˙ = g (x(t), p, t)
(1)
with suitable initial values x(t0 ) = x0 .
(2)
nx
np
Here, x(t) ∈ IR comprises the vehicle’s state variables, and p ∈ IR denotes the model parameters of interest which are constant for all times t. To adjust their values to the observed vehicle behavior, the nonlinear least-squares problem n
r(p) := minimize np p∈IR
t 1 1 2 2 f (p)2 := (ηij − xi (tj , p)) 2 2 j=1
(3)
i∈Ij
must be solved. Here, ηij , i ∈ Ij , are measurements of selected vehicle state variables at the times tj throughout a driving test, and x(t, p) denotes the corresponding numerical solution of (1), (2). Often additional box constraints li ≤ pi ≤ u i ,
i = 1, ..., np ,
(4)
on the parameter range have to be considered which shall ensure optimization results compatible with the real vehicle properties. For the solution of (3), (4) several gradient-based optimization methods as well as an evolutionary algorithm have been investigated [2]. In the sequel, we present results obtained from the Gauss-Newton method NLSCON [8], the Levenberg-Marquardt algorithm LMDER [7], the sequential quadratic programming method NLSSOL [5], and the implicit filtering code IFFCO [6], which is designed for solving noisy minimization problems.
A Parallel Optimization Scheme for Parameter Estimation
4
831
Parallel Optimization
For the solution of the parameter estimation problem a program frame was implemented which integrates veDYNA in the course of the optimization [2]. Due to the complexity of the employed vehicle model and the closely coupled numerical integration, the required objective derivatives cannot be determined by automatic or internal numerical differentiation techniques, but have to be approximated by means of finite differences. For the optimization with NLSCON, LMDER and NLSSOL the partial derivatives ∂i r(p) = f (p)T ∂i f (p) are obtained from the one-sided differences (∂i f (p))±hi =
f (p ± hi ei ) − f (p) ±hi
(5)
depending on the feasibility of p + hi ei or p − hi ei . Here, ei ∈ IRnp denotes the i-th canonical unit vector, and hi > 0 is a finite difference increment which must be chosen carefully such as to account for truncation, condition, and rounding errors. The implicit filtering code IFFCO makes use of the central differences (∂i r(p))2hi =
r(p + hi ei ) − r(p − hi ei ) 2hi
(6)
provided that both points are feasible; otherwise a one-sided difference is used as well. Accordingly, the computation of the gradient ∇r(p) requires np up to 2 np additional evaluations of the objective function. Since most effort is spent on the repeated integration of (1), (2), the computational time is much reduced by distributing these evaluations among further processors. For the one-sided differences the maximum speed-up is achieved, if np additional processors are available. In case of the centered differences one of the additionally required evaluations is performed by the client process, since the objective value at the current iterate need not be computed. The communication between client and server processes across the network is handled by remote procedure calls. For this purpose, the ONC RPC library from Sun Microsystems, ported to Microsoft Windows, is used [4]. The exchange of data is done via the UDP transport protocol, since only arguments of moderate size are communicated.
5
Results
The above parameter estimation scheme was successfully employed to adjust the lateral vehicle dynamics properties in the veDYNA model of a passenger car [2]. Appropriate values for the remaining coefficients of the vehicle model had been validated by TESIS DYNAware beforehand. The underlying data which was provided by an automotive supplier consisted of the steering wheel angle (cf. Fig. 1a) and the corresponding vehicle yaw rate recorded during multiple lane changes. The actual steering maneuver was preceded by a speed-up phase of 16.1 seconds. The sought vehicle parameters were
832
Torsten Butz, Oskar von Stryk, and Thieß-Magnus Wolter
2
0.4
1.5
0.3
1
0.2 yaw rate [rad/s]
steering wheel angle [rad]
given by the x-coordinate of the center of gravity and the cornering stiffnesses at the front and rear wheels which determine the lateral tire forces.
0.5 0 -0.5 -1
0 -0.1 -0.2
-1.5 -2
0.1
-0.3
measured values 18
20
22
24 26 time [s]
28
30
32
-0.4
measured values optimal solution 18
20
(a)
22
24 26 time [s]
28
30
32
(b)
Fig. 1. Steering wheel angle (a) and vehicle yaw rate (b) for the lane change maneuver.
The optimization was carried out on a heterogeneous Windows NT 4.0 and Windows 98 network at TESIS DYNAware, M¨ unchen. Initial guesses p0 = (−1.242, 27075.5, 27075.5)T
(7)
were chosen according to the default parameter values for the employed veDYNA vehicle model. The associated least-squares residual was r(p0 ) = 0.96115. The optimization produced a minimum residual r(p∗ ) = 0.30324 which was assumed for the parameter values p∗ = (−1.298, 16914.9, 15000.5)T
(8)
computed by IFFCO. A comparison between the observed vehicle yaw rate and the corresponding simulation results for (8) is depicted in Fig. 1b. Good agreement is achieved between both characteristics. However, the small values of the estimated stiffnesses indicate that the kinematical axles of the vehicle model cannot depict the elastic properties of the actual suspension system exactly. For reasons of comparison the numerical optimization was also carried out sequentially. In this case, the optimization including the computation of the gradients was done on a Dell 400 MHz PC where for each objective evaluation a CPU time of 8.1 seconds was needed. In the parallel framework, the optimization was running on the same machine. The three function evaluations for the one-sided differences in NLSCON, LMDER and NLSSOL were performed on two Siemens 450 MHz PCs and a Dell 333 MHz notebook where the objective evaluations took 7.3 seconds and 8.8 seconds of CPU time respectively. The two additional evaluations required by IFFCO were carried out on a further Dell 333 MHz notebook and a Siemens 300 MHz PC. The corresponding CPU times were given by 8.8 and 9.7 seconds.
A Parallel Optimization Scheme for Parameter Estimation
833
Table 1 shows a comparison between the results obtained from the different optimization codes [2]. The specified values consist of the least-squares residual at the respective optimal solution and the CPU times tseq and tpar which were needed for the sequential and the parallel optimization. Their ratio, i. e., the achieved parallel speed-up, is given in the last column. Also listed is the number nseq of objective evaluations during the entire optimization, and the share npar that was performed by the client process in the parallel approach. Table 1. Comparison of the computational results for the sequential and the parallel parameter estimation schemes. Algorithm IFFCO LMDER NLSCON NLSSOL
r(p∗ ) nseq tseq [s] npar tpar [s] tseq /tpar
0.30324 151 1223.6 0.30341 52 422.0 0.30455 0.30340
72 583.7 80 648.3
64 25 30 20
543.1 207.8 254.4
185.3
2.25 2.03 2.29
3.50
Applied to this problem the mentioned algorithms have produced meaningful parameter estimates with reasonably small residuals. The parallel execution of the finite difference computations reduces the required CPU times for all algorithms by more than a half. For NLSSOL, the computational time can be reduced to almost 25%, if equally fast remote processors are available. In case of the remaining optimization codes the achieved speed-up is significantly lower, since the employed line-search strategies do not allow a completely parallel treatment.
References 1. Anonymous: veDYNA User’s Guide. TESIS DYNAware, M¨ unchen (1997) 2. Butz, T.: Parameter Identification in Vehicle Dynamics. Diploma Thesis, Zentrum Mathematik, Technische Universit¨ at M¨ unchen (1999) 3. Chucholowski, C., V¨ ogel, M., von Stryk, O., Wolter, T.-M.: Real time simulation and online control for virtual test drives of cars. In: Bungartz, H.-J. et al. (eds.): High Performance Scientific and Engineering Computing. Lecture Notes in Computational Science and Engineering, Vol. 8. Springer-Verlag, Berlin (1999) 157–166 4. Gergeleit, M.: ONC RPC for Windows NT Homepage. World Wide Web, http:// www.dcs.qmw.ac.uk/˜williams/nisgina-current/src/rpc110/oncrpc.htm (1996) 5. Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: User’s Guide for NPSOL 5.0: A Fortran Package for Nonlinear Programming. Numerical Analysis Report 98-2, Department of Mathematics, University of California, San Diego (1998) 6. Gilmore, P.: IFFCO: Implicit Filtering for Constrained Optimization, User’s Guide. Technical Report CRSC-TR93-7, Center for Research in Scientific Computation, North Carolina State University, Raleigh (1993) 7. Mor´e, J.J.: The Levenberg-Marquardt algorithm: implementation and theory. In: Dold, A., Eckmann, B. (eds.): Numerical Analysis. Lecture Notes in Mathematics, Vol. 630. Springer-Verlag, Berlin Heidelberg (1978) 105–116
834
Torsten Butz, Oskar von Stryk, and Thieß-Magnus Wolter
8. Nowak, U., Weimann, L.: A family of Newton codes for systems of highly nonlinear equations. Technical Report TR 91-10, ZIB, Berlin (1991) 9. Rill, G.: Simulation von Kraftfahrzeugen. Vieweg, Braunschweig (1994)
Sliding-Window Compression on the Hypercube Charalampos Konstantopoulos, Andreas Svolos, and Christos Kaklamanis Computer Engineering and Informatics Department, University of Patras and Computer Technology Institute, 11 Aktaiou and Poulopoulou, GR 118 51, Athens, Greece {konstant,svolos,kakl}@cti.gr
Abstract. Dictionary compression belongs to the class of lossless compression methods and is mainly used for compressing text files [1, 2, 3]. In this paper, we present a parallel algorithm for one of these coding methods, namely the LZ77 coding algorithm also known as a slidingwindow coding algorithm. Although there exist PRAM algorithms [4, 5] for various dictionary compression methods, their rather irregular structure has discouraged their implementation on practical interconnection networks such as the mesh and hypercube. However in the case of LZ77 coding, we show how to exploit the specific properties of the algorithm in order to achieve an efficient implementation on the hypercube.
1
Introduction
The main idea of the LZ77 algorithm [6] is that strings in the text are replaced with pointers to a previous occurrences of these strings in the text. Specifically, let x[0 · · · N − 1] be the input string. Assume also the prefix x[0 · · · i − 1] has been compressed so far. The dictionary at this moment consists of all the substrings x[i − j · · · k] where j ∈ [1, · · · , M ], k ∈ [i − j, · · · , i − j + F − 1] and M , F are two parameters of the algorithm. The next step is to find the longest prefix of x[i · · · N ] which matches an entry of this dictionary. If this prefix is of length r and x[i−q · · · i−q +r −1] is the matching string in the dictionary (q ∈ [1 · · · M ]), then we replace the prefix x[i · · · i + r − 1] with the pointer (q, r) and we proceed to the position i + r of the input string. The string up to the position i + r − 1 has now been compressed. Notice that if the character xi does not occur within the last M preceding characters, then we cannot find any matching prefix at position i. In this case, we replace the character xi with the pointer (xi , 1) [7]. In this paper, we present an efficient implementation of the LZ77 (sliding window coding) algorithm on the hypercube network. A basic assumption of our implementation is that the employed multiprocessor is fine grained with only limited local memory per processing element (PE). Taking advantage of the properties of sliding-window coding algorithms we show that this kind of algorithms can be efficiently implemented on the hypercube network by using only a small number of fast communication primitives.
Work is supported in part by the General Secretariat of Research and Technology of Greece under Project ΠENE∆ 95 E∆ 1623.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 835–838, 2000. c Springer-Verlag Berlin Heidelberg 2000
836
2
Charalampos Konstantopoulos, Andreas Svolos, and Christos Kaklamanis
LZ77 Coding on the Hypercube
In the following, we present our parallel algorithm for sliding-window compression on a hypercube-based multiprocessor. The input string x[0 · · · N − 1] is distributed one character per PE, i.e character xi is initially stored in PE i where i = 0, · · · , N − 1. Our basic goal is to present an efficient parallel algorithm using the least amount of memory at each PE. For convenience, we also assume that N = 2n . In LZ77 coding algorithms the size M of the sliding window is finite and usually much smaller than the total length of the input string. In practice, a window of moderate size, typically M ≤ 8192, can work well for a variety of texts [1]. Due to this small value, we can perform the required string matching operations using only simple search techniques without increasing the computational overhead unduly. The parallel algorithm consists of two phases. First phase. In this phase, for each position of the input string we find the longest substring starting at this position which also matches an entry in the adaptive dictionary. Specifically, for each position i of the input string we determine the longest common prefix of string x[i · · · i + F − 1] with the strings x[i − mi · · · i − mi + F − 1] where mi ∈ [1, · · · M ]. This can be carried out in O(M log N ) time by executing a series of M shift and prefix sum operations [8, 9]. If each PE can communicate with all its neighbors at the same time (all-port capability), the above complexity can be reduced to O(M + log N ) time. This can be achieved by overlapping in time the execution of successive shift and prefix sum operations. Second phase. It can be easily seen that the sequence of pointers obtained from the sequential LZ77 coding algorithm is a subset of the pointers derived from the first phase of the parallel algorithm. The goal in the second phase is to determine which of these pointers will be finally included in the compressed file. This can be easily done in two stages by using the following simple technique. Let the pointer (mi , li ) of PE i point to the character i + li of the input string, that is the first character after the longest common prefix at position i. The first pointer in the compressed file is definitely that of position 0, namely (m0 , l0 ). The second pointer is that of position l0 , (ml0 , ll0 ). The third pointer is that of position l0 + ll0 and so on. Clearly, if we start from position 0 and follow the pointers defined above, we will eventually visit all the breakpoints1 defined by the sequential LZ77 parsing. This sequential traversal of pointers can be performed in parallel in O(log N ) steps by using the well-known pointer jumping technique [10]. As this is important, each PE saves the addresses of PEs it visits at each pointer jumping step. However, only O(log N ) local memory at each PE is needed for this information, since each PE visits at most log N PEs during these steps. We prove the following lemma: 1
Breakpoints are the positions in the input text at which the parsing process splits the text.
Sliding-Window Compression on the Hypercube
837
Lemma 1. If pi , pj are the pointers of PEs i, j respectively at a pointer jumping step , then ∀ i,j i ≤ j ⇒ pi ≤ pj . Proof. We prove the lemma by induction on the number of elapsing pointer jumping steps. In the first step, pointer pi is equal to i+li and thus the inequality pi ≤ pj can be written as i + li ≤ j + lj . We distinguish two cases. If i + li ≤ j, it is obvious that the above inequality holds. Now consider the case i ≤ j < i + li . Clearly, character xj belongs to the longest common prefix of position i and thus the longest common prefix at position j cannot be smaller than i + li − j characters, that is lj ≥ i + li − j ⇒ i + li ≤ j + lj . Thus we have proved the lemma for the first step. Assume now that the statement ∀ i,j i ≤ j ⇒ pi ≤ pj holds for all the elapsing pointer jumping steps up to the step k. We will prove that it also holds for the step k + 1. Suppose that at step k position i points to position ai and position ai in turn points to position bi . After step k + 1, position i will point to position bi . We can prove that ∀ i,j i ≤ j ⇒ bi ≤ bj . If j ≥ bi , the proof is obvious. Consider now the case i ≤ j < bi . From the induction hypothesis, it follows that ai ≤ aj . If bi > bj then the statement ai ≤ aj ⇒ bi ≤ bj does not hold, thereby contradicting the induction hypothesis. Thus the lemma is true for the step k + 1 as well.
Now it is clear that each pointer jumping step can be performed using monotone routing [8, 9] in place of expensive sorting steps. Each step takes O(log N ) time and thus the O(log N ) pointer jumping steps of the first stage can be performed in O(log2 N ) time overall. After the above pointer jumping steps, the next stage is to “mark” the positions of the input string that are breakpoints in the sequential LZ77 parsing. PE 0 knows that its position, position 0, is a breakpoint. It also knows O(log N ) positions which are certainly breakpoints as well. These positions correspond to the PEs which it visited during the pointer jumping steps. Let i1 , i2 , · · · , iO(log N ) be the addresses of these processors. PE 0 should notify them that hold breakpoints. After a PE, say PE ik , receives the notification, it should in turn notify those PEs in the interval [ik +1 · · · ik+1 −1] which it has visited during the pointer jumping steps. This process proceeds recursively and finally all the breakpoints of the sequential LZ77 parsing are marked. It can be easily seen that this recursive marking can be carried out by reversing the steps of the first stage. At each step, each PE that has already received a notification packet sends such a packet to the PE that it had visited at the corresponding pointer jumping step of the first stage. The communication complexity of each step is only O(log N ) since we can use monotone routing again. We have described the second phase of the parallel LZ77 coding algorithm. The complexity O(log2 N ) of this phase is mainly due to the fact that pointer jumping steps are performed along a string of length N . However, it is possible to limit pointer jumping steps along shorter segments of the input string, thereby largely decreasing the complexity of the second phase. Let us see how this can be done. After the end of the first phase, each PE i holds the pointer (mi , li ) in
838
Charalampos Konstantopoulos, Andreas Svolos, and Christos Kaklamanis
its local memory. Then, using a prefix sum operation, each PE i estimates the expression maxi = max(e0 , e1 , · · · , ei−1 )2 where ei = li + i (O(log N ) delay). One can easily notice that if i ≥ maxi , pointer (mi , li ) is definitely included in the compressed file. The corresponding position i is called cut-point; cut-point is a position in the input string for which we are certain in advance that it is one of the breakpoints of the LZ77 parsing [1]. Clearly, cut-points split the input string into non-overlapping substrings and thus we can execute the second phase of the parallel LZ77 coding algorithm independently for each substring. Due to the shorter length of these substrings, the complexity of the second phase is largely decreased. Specifically, if L is the length of the longest substring between two successive cut-points, the second phase can be executed now in O(log L·log N ) time. In practice, L 100 since cut-points almost always occur well under 100 characters apart [1].
3
Conclusions
We presented an efficient parallel algorithm for LZ77 coding on the hypercube network. General simulations of PRAM dictionary compression algorithms on the hypercube surely leads to solutions with high communication overhead. However, by carefully examining the way LZ77 parsing splits the text into phrases, we managed to considerably lower the communication overhead. In doing so, we used only a small set of communication primitives which can be efficiently executed on the hypercube network. In addition, we further enhanced the performance by exploiting known facts from the text compression (frequent cut-points).
References [1] T. C. Bell, J. G. Cleary, I. H. Witten, Text Compression Prentice Hall Advanced Reference Series Computer Science, (1990). [2] K. Sayood, Introduction to Data Compression Morgan Kaufmann Publishers Inc. (1996). [3] J. A. Storer, Data Compression Methods and Theory, Computer Science Press, Rockville, MD (1988). [4] L. M. Stauffer, D. S. Hirschberg, Dictionary Compression on the PRAM, Parallel Processing Letters 7 (3) 1997 297–308. [5] M. Farach, S. Muthukrishnan, Optimal Parallel Dictionary Matching and Compression (Extended Abstract), in Proc. SPAA 1995, 244–253. [6] J. Ziv, A. Lempel, A Universal Algorithm for Sequential Data Compression, IEEE Trans. Inf. Theory 23 (3) 1977 337–343. [7] T. C. Bell, Better OPM/L Text Compression, IEEE Trans. Communications COM34 (12) 1986 1176–1182. [8] S. Ranka, S. Sahni, Hypercube Algorithms with Applications to Image Processing and Pattern Recognition, Springer Verlag (1990). [9] T. F. Leighton, Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes, Morgan Kaufmann Publishers, San Mateo, CA (1992). [10] J. J` aj` a, An Introduction to Parallel Algorithms, Addison-Wesley (1992). 2
We assume max0 = 0.
A Parallel Implementation of a Potential Reduction Algorithm for Box-Constrained Quadratic Programming Marco D’Apuzzo1 , Marina Marino2 , Panos M. Pardalos3, and Gerardo Toraldo2 1 2
Seconda Univerit` a di Napoli & CPS-CNR, Napoli, Italia, [email protected] Univerit` a di Napoli Federico II & CPS-CNR, Napoli, Italia, (marino, toraldo)@matna2.dma.unina.it 3 University of Florida, Gainesville Florida, USA, [email protected]
Abstract. In this paper we describe a parallel version of the potential reduction algorithm for MIMD distributed memory machines, in which the computational kernels arising at each step of the algorithm are concurrently performed by using standard parallel software environments. This approach is shown to be very effective, in contrast to what happens in the active set strategies where the linear algebra computational kernels represent a serious drawback to an effective parallel implementation. The computational results show the effectiveness of our approach.
1
Introduction
In this paper we describe a parallel implementation of an interior point algorithm, proposed by Han et al. [8] for the Box-Constrained Quadratic Programming (BCQP) problem, based on the parallelization of the linear algebra kernels arising at each iteration. Our problem can be stated as follows: minimize subject to
1 T x Ax − bT x 2 x ∈ Ω = {x : l ≤ x ≤ u}. f (x) =
(1) (2)
Here A is a n × n symmetric, positive definite matrix, b, l and u are known n-vectors. Moreover, all inequalities involving vectors are interpreted componentwise, and ∇f (x) = Ax − b is the gradient of f . For the sake of simplicity we assume that the bounds, l and u, are finite. Until the advent of interior point algorithms in the mid 1980s [9], the computational scene was dominated by the active set algorithms, implemented in many mathematical software libraries (NAG, IMSL). In contrast with active set
This work was partially supported by the MURST national projects “Analisi Numerica: Metodi e Software Matematico” and “Algorithms for Complex Systems Optimization”.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 839–848, 2000. c Springer-Verlag Berlin Heidelberg 2000
840
Marco D’Apuzzo et al.
algorithms which generate a sequence of extreme points, a generic interior point method generates a sequence of points in the interior of Ω. The algorithm we consider in the present paper is the so-called potential reduction algorithm [8,10,14], successfully used for linear complementarity and bound constrained quadratic problems. We consider the case in which the matrix A is dense and a direct solver is used in the inner iterations. Then, each iteration of the potential reduction algorithm has O(n3 ) computational complexity, and therefore much higher than in an active set strategy where the computational cost per iteration varies between O(n3 ) (if the Cholesky factorization must be recomputed) and O(nm) (if either a projected gradient step or a simple factorization update must be computed). On the other hand, it has been observed that in practice the number of iterations in interior point algorithms does not depend on the number of variables; in contrast the number of iterations for an active set strategy strongly depends on the number of variables (see for example the discussion and the computational results in [11]). We note that the computational kernels of the two approaches are quite different, and so are their parallel features. The parallel implementation of the projected gradient method proposed by Mor´e and Toraldo [11] has been recently discussed by D’Apuzzo et al. [5,6], who pointed out that the main drawback in implementing an active set strategy are the linear algebra operations that at each iteration must be performed on matrices and vectors reduced with respect to the current set of the free variables. In theory, working just on the free variables represents an advantage, because if few variables are free at the solution, eventually very small problems must be solved. Unfortunatly, the standard linear algebra package LAPACK [1] can not be used, as well as the BLAS, [1] to work on submatrices and on subvectors. In addition, the very poor parallel properties of the factorization updating process (due to the large amount of communication among processors), joint with the possible unbalanced data distribution (due to the modification of the active set at each iteration) represent a serious drawback for an efficient parallel implementation. Linear algebra operations on matrices and vectors whose size changes dynamically at each iteration and drastically reduces in the last iterations, are non-suitable for a good parallel implementation. This is also confirmed from the fact that these non standard operations are not implemented in the most popular high performance mathematical libraries. Because of this, the parallel performances of the projected gradient method in [6] appear to be not very satisfactory, despite the considerable efforts required by the algorithm implementation. We note that the key issues arising in the development of a parallel version of the algorithm in [11] are common to any nonlinear optimization algorithms based on active set strategies (matrix updating processes, linesearches, operations with subvectors, etc.). On the other hand, the computationally expensive iterations of the interior point algorithms turn out to be an advantage, since they can be efficiently parallelized. In this paper we describe a parallel version of the potential reduction algorithm for MIMD distributed memory machines, in which the computational kernels arising at each step of the algorithm are concurrently performed by using the standard ScaLAPACK [3] environment.
A Parallel Implementation of a Potential Reduction Algorithm
841
The outline of our paper is as follows. In §2, we review the potential reduction algorithm as presented in [8]. The main issues arising in making a parallel implementation of the potential reduction algorithm are described in §3. Finally, in §4 we present the computational results of the implementation of the parallel algorithm on a MIMD distributed memory machine. Moreover, a comparison among the sequential versions of the projected gradient [11] and the potential reduction algorithms is shown.
2
The Potential Reduction Algorithm for Quadratic Problems with Box Constraints
In this section we outline the primal-dual Potential Reduction (PR) algorithm for solving the BCQP problem (1-2), that was proposed by Han et al. in [8], who extended the primal-dual PR algorithm for the convex linear complementarity problem, developed by Todd and Ye in [12]. Problem (1)-(2) can be transformed to the standard form minimize subject to
1 T x Ax − bT x 2 x + z = e and x, z ≥ 0,
(3)
f (x) =
(4)
where z ∈ IRn , e is the unit vector, and whose dual is: 1 max eT y − xT Ax, 2
s.t. sx = Ax − b − y ≥ 0 and sz = −y ≥ 0,
(5)
where y ∈ IRn . The duality gap of (3)-(4) and (5) is given by: ∆=
1 1 T x Ax − bT x − (eT y − xT Ax) = xT sx + z T sz ≥ 0. 2 2
(6)
The interior points x, z, sx , and sz are characterized by using the primal-dual potential function of Todd and Ye [12]: φ(x, z, sx , sz ) = ρ ln(xT sx + z T sz ) −
n i=1
ln(xi (sx )i ) −
n i=1
ln(zi (sz )i ),
√ where ρ ≥ 2n + 2n. The relationship between the duality gap and the potential reduction function was shown by Ye [14], who proved that if φ(x, z, sx , sz ) ≤ −O((ρ − 2n)L) then ∆ < 2−O(L) , where L is the size of input data (see [14]). Therefore, the method is based on the minimization of the potential function, in order to minimize the duality gap. Kojima et al. [10] showed that the Newton direction of the nonlinear system ∆ XSx e = ρ e (7) ZSz e = ∆ e, ρ
842
Marco D’Apuzzo et al.
with fixed ∆, guarantees a constant reduction in the potential function. In (7) the upper-case letter (X) designates the diagonal matrix of the vector (x) in lower-case, and the slack vectors are z = e − x,
sx = Ax − b − y,
sz = −y.
(8)
Following the Newton direction, given the kth iterates xk and y k , that are interior feasible points for the primal (3)-(4) and the dual (5), it can be verified that the search direction {δx, δz, δy} can be computed by solving the systems: −1 ∆k ¯ k )−1 e − D(Z ¯ δx = ¯ D ¯ + Sxk Z k + Szk X k D ¯ k )−1 e − D(s ¯ kx − skz ) D(X DA ρ (9) k ∆ ¯ k )−1 e − Ds ¯ −1 δx + ¯ k , ¯ = SkX k D D(Z (10) − Dδy z z ρ ¯ is the diagonal matrix, used to deal with where δx = −δz. In (9) and (10) D ill-conditioning [2,13], whose diagonal coefficients are given by: d¯ii = xki zik for i = 1, · · · , n. After obtaining the direction, the step length θ¯ can be simply chosen as θ¯ = β max{θ : xk + θδx ≥ 0, e − (xk + θδx) ≥ 0, −y k − θδy ≥ 0, and sx k + θ(Qδx − δy) ≥ 0} for some 0 < β < 1,
(11)
and the (k + 1)th iterates are generated as ¯ , xk+1 = xk + θδx
¯ y k+1 = y k + θδy.
Han et al. in [8] observed that for ρ large enough xk+1 and y k+1 guarantee a constant reduction in the potential function. This suggests that the use of the line-search is unnecessary and can be replaced by (11), which is used in our implementation. Now we are ready to summarize the primal-dual PR algorithm: PR algorithm for BCQP T
Choose x0 = .5e and α = max{1, 3 Ax0 − b − e2n (Ax0 − b)} z 0 = e − x0 ; y 0 = −αe; sx 0 = Ax0 − b − y 0 ; sz 0 = −y 0 T T ∆0 = (x0 ) sx 0 + (z 0 ) sz 0 ; k = 0 k −L while (∆ ≥ 2 ) do begin Compute δx and δy from (9) and (10) Let θ¯ be given by (11) ¯ xk+1 = xk + θδx z k+1 = e − xk+1 ¯ y k+1 = y k + θδy sx k+1 = Axk+1 − b − y k+1
A Parallel Implementation of a Potential Reduction Algorithm
843
sz k+1 = −y k+1 T T ∆k+1 = (xk+1 ) sx k+1 + (z k+1 ) sz k+1 k =k+1 end endwhile Regarding the theoretical complexity of the algorithm, the following theorem holds [8]. √ Theorem 1. Let assume A and b have integer entries; let ρ ≥ 2n + 2n and L = 2n2 + [log |P |] where P is the product of the nonzero integer coefficients appearing in A and b. Then, the algorithm terminates in O((ρ − 2n)L) iterations and each iteration uses O(n3 ) arithmetic operations.
3
A Parallel Version of PR Algorithm
In this section we present a parallel version of the algorithm described in the previous section to solve the BCQP problem. Our target computational environment is the MIMD distributed memory one. The basic idea to obtain such parallel version was to perform concurrently the linear algebra operations at each iteration of PR algorithm. Therefore, we focused our attention on the choice of appropriate strategies for an efficient parallel implementation of these computational kernels. A first step in such direction was to consider and to use the ScaLAPACK environment. ScaLAPACK [3] is a well known high performance library designed to solve linear algebra problems on distributed memory multiprocessors. It is based on a parallel computational model which provides a convenient and flexible message passing framework for developing efficient and portable parallel software without worrying about the underlying architecture details. Such a model is implemented into BLACS [7], a message passing library that provides tools to perform basic communication operations on matrices or submatrices. We assume that the computational model consists of a two dimensional grid of processes, where each process stores and operates on blocks of block-partitioned matrices or vectors. The blocks are distributed in a cyclic block fashion over the processes and their size represents both the distribution block size and the computational blocking factor used by processes to perform most of the computation. In particular the ScaLAPACK routines are based on block-partitioned algorithms in order to minimize the frequency of data movement between different levels of the memory hierarchy. The choice of the block size has a major impact on load balance, communication cost, and loop and index computation overhead of the parallel algorithm, determining its performance and scalability. A larger block size results in a greater load imbalance, but reduces the frequency of the communication among processes. On the other hand, a smaller block size leads to a better load balance, but results in an inefficient use of the memory hierarchy. However, the
844
Marco D’Apuzzo et al.
performance of the ScaLAPACK routines is not very sensitive to the block size as long as the extreme cases are avoided [3]. In developing our parallel algorithm, we assumed to have a process grid with R process rows and C process columns and each process is identified by its coordinates within the grid using the notation r, c, 0 ≤ r < R, 0 ≤ c < C. We decomposed the Hessian matrix A of BCQP problem into square blocks Ai,j , i, j = 1, ..., N B, where N B = N/BS and BS is the block size, in order to obtain a square block matrix of dimension N B. These blocks are uniformly distributed among the process grid so that the process Pr,c holds the blocks Ai,j such that (i − 1)modR = r and (j − 1)modC = c. We also decomposed vectors into N B blocks, each of dimension BS, and distributed them along each process column according to the matrix distribution. Next, we analyze the PR algorithm. Each iteration requires: – 2 inner products (to compute ∆), – 2 matrix-vector products (to compute θ¯ and sx ), – a linear system solution (to compute δx). The above linear algebra operations can be efficiently implemented using suitable routines from the ScaLAPACK library. In this context, starting from the described data distribution strategy, our goal was to allow processes to perform parallel computation and communication, so that each of them holds only the blocks of the resulting vectors needed to update the corresponding blocks of the iterates xk , y k . Specifically, analyzing a single iteration of the algorithm, the basic computational and communication steps are organized as follows: – Computation of δx. It is performed in two steps. The first one consists of the concurrent generation of the linear system (9) to be solved. Each process generates appropriate blocks of the coefficient matrix and known vector, so that the linear system is distributed among the process grid according to the initial distribution strategy. The second step consists of the parallel block solution of the linear system, and the final vector solution is distributed along the process columns as well as the other vectors. – Computation of δy. The coefficient matrix of the linear system (10) related to δy is diagonal, so each process can compute explicitly its blocks of the vector solution. – Computation of θ. It requires to find the maximum value (11) among values for which some vectors are non negative. One of such vectors is obtained by performing a matrix-vector product. Once this computation is done in parallel, each process computes the maximum value related to the components of the vectors it holds. After that, a global combine operation among all processes is needed to obtain each process has the global maximum. – Vectors updating. Each process is ready to update vectors x, y, z, and sz , while to update sx another parallel matrix-vector product is needed.
A Parallel Implementation of a Potential Reduction Algorithm
845
– Computation of ∆ The computation of (6) requires two parallel inner products, then two global combine operations.
4
Computational Results
In this section we present the results of an implementation of the parallel PR algorithm on an IBM RS6000 SP machine hosted by CPS-CNR. The IBM SP is a MIMD distributed memory machine with 16 Power2 Super Chip Thin nodes running at 160 Mhz, each with 512 MBytes of memory and a 128 KBytes L1 cache. The nodes are interconnected via a proprietary switch, with a peak bi-directional bandwidth of 110 MBytes/sec. All implementations were carried out using the AIX XL Fortran compiler and the BLACS, which are implemented on the top of the IBM proprietary version 2.3 of MPI message passing library. Furthermore some routines from PBLAS and ScaLAPACK were used. The PBLAS [4] is a package representing the parallel version of the BLAS and it is addressed to distributed basic linear algebra operations with the aim of simplifying the parallelization of linear algebra codes. The PBLAS is used to develop the ScaLAPACK library and it is part of it. Specifically we used the PDSYMV and PDDOT routines from PBLAS to perform matrix-vector and inner products respectively; the PDPOSV routine from ScaLAPACK for solving a linear system with symmetric and positive definite matrix; the DGSUM2D and DGAMN2D routines from BLACS to perform the combine communication operations required by the parallel algorithm. In our computational experiments we used test problems similar to the problems introduced by Mor´e and Toraldo in [11], and also used in the paper by Han et al. [8]. In order to construct such test problems, 5 parameters are considered: the number of variables (n), the magnitude of the condition number of A (ncond), the number of active constraints at the solution x∗ (na(x∗ )), the magnitude of the amount of degeneracy of x∗ (ndeg) 1 , and the number of active constraints at the starting point x0 (na(x0 )). For the results in this section we set ncond = 6 and ndeg = −6, that is we consider problems that are both midly degenerate and ill-conditioned. Furthemore, the value of na(x∗ ) is set to 50, which means that about half of the constraints are active at the solution. In our experiments we first performed some computations in order to compare the PR algorithm with a method based on an active set strategy. In particular, we considered the algorithm (henceforth called PG algorithm) proposed by Mor´e and Toraldo, that combines standard active set strategies with the gradient projection method. Specifically, this algorithm alternates gradient projection steps, in order to identify the optimal active set, and Newton steps, in order to explore the face determinated by such active set. We used an existing Fortran 77 code based on PG algorithm, that we compiled and executed on one processor of the IBM SP. 1
The degeneracy parameter ndeg is such that [∇f (x∗ )]i = ±10ri ndeg ∀i : x∗i = ui or x∗i = li , where ri ∈ (0, 1) is randomly generated.
846
Marco D’Apuzzo et al.
In the PR algorithm the starting point should be an interior point of the feasible domain, so we do not consider the parameter na(x0 ), and we choose x0 as the middle point of the feasible domain. Furthemore, since according to some computational results Han et al in [8] concluded that, from the convergence rate point of view, a good combination of values for the two parameters β and ρ is given by β = 0.99 and ρ = n1.5 , we use such values in our experiments. For the stopping criteria, we used ∆k ≤ 10−5 , and additionally ∆k /(1 + |f (xk )|) ≤ 10−8 . For the PG algorithm the choice of the active set at the starting point is made using the same technique used for the solution, on the basis of the parameter na(x0 ). In particular, the values of na(x0 ) are restricted to 10, 50 and 90, that is we considered only the cases when about 10%, 50% and 90% of the constraints are active at the starting point, respectively. Iterations are stopped when g(xk ) ≤ 10−10 , where g(xk ) is the residual of the Kuhn-Tucker conditions. Table 1 shows the CPU times (in seconds) and the number of iterations required to solve the test problem with both PR and PG algorithms on one processor of the IBM SP2, varying the problem size from 504 to 2520. For the PG algorithm we report times for the 3 mentioned different percentages of active constraints at starting point and their average. In the last column of the table ratios between the CPU times of PR algorithm and the average times of PG algorithm are shown. We first observe that these results confirm that for the PR algorithm the number of iterations is independent of the problem size. This is not true for the PG algorithm, where the number of iterations varies when the problem size grows up. Such behavior justifies the fact that, although the computational cost of each iteration of PR algorithm is greater than that of PG algorithm of about 1 order of magnitude, the PR algorithm always requires less time than the PG algorithm. So we can state that PR algorithm is competitive with a well representative algorithm from the active set class. However it is worth to note that we are comparing results obtained from an implementation of PR algorithm that uses collections of high-performance linear algebra routines tailored and optimized to the IBM SP architecture with results obtained from an existing PG code which has not been tailored to the target machine. In any case, the results obtained show that the PR algorithm can be considered an effective tool to solve medium size dense problems. Next, we investigated the efficiency of the parallel version of PR algorithm. The number of processors used is 6, 9 and 12, logically organized as 2 × 3, 3 × 3, and 3 × 4 grids, respectively. In Table 2 we report the overall CPU times (Tp ), the CPU times (tp ) needed to solve the linear systems (9), and the efficiency values (Ep ) obtained varying the problem size from 1008 to 3528. Note that the CPU times Tp and tp are given in seconds and refer to the parallel algorithm on p processors, and Ep is defined as Ep = T1 /pTp . In the last column of the table, the efficiency values for 12 processors with respect to 6 processors are shown. Moreover, the results shown have been obtained using BS = 42. Our computational experiences done using different block size values showed that BS = 42 leads to good performance on the considered target machine. Moreover, such value allows to have uniform data distribution among processes for
A Parallel Implementation of a Potential Reduction Algorithm
847
Table 1. CPU times and number of iterations of PG and PR algorithms n 504 756 1008 1512 2016 2520
10% Itrs 229 210 265 321 278 326
Time 10.2 23 75 200 304 548
50% Itrs 195 216 246 257 233 344
PG
Time 7.6 25 60 147 225 595
90%
Itrs 197 219 248 256 243 309
Time 7.9 27 60 153 254 509
Av. PG Itrs 207 215 253 278 248 326
Time 8.6 25 65 167 261 551
PR Itrs 15 16 16 16 16 17
PR/PG
Time 3.9 12 25 74 170 328
0.45 0.48 0.39 0.45 0.65 0.59
the considered problem sizes. The experimental results have to be considered satisfactory and confirm that PR algorithm has been parallelized successfully, obtaining good parallel performances even with small/medium size dense problems. We observe that efficiency values greater than 0.5 were obtained for 6 and 9 processors with problems of size greater than 2000, and for 12 processors when n ≈ 2500. Finally, we point out that, as expected, the most contribution to the total amount of work is given by the solution of the linear systems, which takes about the 90% of the overall time for n > 2000, as the values of tp show. We conclude that the parallel performance of PR algorithm is heavly influenced by the performance of the ScaLAPACK routine used to solve the linear systems. Table 2. CPU times and efficiency of the parallel PR algorithm n
T1 (t1 )
T6 (t6 )
1008 25 (21) 9.4 (7.4) 1512 74 (64) 22 (19) 2016 170 (153) 42 (38) 2520 328 (300) 78 (71) 3024 114 (108) 3528 184 (178)
5
T9 (t9 ) 9.6 18 33 57 83 134
(5.9) (14) (28) (51) (78) (122)
T12 (t1 2) E6 E9 E12 E6/12 9.1 18 31 54 77 118
(6.6) (15) (29) (49) (73) (113)
0.44 0.56 0.67 0.70
0.29 0.46 0.57 0.64
0.23 0.34 0.46 0.51
0.52 0.61 0.68 0.72 0.74 0.78
Concluding Remarks
We presented the results concerning the development and the implementation of a parallel PR algorithm for the BCQP problem based on the parallelization of the computational linear algebra kernels arising in the algorithm. A comparison with a projected gradient method showed that PR algorithm is competitive with active set strategies. Moreover, our approach to the parallelization of the
848
Marco D’Apuzzo et al.
algorithm seems to be very effective. In conclusion, we believe that interior point algorithms represent a very appealing approach for the development of efficient parallel software for the more general nonlinear optimization problem. Acknowledgments The authors would like to thank the anonymous referees for their helpful suggestions.
References 1. E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarlig, A. McKenney, S. Ostrouchov and D. Sorensen - LAPACK User’s Guide, Second Edition - SIAM, Philadelphia, PA (1995). 2. J.L. Barlow and G. Toraldo - The effect of diagonal scaling on projected gradient methods for bound constrained quadratic programming problems - Opt. Methods and Soft., vol. 5 n.3, pp. 235-245 (1995). 3. L. S. Blackford, J. Choi, A. Cleary, E. D’Azevedo, J. Demmel, I. Dhillon, J. Dongarra, S. Hammarling, G. Henry, A. Petitet, K. Stanley, D. Walker, and R. C. Whaley - ScaLAPACK User’s Guide - SIAM, Philadelphia, PA (1997). 4. J. Choi, J. Dongarra, S. Ostrouchov, A. Petitet, D. Walker, and R.C. Whaley A Proposal for a Set of Parallel Basic Linear Algebra Subprograms - LAPACK Working Note #100, University of Tennessee (1995). 5. M. D’Apuzzo, V. De Simone, M. Marino, and G. Toraldo - Parallel Computational Issues for Box-Constrained Quadratic Programming - Ricerca Operativa, vol. 27 n.81/82, pp. 125-141 (1997). 6. M. D’Apuzzo, V. De Simone, M. Marino, and G. Toraldo - Modifying the Cholesky Factorization on MIMD Distributed Memory Machines - in “High Performance Algorithms and Software in Nonlinear Optimization”, R. De Leone et al. (eds.), Kluwer Academic Publishers, pp. 125-141 (1998). 7. J. Dongarra and R.C. Whaley - A User’s Guide to the BLACS v1.0 - Technical Report UT CS-95-281, LAPACK Working Note #94, University of Tennessee (1995). 8. C.G. Han, P.M. Pardalos, and Y. Ye - Computational aspects of an interior point algorithm for quadratic problems with box constraints - in “Large-Scale Numerical Optimization”, T. Coleman and Y. Li eds., SIAM, PA, pp. 92-112 (1990). 9. N. Karmarkar - A New Polynomial Time Algorithm for Linear Programming Combinatorica, vol. 4, pp. 373-395 (1984). √ 10. M. Kojima, N. Megiddo, and A. Yoshise - An O( nL) Iteration Potential Reduction Algorithm for Linear Complementarity Problems - Mathematical Programming, vol. 50, pp. 331-342 (1991). 11. J.J. Mor´e and G. Toraldo - Algorithms for bound constrained quadratic programming problems - Numerische Mathematik, n. 55, pp. 377-400 (1989). 12. M.J. Todd and Y. Ye - A Centered Projective Algorithm for Linear Programming - Mathematics of Oper. Res., vol. 15, pp. 508-529 (1990). 13. A. van der Sluis - Condition numbers and equilibration of matrices - Numerische Mathematik, vol. 14, pp.14-23 (1969). 14. Y. Ye - An O(n3 L) potential reduction algorithm for linear programming - Mathematical Programming, vol. 50, pp. 239-258 (1991).
Topic 12 European Projects Roland Wism¨ uller and Renato Campo Topic Chairmen
Between 1994 and 1998 projects in the area of High Performance Computing and Networking (HPCN) were funded by the European Commission in the framework of the ESPRIT Programme. The three papers in this session originate from three different ESPRIT projects. Since its original conception in the early 1980s the European R&D Programme in Information Technologies (ESPRIT) evolved from the initial technology-push supply-side approach towards IT applications and technology take-up with an increasing involvement of the users. Currently as part of the V Framework Programme the IST Programme continues its support of research, development and demonstration of Information Society Technologies. The first paper presents work carried out in the NEPHEW project, which promotes cluster computing as a cost effective platform for parallel computing. In particular, the project investigates clusters consisting of PCs with Windows NT and an SCI interconnect. Three computationally intensive applications are implemented on this platform: Restoration of historic film material, low-cost flight simulation for pilot training, and reconstruction of medical PET (positron emission tomography) images. In order to reduce the effort of implementing and maintaining parallel applications, the PeakWare programming environment has been ported to the target platform. The paper by W. Karl et al. reports on first experiences with this environment gained with the PET application. The experiences are promising, both with respect to the ease-of-use of the programming environment and the communication speed of the PC cluster. The next two papers both are devoted to the development of high performance simulators, which reflects the focus of the ESPRIT programme on applications in the area of simulation. Partially by chance and partially because it is an important topic for our society, they are both in the area of traffic control. The SEEDS project especially addresses the problem of ground traffic control at airports. It developed a simulation environment for the analysis and evaluation of distributed traffic control systems. The distributed environment consists of components for, e.g., scenario generation, visualization, actor modeling and automatic decision support systems, which are connected via a CORBA middleware. The paper by T. Hruz et al. presents one special component of this environment: The Airport Management Database System. It presents the requirements and the design of this component and also some interesting experiences with the industrial use of Java, CORBA and Open Software tools. The HIPERTRANS project puts a focus on the modeling and simulation of urban road networks. The traffic simulator is intended to be used for the testing and the assessment of traffic control systems, for the training of operators, and A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 849–850, 2000. c Springer-Verlag Berlin Heidelberg 2000
850
Roland Wism¨ uller and Renato Campo
for forecasting, i.e. for predicting the effects of control decisions. Obviously, the latter use requires that the traffic simulation can be performed faster than the real traffic evolves. Thus, high performance simulation techniques are needed. The paper authored by S. E. Ijaha et al. presents an overview on the simulation system, which uses parallel execution based on model partitioning to achieve the performance necessary for forecasting.
NEPHEW: Applying a Toolset for the Efficient Deployment of a Medical Image Application on SCI–Based Clusters Wolfgang Karl1 , Martin Schulz1 , Martin V¨ olk2 , and Sibylle Ziegler2 1
Lehrstuhl f¨ ur Rechnertechnik und Rechnerorganisation, LRR–TUM Institut f¨ ur Informatik, Technische Universit¨ at M¨ unchen, Germany {karlw, schulzm, voelk}@in.tum.de 2 Nuklearmedizinische Klinik und Poliklinik, MRI-NM, Klinikum Rechts der Isar, Technische Universit¨ at M¨ unchen, Germany [email protected]
Abstract. With the rise of cluster architectures, high–performance parallel computing is available to more users than ever before. The programming of such systems, however, has not yet improved beyond the cumbersome programming style used in message passing libraries like PVM and MPI. In order to open clusters to a broader audience, higher level programming environments have to be designed with the goal of giving the end user an easier access to parallel computing. Such an environment is currently being developed within the NEPHEW project allowing the graphical specification of global dependencies. This work presents the NEPHEW approach and discusses its applicability using an example application from the area of nuclear medical imaging, the reconstruction of PET (Positron Emission Tomography) images.
1
Motivation
With the rise of commodity clusters interconnected with high-speed system area networks (SANs), such as SCI [6][3] or GigaNet [8], high performance parallel computing has become available to more users than ever before. The programming of such systems, however, is still mostly based on pure message passing in the form of libraries like PVM [2] and MPI [7]. This is generally perceived as inappropriate for most end users prohibiting cluster architectures to be fully exploited. The NEPHEW1 (NEtwork of PCs HEterogeneous Windows-NT Engineering Toolset) project attempts to tackle this problem by providing a high level programming environment for the implementation of parallel applications on top of commodity Windows NT based clusters. To ease the implementation process, lower the learning curve, and therefore giving also non computer scientist better 1
The NEPHEW project is funded by the European Commission in the Fourth Framework Programme at the ESPRIT Project EP29907 / NEPHEW. More information is available under http://wwwbode.in.tum.de/Par/arch/smile/nephew/.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 851–860, 2000. c Springer-Verlag Berlin Heidelberg 2000
852
Wolfgang Karl et al.
access to cluster architectures, the deployed tool from Matra Systems & Information, called PeakWare [10], offers a graphical development environment that allows a simple and intuitive abstraction of communication paths within the application. Only the actual application functionality has to be hand coded in conventional sequential style using C. In addition, the tool environment provides mechanisms allowing for arbitrary mappings between functions and nodes they are supposed to be executed on. PeakWare then automatically combines these individual parts into the final application. Any functionality needed for global operations and communication are generated by PeakWare from the graphical description and combined with the sequential routines. The generated final binary can be executed on a Windows-NT based cluster according to the specified node mappings. An additional advantage of this approach lies in the fact that the PeakWare toolset is by default independent of the underlying communication network and is generally intended to be used with standard Fast Ethernet. For applications with higher communication demands, however, other interconnection technologies have to be deployed. Within the NEPHEW project, the Scalable Coherent Interface (SCI) [3] is used for this purpose. SCI is a state-of-the-art IEEE standardized [6] high performance SAN. It provides a end-to-end bandwidth of up 118 MB/s and process-to-process latencies as low as 2 µs. This high performance is offered to the user through PeakWare in two different ways: using a transparent implementation of fast sockets and through a special user-level protocol. In both cases, the implementation is done automatically by PeakWare based on the graphical description of the communication behavior of the application. At the end, the NEPHEW environment will generate an optimized application fully exploiting the capabilities of a SAN interconnected commodity cluster with minimal efforts on the side of the end user. This will open the cluster architecture to a whole range of new users and applications accelerating the acceptance of clusters also in the none computer science community. The remainder of this paper is organized as follows: Section 2 introduces the NEPHEW project and its partners. Section 3 then describes the project’s core, the PeakWare tool set. In Section 4, one of the three NEPHEW applications is introduced in detail, followed by the implementation strategy using PeakWare in Section 5, and first experiences on top of the Windows NT cluster in Section 6. The paper is then rounded up in Section 7 with a brief outlook on future work and some concluding remarks.
2
Background for NEPHEW
NEPHEW is a EU funded project in the 4th ESPRIT framework programme. It brings five partners from industry, research, and academia together that have the strong competence not only to successfully implement a comprehensive and efficient tool environment for SAN connected clusters, but also its thorough evaluation. The consortium, whose structure is depicted in Figure 1, consists of Dolphin ICS, the leading manufacturer of SCI technology, Matra Systems &
NEPHEW
853
Information, the provider and developer of PeakWare, and three end users with high–performance cutting–edge applications, namely ELCO Sistemas, Joanneum Research, and the Technische Universit¨at M¨ unchen.
Flight simulation by ELCO Sistemas
Film restauration by JRS
PET reconstruction by TUM
PeakWare toolset for Windows NT clusters by Matra Systems & Information
SCI infrastructure by Dolphin ICS
Fig. 1. The structure of the NEPHEW consortium.
The core of the NEPHEW project, the actual toolset is provided by Matra Systems & Information. It based on an earlier version for embedded real–time multiprocessor systems called PeakWare [10]. Within the NEPHEW project, Matra is porting this toolset to Windows NT and tailoring it to the specific requirements in general purpose cluster environments. This includes easing the user interface, removing specifics for real-time environments, and adding additional support for adding and removing cluster nodes. In addition, the back-end of PeakWare is adapted to the new hardware environment by adding support for Win-Sockets and high-performance communication via SCI. The latter one is achieved in a close cooperation with Dolphin ICS. They bring their extensive low-level expertise about SCI hardware and low-level software into the consortium guaranteeing a highest possible performance of the final solution. As described above, two separate ways are envisioned to achieve this goal: through a high-performance socket implementation on top of Windows 2000 or through a specific low-level implementation taking direct advantage of SCI’s hardware distributed shared memory (DSM). While the first approach provides a maximum of portability and reliability, the second approach delivers maximal performance. The right tradeoff between these two approaches can be done by the application programmer through easy graphical annotations within PeakWare to tailor the generated code to the application’s specific needs and requirements. For the evaluation of the PeakWare/NEPHEW toolset the consortium includes several end user application providers from different areas and backgrounds. ELCO Sistemas, a Spanish producers of flight simulation systems, will use PeakWare for the efficient implementation of low-cost PC based helicopter simulators. These systems are intended to close the gap between expensive spe-
854
Wolfgang Karl et al.
cialized embedded simulators and game-like single systems programs and hence also allow small companies to gain access to serious and realistic simulation environments. Joanneum Research, an Austrian research institute, is specialized on the restauration of historic film material that has deteriorated after long storage. This is a very compute intensive application and requires efficient parallelization. The third application, which is provided by the Technische Universit¨ at M¨ unchen, more specifically by a cooperation between the “Nuklearmedizinische Klinik und Polyklinik” 2 at the department of medicine and the “Lehrstuhl f¨ ur Rechnertechnik und Rechnerorganisation” 3 at the department of Computer Science, comes from the area of Positron Emission Tomography and implements a necessary postprocessing step, the reconstruction of PET images from raw scanner data with a high image quality. This application will be used in this work as the example on how the NEPHEW toolset can be applied and is therefore described in more detail below in Sections 4 and 5.
3
PeakWare: Toolset for Efficient Cluster Computing
PeakWare [10] is a software product by Matra Systems & Information that was originally targeted to program real-time multiprocessor systems like the ones from Mercury Inc. [9]. In the NEPHEW project the system is ported to Windows NT and adapted to the specific needs in cluster environments. Application development in PeakWare is generally done in five steps. First the data flow behavior has to be extracted from the overall application design and the main communication paths have to be determined. This analysis forms the basis for the decomposition of the application into modules, one of the central concepts of PeakWare. Modules are logical units of functionality that form the basis for global communication. The separation into these modules has to be done in a way that all communication between the modules can cleanly be specified. The information gained through this analysis is then used to graphically describe the communication behavior in the so called software graph. An example with two modules and bidirectional communication is shown in Figure 2. PeakWare also offers the ability to scale individual modules, i.e. to replicate and distribute them among the cluster. This concept offers an easy way to introduce data parallelism into an application and allows the easy scaling on potentially arbitrary numbers of nodes. Each module consists of several functions with the global communication channels as input and output arguments. The implementation of the functions themselves is done in external source files using conventional sequential programming in C. The source code can be augmented by macros defined by PeakWare in order to establish a connection between the functions and the PeakWare system. The next step is the definition of the hardware that is supposed to be used for the application. This is again done with a graphical description, the hardware graph. An example of such a graph can be seen in Figure 3. It shows a small 2 3
Clinic for nuclear medical imaging Chair for computer technology and organization
NEPHEW
855
Fig. 2. Example of a PeakWare software graph (simple ping-pong)
Fig. 3. Example of a PeakWare hardware graph (2 node cluster)
cluster of two compute and one host node connected by Fast Ethernet and SCI (between the two compute nodes). For any communication between two nodes connected by more than one network, PeakWare then selects the appropriate one based on the user’s choice of protocols for the particular communication path. These parameters, however, are defined in the software graph leaving the specification of the hardware graph fully independent of the software graph. This enables the option to change the hardware in terms of node description and/or number of nodes without requiring changes in the software graph. Once the hard- and software graph have been completed, PeakWare gives the user the option to specify a mapping between modules (from the software graph) and nodes (as specified in the hardware graph). This mapping defines which module is executed on which node and hence represents the connection between the two graphs. It also allows the easy retargeting of applications to new hardware configurations as well as a simple mechanisms for static load balancing. The last step, after the mapping has been done and all routines have been implemented in external source files, is the code generation. In this process PeakWare uses the information from the graphical description of the software and hardware graphs and generates C source code that includes all communication and data distribution primitives. This code can then be compiled with conventional compilers resulting in a final executable and a shell script to start the application on all specified nodes; no further user intervention is necessary resulting a very easy-to-use environment.
856
4
Wolfgang Karl et al.
Nuclear Medical Imaging Using PET
This easy-to-use programming environment is evaluated within the NEPHEW project using three large–scale real–world applications from a wide range of domains. One of them comes from the area of nuclear medical imaging and implements the reconstruction of Positron Emission Tomography (PET) images, a necessary post-processing step in the daily clinical routine when working with nuclear imaging. Positron Emission Tomography is a nuclear medicine technique which allows to measure quantitative activity distributions in vivo. It is based on the tracer principle: A biological substance, for instance sugar or a receptor ligand, is labeled with a positron emitter and a small amount is injected intravenously. Thus, it is possible to measure functional parameters, such as glucose metabolism, blood flow or receptor density. During radioactive decay, a positron is emitted, which annihilates with an electron. This process results in two collinear high energy gamma rays. The simultaneous detection of these gamma rays defines lines-of-response along which the decay occurred. Typically, a positron tomograph consists of several (e.g. 32) detector rings covering an axial volume of 10 to 16 cm. The individual detectors are very small, since their size defines the spatial resolution. The raw data are the line integrals of the activity distribution along the lines-of- response. They are stored in matrices (sinograms) according to their angle and distance from the tomograph’s center. Therefore, each detector plane corresponds to one sinogram. Image reconstruction algorithms are designed to retrieve the original activity distribution from the measured line integrals. From each sinogram, a transverse image is reconstructed, with the group of all images representing the data volume. Ignoring the measurement of noise leads to the classical filtered backprojection (FBP) algorithm [4]. Reconstruction with FBP is done in two steps: Each projection is convolved with a shift invariant kernel to emphasize small structures but reduce frequencies above a certain limit. Typically, a Hamming filter is used for PET reconstruction. Then the filtered projection value is redistributed uniformly along the straight line. This approach has several disadvantages: Due to the filtering step it yields negative values, particular if the data is noisy, although intensity is known to be non-negative. Also the method causes streak artifacts and high frequency noise is accentuated during the filtering step. Iterative methods were introduced to overcome the disadvantages of FBP. They are based on the discrete nature of data and try to improve image quality step by step after starting with an estimate. It is possible to incorporate physical phenomena such as scatter or attenuation directly into the models. It is generally acknowledged that iterative methods yield better images in low count situations. On the down side, however, these iterative methods are very computational intensive. For a long time, this was the major drawback for clinical use of these methods, although they yield improved image quality. Speedup of iterative reconstruction is gained by the use of algorithms which use only subsets of data within one iteration (Ordered Subset Expectation Maximization, OSEM) [5]. Still, due to the iterative nature a substantial amount of time is required for the
NEPHEW
857
reconstruction of one image plane. Thus, parallel processing of iterative reconstruction would help improving medical imaging.
5
PET Image Reconstruction Using NEPHEW
The usual way to implement PET image reconstruction on top of loosely coupled architectures like clusters is based on the master-slave-principle. With image reconstruction it is quite easy to distribute the computational load on the different machines, as an image consists of a fixed amount of planes (typically 47 or 63 depending on the scanner type). For each pixel of these planes the same arithmetic operations are executed. The basic approach is therefore to distribute the raw scanner data from the master plane by plane to the slaves running on the different nodes. Each slave process reconstructs one plane of the image and sends the result back. The master is then responsible for finalizing and storing the reconstructed image. Using the modular design of PeakWare a slightly different approach has to be done: Three modules are used for reconstructing the image. A sender-module distributes the planes to a scaled consumer module, which corresponds to the slave in the approach described above. The planes are distributed in a round-robin fashion and one plane at a time. After a consumer instance has reconstructed a plane the receiver module is informed, which plane was reconstructed. In addition the sender is informed, that a next plane can be sent. The sender distributes the planes of the image until the reconstruction of all image planes has been acknowledged by the receiver. If no planes are left, planes might be resubmitted to idle consumers resulting an easy, yet efficient fault tolerance scheme with respect to the consumer modules. The software graph based on the design described above, is depicted in Figure 4. It shows the three modules, with the consumer module scaled to $(SCALE) instances. In addition, the main data paths form the sender through the consumer to the receiver is visible in the middle augmented by the two acknowledgment paths leading back to the sender. The software-graph with the three modules can be mapped on to arbitrary hardware graphs. This enables the easy distribution of consumer module instances across arbitrarily scaled clusters. It is also possible to change the number of nodes without changing the software-graph by simply remapping it to a different hardware-graph. This independence of hardware and software description drastically eases the port of application between different hardware configuration and allows for an easy-to-handle scalability for the PET reconstruction application.
6
Preliminary Experiences on Windows NT Clusters
The project is currently in an early stage of development. Therefore, only limited results can be presented here. They, however, give a good first impression of
858
Wolfgang Karl et al.
Fig. 4. Software graph for the PET reconstruction application.
the direction the project and allow a first insight into the capabilities of the NEPHEW approach. In the first phase of the project, a prototype of the communication framework of the PET reconstruction application has been designed and implemented using PeakWare. Its software graph is shown above in Figure 4. The development of this prototype clearly showed the potential of the NEPHEW approach. The actual routines that had to be hand coded are extremely small only limited to the core functionality. Any required communication handling, which in standard message passing environments contributes to a majority of the code size, is created automatically by PeakWare, leaving the programmer with convenient and easily understandable code segments. In addition, the orthogonal design of the hardware and software graphs, in addition with the extensive mapping capabilities of PeakWare, allows an easy application prototyping on smaller clusters or even single node machines. Once completed, the final application can easily moved to a target cluster of arbitrary size by remapping the modules. This significantly reduces the implementation time and lowers the usage threshold for non computer science users. First concrete performance results have been achieved in the low-level communication layers provided by Dolphin. The performance data is summarized in Table 1 [1], which compares three different approaches. All numbers are gathered on top of Windows 2000 using a high–end PC cluster with a 64 bit PCI bus and the 64 bit PCI adapter D320 by Dolphin. The first two options provide a socket-like interface to the user, one time through a standard network driver within the Windows kernel (NDIS) and one time using a fast socket library with user level communication transparently embedded into the overall system through a so called fast socket switch4 . This convenient API for the user, however, comes at the price of reduced performance 4
This switch allowing the transparently switching between standard Ethernet and fast communication networks is currently developed at Microsoft and will be included in the Windows 2000 Data Center.
NEPHEW
859
caused by the overhead of the additional software layers. Especially latency is drastically increased. In addition, the NDIS packages pays an extra price for being a kernel component further reducing the performance. It is worth noting, however, that these numbers are only first experiments and better performance is expected for the final product. In any case, however, when optimal performance and lowest possible latency is required, SCI has to be deployed using its native HW-DSM interface by performing communication directly on shared memory segments. In this case a bandwidth of up to 118 MB/s and a latency as low as 2 µs can be achieved. Bandwith Latency SCI over NDIS 62 MB/s 105 µs SCI over Fast sockets 90 MB/s 75 µs SCI raw 118 MB/s 2 µs
Table 1. First experimental results of SCI as the interconnection network.
In summary, SCI provides state-of-the-art performance to the user. Depending on the needs of the application the user has the choice of either relying on the existing convenient messaging standards, like Winsockets, or defining their own protocols and getting access to optimal performance. PeakWare will be able to use any of these communication mechanisms providing the user with the optimal choice for their particular application. Experiences with earlier versions of PeakWare on Mercury system have shown that the system due to its lean and efficient implementation only adds a very small latency penalty to the raw communication times, delivering close to optimal performance to the end users.
7
Conclusions and Future Work
With the availability of clusters to more users than ever before, it is necessary to provide easy-to-use, high-level programming environments. Only this will give also give unexperienced end users and non computer scientists access to lowcost parallel computing. Within the NEPHEW project, such an environment, targeted towards Windows NT based commodity clusters connected with SCI, a state-of-the-art high bandwidth, low latency SAN, is under development. This approach is being evaluated within the project using three computational intensive application. One of these, the reconstruction of PET images, is used in this work to demonstrate the applicability of developed concepts. First experiments with the PET application have proven the ease-of-use of PeakWare. The application development phase is drastically shortened as all the overhead associated with communication management is automatically being taken care of by the toolset. In addition, the PeakWare software graph allows for an easily understandable presentation of the application’s behavior. This helps
860
Wolfgang Karl et al.
to significantly lower the learning curve for parallel programming and reduces potential errors. Besides these ergonomic experiences, this work has also shown the performance potential of the underlying architecture. SCI networks deliver highperformance for direct process–to–process communication. By embedding SCI into the PeakWare toolset, the SCI performance potential is made available to the end user in the easiest possible way. The next steps in the NEPHEW project include, besides further tests with cluster hard- and software, the completion of the three NEPHEW applications. In the end, all three application partners will have a running and robust parallel solution of their application on top of PeakWare. The PET reconstruction code will then be used in daily clinical routine as a standard instrument for the medical personal to the benefit of the patients. In summary, the NEPHEW framework will allow users the easy implementation of parallel applications for PC based clusters and the efficient utilization of high-speed networks like SCI. It will form the basis for many application ports from a large variety of application domains and therefore open the cluster architecture to wide range of new users and applications. This will further increase the attractivity of the cluster platform and accelerate their acceptance in daily production scenarios.
References [1] T. Amundsen. SCI performance under Windows 2000. personal communication with Dolphin ICS, May 2000. [2] A. Bengelin, J. Dongarra, A. Geist, R. Manchek, and V. Sunderam. A User’ s Guide to PVM Parallel Virtual Machine. Oak Ridge National Laboratory, Oak Ridge, TN 37831-8083, July 1991. [3] H. Hellwagner and A. Reinefeld, editors. SCI: Scalable Coherent Interface. Architecture and Software for High-Performance Compute Clusters, volume 1734 of LNCS State-of-the-Art Survey. Springer Verlag, October 1999. ISBN 3-540-666966. [4] G.T. Herman. Image Reconstruction from Projections. Springer-Verlag, Berlin Heidelberg New York, 1979. [5] H. Hudson and R. Larkin. Accelerated image reconstruction using ordered subsets of projection data. IEEE Transactions on Medical Imaging, 13:601–609, 1994. [6] IEEE Computer Society. IEEE Std 1596–1992: IEEE Standard for Scalable Coherent Interface. The Institute of Electrical and Electronics Engineers, Inc., 345 East 47th Street, New York, NY 10017, USA, August 1993. [7] Message Passing Interface Forum (MPIF). MPI: A Message-Passing Interface Standard. Technical Report, University of Tennessee, Knoxville, June 1995. http://www.mpi-forum.org. [8] WWW:. Welcome to Giganet . http://www.giganet.com/, May 1999. [9] WWW:. Mercury Computer Systems (NASDAQ:MRCY) Home Page. http://www.mc.com/, January 2000. [10] WWW:. Peakware — Matra Systems & Information. http://www.matramsi.com/ang/savoir infor peakware d.htm, January 2000.
SEEDS: Airport Management Database System Tom´ aˇs Hr´ uz1,2 , Martin Beˇcka3 , and Antonello Pasquarelli4 1
4
SolidNet, Ltd., Slovakia, www.sdxnet.com, [email protected] 2 Department of Adaptive Systems, UTIA, Academy of Sciences of the Czech Republic, Prague‡ 3 Mathematical Institute, Slovak Academy of Sciences, P.O.Box 56, 840 00 Bratislava, Slovakia, [email protected] Alenia Marconi Systems, Via Tiburtina km 12.4, I-00131 Rome, [email protected]
Abstract. The article describes an airport database management system, which is a part of large simulation environment developed under the ESPRIT project SEEDS. Airport management database is a distributed computing system written entirely in Java and connected to the rest of the SEEDS system through CORBA interface. The majority of the SEEDS modules is written in C++ and communicates through CORBA. The airport management database consists of an application server, three different clients and a CORBA connection module.
1
Introduction
This article describes an Airport Management Database (AMDB) system which is a part of larger simulation environment SEEDS [1,6]. SEEDS is a distributed simulation environment composed of powerful workstations connected in local network, suitable to evaluate advanced surface movement guidance and control systems, to validate new international standards and to train operators. The aim of AMDB software module is to describe various external aspects of the core airport simulation model like meteorological situation and its changes, flight data list (FDL) and its changes, Initial Climbing Procedures (ICP), Instrument Approach Charts (IAC), Standard Instrumental Departure (SID) and Standart Approach Route (STAR) description and visual information. To emphasize the external aspect of AMDB with respect to the core simulator we sometimes use the term external world model instead of AMDB. The AMDB module is designed to contain a relational database subsystem to handle large data sets and a three level architecture (SQL server [7], application server and clients) to achieve high level of flexibility. Another aspect of AMDB module design is a wide area network operation emphasis, which is leading to the architecture centered around the Java system. ‡
Grant GACR 102/1999/1564
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 861–868, 2000. c Springer-Verlag Berlin Heidelberg 2000
862
Tom´ aˇs Hr´ uz, Martin Beˇcka, and Antonello Pasquarelli
The core of the SEEDS simulator is written in C++ and uses the CORBA standard to communicate between different system modules. The AMDB module is entirely written in Java. This situation provides an excellent occasion to test the cooperation between Java and CORBA based C++ subsystems. In particular, the application server (APS) is written in Java to see whether the speed of recent Java virtual machine (JVM) implementations is able to cope with the tasks arising in such complex applications. As far as SEEDS together with AMDB module contains various hardware and software platforms we have decided to use only open software tools for which we had the source code. This approach has two reasons: 1. In very complex and multi-platform systems we consider the absolute control over the development process provided by source code as inevitable for any high quality software. 2. Open software systems usually contain a less powerful visual development tools and we were interested in knowing whether these high level visual features are really necessary for development of very complex systems. The article is organized as follows: in Sec. 2 we describe architecture, design specification and basic features of the AMDB system. Then we conclude with some notes and experiences drawn from this work.
2
Airport Management Database System
The information stored in the airport management database can be structured to groups according to the importance and complexity. In the SEEDS airport model architecture [1] the following navigation data have been considered for inclusion to the external world model. (1) Meteo information, (2) FDL and (3) SID, STAR, ICP and IAC charts. These information groups are implemented in the AMDB module through relational database model, APS and clients. The relational database stores the information about the navigation data in the structured way and APS is responsible for all non-relational model aspects. There are three main clients in the system. 1. Meteo client responsible for meteo information rendering and changes. 2. Flight data list client responsible for the FDL processing. 3. Image viewer client responsible for SID, STAR, ICP, IAC charts previewing and processing. The synchronization, authentication and all other aspects of AMDB allow arbitrary number of clients (limited only by system resources) of each type running at the same time. 2.1
Architecture
Today’s computing environment enforces definitively a server/client architecture. The main reasons are flexibility, scalability and reusability. Moreover, the AMDB module architecture is Java (network computing) oriented and has a
SEEDS: Airport Management Database System
863
SQL relational database in the background to meet the design goals. The client Java classes communicate with the SQL database engine through JDBC layer and driver [8]. However, an architecture conforming to the above mentioned points can be designed in various ways. Let us consider first the architecture in Fig. 1 but without the third quadrant containing AS. With respect to the relation between SQL database and clients this can be called two level architecture because there are two levels: 1.clients and 2.SQL servers. The advantage of such architecture is simplicity, a disadvantage is limited modeling strength and flexibility. With modeling strength and flexibility we mean the following concept. Any software system can be considered as a model of some process. If we consider a two level architecture as above, we can efficiently construct a model for a process which can be well represented in terms of relational database. However, if the modeled process does not fit well in this class we have to add software modules which represent its non-relational structures. If a two level architecture is used these modules must be split between the SQL server (as embedded procedures) and the clients. Such splitting can be very inefficient from the software design point of view as well as from the final system efficiency point of view. As a solution to the above problem a concept of three level architecture can be used. The three level architecture has: 1. clients, 2. application servers, 3. SQL servers. Following the ideas above, all non-relational aspects of the model are located in the APS. It is also responsible for managing and forwarding the database queries of clients to the SQL server. In the SEEDS project an absolute necessity of APS comes from the fact that any change of the database state must be transmitted to the rest of the system in form of events, so that all modules can change their behavior according to the new state of the external world. The final three level architecture used for AMDB is shown in Fig. 1 where APS is designed in Java and is called from the clients through Java Remote Method Invocation (RMI). Another advantage of this architecture is that design of such system contains also a task of application protocol development. Moreover, in the case of very heavy server computation related to some non-relational modeling aspects, the server class can call C or C++ libraries through JNI (Java Native Interface). The airport management database architecture and its relation to the processing of events in SEEDS is illustrated in Fig. 2 on an example of the meteo data processing subsystem. AMDB calls a meteo event server written in C/C++ with a CORBA interface, which is a part of core SEEDS system. The SEEDS modules register to the meteo event server for a particular class of events. The major events with respect to the external world model are changes in the model database state. For example an operator wants to change the visibility at some airport object. He starts a master mode of client module, which allows him to change the database state. The request is forwarded through RMI to AS, which generates a SQL operation to the SQL server and a notification event to the meteo event
864
Tom´ aˇs Hr´ uz, Martin Beˇcka, and Antonello Pasquarelli 00 11 00 11 1 11 00
WEB BROWSER WEB SERVER
VIRTUAL MACHINE
X11 SERVER
00 11 00 11 2 11 00
CLIENT JAVA CLASS
WEB PAGE
DOWNLOAD
CLIENT JAVA CLASS
JDBC DRIVER
RMI C INTERFACE
RMI REGISTRY
RELATIONAL SQL
DATABASE SERVER VIRTUAL MACHINE HOST VIRTUAL MACHINE SQL
00 11 00 11 4 11 00
JDBC DRIVER
APPLICATION SERVER
DOWNLOAD
APPLICATION SERVER CLASS
00 11 00 11 3 11 00
Fig. 1. The Java client/server 3 level architecture for Airport Management Database server. APS notifies also all registered external world clients about the change so that all other operators see the change on the external world client screen. Because the event server has been notified all SEEDS modules registered for this event class will obtain a notification about the event. Then they can ask through a CORBA interface of the application server about the new values of AMDB and they can change the behavior according the new data. For example the scenario generation module can change the picture on appropriate operator and pilot screens etc. To summarize the architecture, the AMDB system consists of the following subsystems: (a) SQL server. The SQL server used is PostgreSQL [7], which is running on most UNIX platforms. (b) SQL relational database. The AMDB use a database maintained by SQL server. The relational database consists in a set of tables, which provide a relational model of AMDB subsystems like airports, meteo data, navigation procedures etc. (c) Web server and web pages. The client applets are stored and transmitted through the Apache web server. (d) The clients. The code of clients is stored in signed archives on the web server. Clients are downloaded from the web server to the browsers where they are run on browser virtual machine. (e) The application server. APS is running as a Java application under JVM. In the prototype configuration the application server, the web server and the SQL server are running on Alpha station with Digital 64bit UNIX.
SEEDS: Airport Management Database System
METEO EVENT SERVER
CORBA
865
APPLICATIONS
CORBA
CORBA
SEEDS WORLD
WAN AMDB SERVER SIDE VIRTUAL MACHINE
CLIENT 1
APPLICATION SERVER
CLIENT 2 EVENTS
SQL SERVER
RMI CLIENT3
JDBC
Fig. 2. External World Model Architecture and Communication. The communication pattern between application server, SQL server and clients in the AMDB module is shown as the emphasized triangle structure
2.2
Data Transmission Rules
The SEEDS system is highly heterogeneous and contains more platforms (e.g. Microsoft NT on Intel, UNIX on Silicon Graphics, UNIX on Alpha, Java etc.) therefore it is necessary to establish communication rules which allow robust and efficient communication between the modules and platforms. For the AMDB module design we have identified the following rules: (1) All aspects of the system, which can be modeled by relational database are modeled by the database structure and are centered in SQL server. (2) If a client is interested in data which are stored in the database and do not require non-relational processing they read the data directly from the SQL server as is illustrated in Fig. 2 with an emphasized triangle pattern. (3) The only agent allowed to write the data to the SQL database is the application server. (4) All non-relational modeling and processing is centered in application server. In the case of the AMDB module this means for example notification and data transmission between the AMDB module and the rest of SEEDS. (5) Because of the heterogeneous character of SEEDS it is preferable to convert most of the data which will be transmitted over the network to integer formats. This is an obligatory rule for primary key data. It means that all information entities are coded as integers and only these codes are sent over the network. The clients are responsible for transformation between the integer representation and other forms of representation like strings etc. For this purpose the clients call directly the SQL server.
866
2.3
Tom´ aˇs Hr´ uz, Martin Beˇcka, and Antonello Pasquarelli
Communication with SQL Server
Following our open software strategy we have used the PostgreSQL [7] database system. The resulting experience is very positive. The system is robust, stable and efficient. Moreover, it defines a very rich set of database types. On the other hand, the main disadvantage of this system, when compared with commercial systems like Oracle SQL servers is a certain lack of preprocessing, postprocessing and visual tools. However, the minimal price (only the maintenance cost) and the open character of this system have far outweighted its disadvantages especially in a research and development project as is SEEDS. The clients and APS communicate with the SQL server using JDBC layer defined by Java system [8] which in turn calls JDBC driver which is provided by SQL server manufacturer, in our case by PostgreSQL system. 2.4
Security Model
The AMDB module is designed as a network computing structure therefore the security aspects are very important. The PostgreSQL server is well equipped with security options ranging from a simple password authentication to KERBEROS and data channels encryption. To choose the appropriate option is a matter of configuration depending on the actual AMDB usage. APS is written in Java as a Java application. One of the specification and design problems here is related to the fact that Java version 1.1, which is still not widely accepted by the browser industry does not contain interface to CORBA. This is provided in a new generation of Java, version 1.2. Therefore, we have adopted the following security design rules illustrated in Fig. 3. – The application server is written as a Java application. – The server code is conformant to the Java 1.1 standard. – The server is compiled and running in the Java 1.2 system. The security restrictions are defined with a special security manager written for the AMDB module. – The clients are written as Java applets. – The code of clients is conformant to the Java 1.1 standard. – The clients are compiled and running in the Java 1.1 system. The security restrictions are handled according to the Java 1.1 system. This means that we generate signed archives stored on a web server.
2.5
Application Server and Clients
APS is responsible for all non-relational modeling aspects. After successful reconstruction of the initial state an object for the SQL communication is constructed, which opens a channel to SQL database server used for data processing. This SQL channel is used to read and write the data from the database during the
SEEDS: Airport Management Database System
867
WEB BROWSER
WEB SERVER
VIRTUAL MACHINE
DOWNLOAD SIGNED ARCHIVE
CLIENT JAVA CLASS
CHECK IDENTITY
WEB PAGE
IDENTITY DEFINED ON THE PLATFORM DOWNLOAD
VIRTUAL MACHINE HOST
CLIENT JAVA CLASS
GENERATE THE CERTIFIIED ARCHIVE
VIRTUAL MACHINE APPLICATION SERVER
JDK
Fig. 3. Illustration of the client security model whole life period of APS. All other agents are allowed (even sometimes obliged) to read directly from SQL server but not to write. Clients connect to APS through well defined ”channels” which are defined as two sets of remote procedures. One set for each direction. RMI connection of a client is initialized through a well known remote object. The reference to the remote object is obtained through the RMI naming services. Then a private channel is constructed which contains private remote references to the procedures above. APS maintains a list of channels which is periodically checked for dead clients. The channels for dead clients are deleted so that there is no resource deficit during a long server runs. The channel maintenance subsystem use a watch dog mechanism where in regular intervals each client must call the following time synchronization procedure: Timestamp watchDog(Timestamp clientSimulationTime) through which the client sends its value of the simulation time and obtains back the server value of the simulation time. If they differ in larger value, the client is obliged to synchronize with APS. At the same time, when APS obtains such call it updates the watch dog structure. This mechanism provides a means how to detect dead channels as well as synchronize simulation time between the clients and APS. We have implemented three main AMDB clients, which are connected to the database through JDBC. During the client run it is necessary to load information from the database concerning the user identity tests, simulations, airports, aircraft types, airport objects and its sites, predefined meteo values, airport map images, etc. Some of these data are loaded once and cashed in the application, other ones are dynamically loaded when they are needed. We have introduced a concept of two edit modes of the client, a slave mode and a master mode. In the slave mode a user can only view all information the client offers. In the master mode the user can change the data. The slave mode is
868
Tom´ aˇs Hr´ uz, Martin Beˇcka, and Antonello Pasquarelli
default, for switching to the master mode the user’s identification and password is required. User is notified about the number of masters working on the same configuration, however users can concurently write to the database.
3
Conclusions
Concerning the software engineering point of view there are three main conclusions (experiences) from the SEEDS project. One concerns CORBA and the other two Java and open software systems. CORBA together with Internet proved to be extremely efficient in development and integration of large systems which contain different platforms and geographically distant development teams. The introduction of new platform (Java) into the project has stressed the necessity to stick very closely to CORBA standards, otherwise the reusability of the software modules can be decreased. We used Java extensively in AMDB design to test its reliability and efficiency. Concerning its complexity, Java is surprisingly matured even for industrial applications. We where running the application server on different platforms (Sun, Alpha) with very good results concerning speed and reliability. The results with open software tools are very positive and the possibility to inspect the source code in some situations has far outweighed the disadvantage of not having a strong visual development tools. This do not indicate that they are not needed in current software development industry but it might indicate that their role is little bit overestimated and that the complexity level on which they are really inevitable lies higher as is usually considered. In some situations it was necessary to change or correct the source code of systems which we used and this was far more efficient solution as to circumvent the problems with other means.
References 1. Bottalico, S., de Stefani, F., Ludwig, T., Rackl, G. : SEEDS – Simulation Environment for the Evaluation of Distributed Traffic Control Systems. In: Lengauer, Ch., Griebl, M., Gorlatch (eds.): Euro-Par ’97, Parallel Processing, LNCS 1300, Springer-Verlag, Berlin Heidelberg New York (1997), 1357–1362 2. Bottalico, S. : SEEDS (Simulation Environment for the Evaluation of Distributed Traffic Control Systems). A Simulation Prototype of A-SMGCS. In: Proceedings of the International Symposium on Advanced Surface Movement Guidance and Control System, Stuttgart, Germany, 21-24 June 1999 3. Hr´ uz, T., Beˇcka, M. : Airport Management Database. SEEDS Internal Documentation SEEDS/MISAS-WP4-D4.6A-REV1-SW, 1999 4. Lewis, G., Barber, S., Siegel,E. : Programming with Java IDL. John Wiley & Sons, 1998 5. Flanagan, D. : Java in a Nutshell. O’Reilly, 1997 6. SEEDS official home page, http://www.lti.alenia.it/EP/seeds.html 7. http://www.postgresql.org, 1998–1999 8. http://www.javasoft.com, 1998–1999
HIPERTRANS: High Performance Transport Network Modelling and Simulation Stephen E. Ijaha, Stephen C. Winter, and Nasser Kalantery Centre for Parallel Computing, Cavendish School of Computer Science, University of Westminster, 115 New Cavendish Street, London W1M 8JS, UK {ijahas, wintersc, kalantn}@wmin.ac.uk http://www.wmin.ac.uk Abstract. HIPERTRANS (High Performance Transport network modelling and Simulation) is a fast and visually representative simulator that can predict traffic on a given urban road network. It was designed using object-oriented techniques and by re-engineering established road traffic models. It has the capability of interfacing to a range of UTC (urban traffic control) systems. Its high performance version was implemented by using a novel parallel programming platform called SPIDER; it has the capability of executing faster than real time and runs on distributed processors. The simulator provides a powerful GUI (graphical user interface) for entering road network models, configuring simulation runs, and visualising simulation results. The system provides helpful traffic diagnostic tools enabling local transport authorities, policy makers, researchers, and UTC manufacturers to gainfully exploit its functionality. The HIPERTRANS project was funded by the European Commission under the 4th Framework programme.
1 1.1
Introduction The HIPERTRANS Requirements and Specifications
Traffic congestion problems in urban areas have prompted governments, local authorities and UTC manufacturers to be interested in a modelling and simulation tool that can be used to predict traffic situations. Such a tool is needed to achieve real-time traffic control measures and assist policy makers to make informed decisions [1]. The HIPERTRANS project conceived the idea of a fast, representative, flexible and visually comprehensive traffic simulation tool that can be achieved through the application of low cost but high performance computing environments, advanced simulation technologies, and industrial software production methodologies [2]-[3]. The HIPERTRANS simulation system consists of a new range of facilities for transport consultants, researchers, traffic engineers and UTC centre managers. The tool can enable transportation network operators to assess the performance of their road network quickly under a variety of operational conditions and behavioural patterns. Additionally, it can be used as a tool for hardware commissioning and operator training. Using event driven approach, the HIPERTRANS system is capable of putting vehicles into a network and representing the movement of individual vehicles. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 869-874, 2000. Springer-Verlag Berlin Heidelberg 2000
870
Stephen E. Ijaha, Stephen C. Winter, and Nasser Kalantery
This paper describes the work carried out within HIPERTRANS during April 1997 to June 1999. After the description of the project in this Section, the next Section states the objectives of the work. Section 3 describes the methodology and approach employed by which these objectives were achieved. Section 4 provides the results achieve. The conclusions and recommendations for future research are presented in Section 5.
1.2
HIPERTRANS Partnership and Test Sites
HIPERTRANS, an EC-funded consortium, was led by the University of Westminster (UoW) and had 8 partners spread across 4 different EC countries. The partners participated within two broad groups: Technology Group and User Group. The 3 Technology Group partners came from 2 universities – Facultés Universitaires NotreDame de la Paix à Namur (FUNDP) in Belgium and UoW in the UK; and a research institution - Institut National de Recherche en Informatique et Automatique (INRIA), in France. They were responsible for the definition, design, building, testing, and verification of the simulators that were developed within the project. The 5 User Group partners were formed by: 2 transport consultants - BKD Consultants Ltd (BKD), and W.S. Atkins (WSA), both in the UK, 2 UTC (urban traffic control) system manufacturers - Electronic Trafic S.A. (ETRA), in Spain and Peek Traffic Limited (PTL), UK, and a software house - SIMULOG in France. The User’s main role was to validate and demonstrate the simulators in a real traffic environment using real traffic scenarios. The work on the real-time simulator and predictor made enormous use of two different kinds of proprietary UTC systems kindly made available to the project by two of our partners: SCOOT [4] partly owned by PTL; and STU (Sistema de Trafico Urbano), [5] wholly owned by ETRA. SCOOT is the predominant type of UTC system in the UK and STU in Spain. The GUI developed for visualising the results and inputting road network data models into the simulator was developed by using Simulog’s OGL graphical library. This library is based on the ILOG [6] system and is available on Windows and Unix platforms. Two test sites, Hyde in the UK and Valencia in Spain were employed by the project. The criteria used in selecting these test sites were that they had the UTC systems used by our User partners and whose local authorities were willing to participate in the evaluation of the project. The participation of different UTC manufactures and local authorities in this project is also helping to overcome the challenges of noninteroperability of UTC systems not only in Europe but also throughout the world.
2
Objectives
The key aim of the project was to produce microscopic modelling and simulation tool for the easy and cost-effective development of road traffic control systems and for the effective management of traffic flow on road networks. Specific aims were to: • develop a simulator able to interact with UTC systems at real-time speeds and develop a predictor consistent with users’ requirement for look-ahead; • implement a low-cost high-performance system on a scaleable workstation cluster;
HIPERTRANS: High Performance Transport Network Modelling and Simulation
871
• demonstrate that the use of high-performance simulation can speed up and dramatically improve the operation of urban and inter urban transportation networks; • compare the execution times performance of the simulation algorithms; • model road traffic on a small-to-medium sized network; and study technology and best practice for extending the simulator to handle inter-urban situations. In general, the HIPERTRANS system aimed to have the following basic functionality: • performing microscopic simulation having a user-friendly graphical user interface; • interfacing capability to UTC systems and capability of working in real time; • being able to simulate and predict the state of the road traffic; and • achieving high-performance computing on low cost platforms.
3
Technical Description
The implementation of the system was performed in three different but closely related stages involving the component software modules being developed on several sites and eventually integrated on a single platform. This provided essential co-ordination points within the software development and enabled the User Group to evaluate progress. They also assisted all partners to monitor and steer the development activities in an optimal manner. The strategy used in the work was geared towards applying results from the science base of advanced traffic modelling and control techniques as well as the technology of high-performance computing to achieve the simulators. The overall architecture consists of the simulator; a UTC system; and an interface between the simulator and the UTC system. The simulator is software and hardware configurable to: • enable it to be used in real-time with different UTC types; • provide a stand-alone simulation capability when not interfaced to a UTC; • allow information relevant to a wide range of road networks to be used; and • scale from low cost and moderate performance to higher cost higher performance. The architecture shown in the Figure below is a combination of an object-oriented framework for traffic simulation, user simulation models, and application programming interfaces for interconnection to UTC systems. The background software used existed at three partner sites: PACSIM [7] developed by FUNDP; PROSIT [8] co-developed by INRIA and Simulog, and SPIDER [9] developed by UoW. GRAPHICAL USER INTERFACE USER SIMULATION PACSIM
SPIDER
TRAFFIC SIMULATION FRAMEWORK PROSIT
The HIPERTRANS Architecture
872
4
Stephen E. Ijaha, Stephen C. Winter, and Nasser Kalantery
Results
The project has delivered a micro-simulator as specified, and was used for highprofile public demonstrations and evaluation at Hyde in the UK and Valencia in Spain [10]. The results provide a proof of concept of high-performance simulation as ascertained from the needs of users. The resulting simulator can run on a network of workstations, ensuring that simulation of large traffic networks can be achieved within a pre-determined real time limit. Starting with the current road conditions supplied by an operator, the simulator was shown to run much faster than real time. A testing exercise [10] was carried out to check the simulator’s correctness, and to evaluate its performance by comparing the outputs from the sequential version with those from the parallel version. These measured the speed-up when different numbers of processors were employed compared to when only one processor was used. The correctness tests were designed to prove that the algorithms employed by both versions do not alter the nature of the results when the simulation is run under identical conditions. The strategies used in the ‘correctness’ test involved making the size of the road traffic model a variable parameter while keeping the number of vehicles generated by each of the two versions constant. In the performance test, the number of vehicles generated by each of the two versions was varied while keeping the size of the road traffic model constant. The results of the tests for one processor and using 4 processors show the different execution times for the two versions both using a grid road network and a total simulation time of 35 minutes for different number of vehicles. When the number of vehicles becomes very large, the speed up may approach a limit that is approximately near the theoretical limit n where n is the number of workstations used in the distributed version. For realistic as well as meaningful evaluation, the correctness tests were carried out on a real road network in Nice, south France.
5
Summary and Conclusions
Through the application of low cost but high performance computing environments, advanced simulation technologies and industrial software production methodologies within the project, a fast, representative, flexible and visually interactive traffic simulation tool have become available. This paper has described the work and results of the HIPERTRANS traffic simulation system. The software architecture employed comprises: the object-oriented framework for traffic simulation, PROSIT, PACSIM, and SPIDER. SIPIDER enabled the capability of distributed simulation so that the software resides on a cluster of workstations and provides a sequentially consistent parallel execution. The speed of the simulator permits faster than real-time prediction of road network behaviour to be made. The simulator can enable UTC operators to examine the future effect of their selected actions on levels of congestion fast enough to be able to revise and re-test the performance before selecting the best action to take. The HIPERTRANS project has proven that microscopic simulation can be one of the most effective tools in the design and management of road traffic systems. In comparison to related work, other useful simulation tools [11]-[14] have been developed in the past, but further advancement in high performance simulation
HIPERTRANS: High Performance Transport Network Modelling and Simulation
873
technology was motivated by the fact that large scale microscopic simulation are time consuming and so further research was carried out to look into the effective exploitation of distributed computing to speed up the simulation further in realistic road networks. This project has assured that the use of scalable parallel computing ensures that irrespective of the size and complexity of the modelled network, faster than real time criterion can be met. Notwithstanding the advancement made within the HIPERTRANS project, more practical aspects need to be addressed. For future work, we recommend the development of an open simulation farmework which has the capability of integrating distributed executions with more user-friendly 3-dimensional graphical and animation interfaces, a wider variety of real-time UTC systems, and capability of receiving data from and sending data to traffic data and information systems. Future systems should not only be able to receive real-time traffic data and predict the future state of traffic, but should be able to incorporate emission models and GIS data. Other EC, national, and other research projects who have already built other types of simulators such as macroscopic simulators, emission models, and automatic debiting simulators, freight transport and crowd management simulators could benefit from HIPERTRANS results. It is possible for such other simulators to integrate with a microscopic road traffic simulator. Our collaboration with such builders will yield the possibilities of interfacing their work with our system. The purpose of such a modular and open simulator would be to use it in the area of transport, traffic and telematics where several opportunities are foreseen in typical applications such as traffic management, floating car data, and route guidance.
6
Acknowledgement
The HIPERTRANS project was funded by the European Commission’s DG 7 under the Framework 4 programme. We are grateful to the EC and all the project partners.
7 1. 2. 3. 4. 5. 6. 7. 8.
References HIPERTRANS Consortium, 1997, Deliverable D1: System Specification, HIPERTRANS RO-97-SC-1005, July 1997 IJAHA S.E. WINTER, S.C, KALENTERY N. and DANIELS B.K.: ‘HIPERTRANS: a road traffic simulation as an operational tool’, Paper No. 46 in the 1998 IEE International Conference on Simulations, University of York, UK, 30 Sept – 2 Oct 1998. SIEGEL, G, FURMENTO, N, and MUSSI, P.: "A Traffic Simulator for Advance Transport Telematics (ATT) Strategies" IEEE conference Electrotechnological Services for Africa AFRICON'99, 29/09/99 - 01/10/99, Cape Town, South Africa. PEEK Traffic Systems, ‘SCOOT System Overview’, 1995 ETRA I+D, 1996, STU-I+D-05-01, Version 3.1/1.2, Sistema de Trafico Urbano, Descripcion General STU. HIPERTRANS Consortium: HIPERTRANS GUI and User’s Manual, Annex 2 of Deliverable D5: Demonstration, HIPERTRANS RO-SC-1005, June 1999. CORNELIS, E., and TOINT, P. L.: ‘An introduction to PACSIM: A new dynamic behavioural model for traffic assignment’, Report 1997/98. GAUJAL, B., and MUSSI, P.: PROSIT Manual and HIPERTANS: Object Oriented Simulation for Urban Traffic’, INRIA Internal Report, 1995.
874 9. 10. 11. 12. 13. 14.
Stephen E. Ijaha, Stephen C. Winter, and Nasser Kalantery KALENTERY, N., SPIDER, Internal Report, Centre for Parallel Processing, University of Westminster, 1995 HIPERTRANS Consortium: Deliverable D5: HIPERTRANS Demonstration, HIPERTRANS RO-97-SC-1005, July 1999. TSS: Transport Simulation Systems. AIMSUN. http://www.tss-bcn.com/aimsun.html Rickert M. and P. Wagner. Parallel Real-time Implementation of Large-scale, Route-plandriven Traffic Simulation. http://www.zpr.uni-koeln.de/GroupBachem/VERKEHR.PG/ PARAMICS: Traffic Simulation. PARAMICS, see also http://wwwusa.quadstone.com/paramics/ext/index.html Nuttall I, and Fellendorf M.: VISSIM for Traffic Signal Optimisation. Traffic Technology International '96. Annual Review Issue. 1996. pp190-2.
Topic 13 Routing and Communication in Interconnection Networks Jose Duato Vice-Chair
The papers in this workshop cover a wide spectrum, dealing with both theoretical and practical aspects. First of all, four regular papers propose improved strategies and models for collective communication, routing in irregular topologies, reliability estimation, and deadlock avoidance. These papers are summarized below: – A Bandwidth Latency Tradeoff for Broadcast and Reduction, by Peter Sanders and Jop Sibeyn: This paper proposes the fractional tree algorithm, a new hybrid scheme for broadcasting and reduction, whose communication pattern interpolates between the sequential pipeline and pipelined binary tree algorithms. The paper also plots the reduction in broadcast latency over the sequential pipeline and pipelined binary tree algorithms in the absence of contention. The proposed broadcasting and reduction scheme is very interesting and can be directly included in communication libraries such as MPI. It significantly improves latency for relatively long messages. – Improving the Up/Down Routing Scheme for Networks of Workstations, by Jose Carlos Sancho and Antonio Robles: This paper proposes new heuristic rules to compute the routing tables in NOWs with irregular topology. The paper also proposes a traffic balancing algorithm to obtain more efficient routing tables when source routing is used. Evaluation results show that the routing algorithm based on the new methodology increases throughput by a factor of up to 2.8 in large networks, also reducing latency significantly. – A New Reliability Model for Interconnection Networks, by Rosa Alcover and Vicente Chirivella: This paper proposes a new reliability model for interconnection networks that considers all the fault patterns and their probability of occurrence in a tractable manner. The proposed model considers the behavior of the routing algorithm. This model uses Markov chains to model degraded network states. Fault patterns are grouped by considering the symmetry of the network and the fault patterns supported by the routing algorithm. Moreover, in order to make the analysis tractable, the number of degraded states is limited to those that have a non-negligible probability of being reached. The paper compares the behavior of the proposed reliability model with the simpler approach consisting of considering the worst case, showing the drastic increase in accuracy achieved by the proposed model. – Deadlock Avoidance for Wormhole Based Switches, by Ingebjørg Theiss and Olav Lysne: This paper applies well-known deadlock avoidance techniques A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 875–876, 2000. c Springer-Verlag Berlin Heidelberg 2000
876
Jose Duato
to a new and interesting problem of building scalable ”compound” switches from smaller ”networked” switch modules in a way in which deadlocks are avoided. In doing so, the authors use extra logic in each switch module to perform flow control such that internal blocking within the switch is eliminated between different packet and control streams. Six short papers complete the contents of this workshop. They mostly cover the theoretical aspects of the workshop, including analytical models and theoretical bounds. But some of the papers also cover performance evaluation and techniques to improve performance. These papers are summarized below: – Probability-Based Fault-Tolerant Routing in Hypercubes, by J. Al-Sadi, K. Day and M. Ould-Khaoua: This paper describes a new fault-tolerant routing algorithm for hypercubes based on probability vectors. The proposed routing algorithm significantly improves over previously proposed schemes based on safety vectors. – Performance Analysis of Pipelined Circuit Switching, by Geyong Min and Mohamed Ould-Khaoua: This paper proposes an analytical model of pipelined circuit switching for hypercubes. It supports virtual channels. The evaluation results show that it is accurate for low and medium network loads. – Optimal broadcasting in even tori with dynamic faults, by Stefan Dobrev and Imrich Vrt’o: This paper obtains an upper bound for broadcast in faulty k-ary n-cubes with even k using the all-port model. The paper assumes that faults are dynamic, at most 2n-1 links are faulty, and faults are distributed in the worst possible manner. – Experimental Evaluation of Hot-Potato Routing Algorithms on 2-Dimensional Processor Arrays, by Constantinos Bartzis, Ioannis Caragiannis, Christos Kaklamanis and Ioannis Vergados: This paper presents an application of a very well known routing scheme under static and dynamic packet generation. The authors state that the approach they are proposing would be very suitable for all-optical networks. – Broadcasting in all-port wormhole 3-D meshes of trees, by Petr Salinger and Pavel Tvrdik: This paper proposes a broadcast algorithm for threedimensional meshes of trees, assuming wormhole switching and the all-port model. The paper also proves the optimality of the proposed algorithm with respect to the number of steps or rounds. – A Clustering Approach for Improving Network Performance in Heterogeneous Systems, by V. Arnau, J. M. Ordu˜ na, S. Moreno, R. Valero and A. Ruiz: This paper proposes a clustering technique to split networks of workstations with irregular topology into clusters. It also proposes a metric to assess the quality of process-to-processor mappings and experimentally shows the high correlation existing between the proposed metric and network performance.
Experimental Evaluation of Hot–Potato Routing Algorithms on 2–Dimensional Processor Arrays Constantinos Bartzis1 , Ioannis Caragiannis2, Christos Kaklamanis2, and Ioannis Vergados2 1
Department of Computer Science and Engineering University of California, Santa Barbara, USA 2 Computer Technology Institute and Dept. of Computer Engineering and Informatics, University of Patras, 26500 Rio, Greece.
Abstract. In this paper we consider the problem of routing packets in two–dimensional torus–connected processor arrays. We consider four algorithms which are either greedy in the sense that packets try to move towards their destination by adaptively using a shortest path, or have the property that the path traversed by any packet approximates the path traversed by the greedy routing algorithm in the store–and–forward model. In our experiments, we consider the static case of the routing problem where we study permutation and random destination input instances as well as the dynamic case of the problem under the stochastic model for the continuous generation of packets.
1
Introduction
We consider a form of packet routing known as hot–potato routing. The network is modeled as a directed graph where the nodes are the processors and the unidirectional edges are communication links between processors. Each processor has an injection buffer and a delivery buffer. When a new packet is generated, it is stored in the injection buffer of its source processor; when a packet reaches its destination processor, it is stored in the delivery buffer. The routing is performed in discrete, synchronous time steps. During each step, a processor receives zero or one packet along each incoming edge and must send all the packets it received out along outgoing edges with at most one packet leaving per outgoing edge. No buffers are required to hold the packets between the time steps. Any packet that arrives at a node other than its destination must immediately be forwarded to another node. The topology we consider in this paper is that of the 2–dimensional torus–connected processor array.
This work was partially funded by the European Union under IST FET Project ALCOM–FT and RTN Project ARACNE. An extended version of the paper can be found at http://students.ceid.upatras.gr/˜caragian/bckv00.ps Part of this work was performed while the author was at the Department of Computer Engineering and Informatics, University of Patras, Greece.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 877–881, 2000. c Springer-Verlag Berlin Heidelberg 2000
878
Constantinos Bartzis et al.
In static (or batch) routing problems, all processors generate a single packet simoultaneously. The running time of a routing problem is the number of time steps required to deliver all packets to their destinations. In dynamic routing problems, each node continously generates packets with an injection rate λ. New packets are stored in the injection buffer and wait to be served. When a processor receives less than four packets along its incoming edges, it considers a packet from its injection buffer. When a packet starts moving, it is never buffered at any node until it reaches its destination, where it is stored in the delivery buffer and absorbed. The first hot-potato algorithm was proposed by Baran [1]. Borodin and Hopcroft [4], Prager [13] and Hajek [8] presented algorithms for hypercubes. Hot–potato routing algorithms for 2–dimensional meshes and tori were proposed by Bar–Noy et al. [2], Ben–Aroya et al. [3], Newman and Schuster [12], Kaufman et al. [10], Feige and Raghavan [7], and Kaklamanis et al. [9]. All of them deal with batch routing problems. The only study of the dynamic case we are aware of is that of Broder and Upfal [5]. An important class of hot–potato routing algorithms is that of greedy algorithms. In these algorithms, each node forwards each packet closer to its destination whenever possible. Although greedy algorithms have been observed to work well in practice (for static routing problems), the known theoretical results for their performance are far from being tight (see Busch et al. [6]). Especially for meshes and tori, a class of hot–potato routing algorithms that has received much attention is that of algorithms that make packets follow paths that approximate their natural greedy path (i.e., the path utilized by the greedy routing algorithm in the store–and–forward model [11]). Such algorithms were proposed and analyzed in [9].
2
Short Description of the Algorithms
In this section, we briefly describe four algorithms on the 2–dimensional torus network, namely the greedy algorithm, algorithm A1, algorithm KKR, and algorithm A2. The greedy algorithm is a variation of the folklore algorithm mentioned in the bibliography. Algorithm KKR was proposed in [9]. Algorithm A1 is a simple algorithm that “approximates the greedy path” while algorithm A2 is a variation of algorithm KKR. To our knowledge, algorithms A1 and A2 have not been studied in previous work. The greedy algorithm. The greedy algorithm which was implemented tries to move packets towards their destination by adaptively using shortest paths, also trying to minimize the difference between the horizontal and the vertical distance of each packet from its destination. The decisions the algorithm makes are local, since they depend on the destination of the incoming packets and the order in which these packets are examined. The implementation of the algorithm obeys the one–pass property [8]. A simple algorithm that approximates the greedy path. From the point of view of the motion of the packets, packets routed by algorithm A1, start
Experimental Evaluation of Hot–Potato Routing Algorithms
879
moving into their rows and continuously turn right so that they move around their destination row, until they reach it (see Figure 1).
(a)
(b)
(c)
(d)
Fig. 1. Typical movement of packets performed by (a) the greedy algorithm, (b) algorithm A1, (c) algorithm KKR, and (d) algorithm A2.
The algorithm KKR [9]. Each packet p starts moving along its row, following the shortest path towards its destination column. When it reaches its destination column, p attempts to enter the column and moves towards its destination. If it fails, it moves “back and forth” until it succesfully turns into the right column. A variation of the algorithm KKR. Algorithm A2 is based on the following idea: during the time steps that a packet p is moving “back and forth” along its row trying to turn into the correct column, it could also try to turn at some other node trying to decrease the vertical distance of the packet from its destination. This can be seen as a movement of a packet in its row until it reaches its destination column, and then, greedy movement to the destination processor. Thus, algorithm A2 maintains both properties: it is greedy and also approximates the greedy path. The typical shape of the paths traversed by the packets during the execution of the algorithms is depicted in Figure 1.
3
Experimentation
The four algorithms where implemented in C, and the results were conducted by simulation experiments on a Pentium III/500Mhz running Solaris 7. In the static model, packets are initially stored at the injection buffer and start moving according to the routing algorithm. Initially, each processor has one packet stored in its injection buffer. We consider routing problems with random destinations (i.e., each packet is assigned as destination a processor, selected among all the processors of the network, uniformly at random) and random permutations (i.e., the routing problem is selected uniformly at random among all possible permutations). In our experiments, the parameter of interest was the running time of the algorithms. Statistics on the running time of the algorithms on routing problems with random destinations are depicted in Table 1. The results for random permutations are close to those for random destinations.
880
Constantinos Bartzis et al. 200 × 200 500 × 500 Average Max. Std dev. Average Max. Std dev. Greedy 202.69 207 1.509 505.85 512 2.032 A1 201.39 205 1.214 501.16 506 1.412 KKR 205.19 214 2.881 505.13 511 2.553 A2 204.98 217 3.387 505.2 515 2.785
Table 1. Statistics from the execution of the four algorithms on 100 routing problems with random destinations in tori 200 × 200 and 500 × 500. The average and maximum observed running time, as well as the standard deviation of the running time is shown. The performance of all algorithms is very close to the optimal. We believe that all the four algorithms route (almost all) batch routing problems in time n + O(log n) on the n × n torus. Such a strong theoretical result has only been proved for algorithm KKR in [9]. Algorithm A1 has slightly better performance than the other three algorithms. Surprisingly, algorithm A2 does not improve the running time of algorithm KKR. In the dynamic model, packets with random destinations are continuously generated at each processor with a rate λ (i.e., at each time step, a processor generates a packet with probability λ, independently from the other processors) and stored in the injection buffer. Injection buffers have been implemented as FIFO (first–in–first–out) queues; so the network together with the injection buffers is considered as a queueing system. Once a packet leaves the injection buffer of its origin processor, it starts moving according to the routing algorithm until it reaches its destination, where it is stored in the delivery buffer and absorbed (leaves the system). In our experiments under the dynamic model, parameters of interest were the delay of packets, the number of packets in the system (i.e., the number of packets in injection buffers and packets being routed) and the network throughput, i.e., the maximum injection rate for which the system is stable. A theoretical maximum value for the maximum injection rate on the n × n torus is given by λmax = 8/n [11]. We alternatively express the injection rate (and the network throughput) as a percentage of its theoretical maximum value. Although we never observe stable behavior of the system for injection rates very close to the theoretical maximum, we observed network throughput up to 72, 5%. We performed experiments on the 200 × 200 torus for injection rates smaller than 50% (see Table 2). In this case, for the four algorithms we study, the network is stable. We observe that, even for small injection rate, packets experience significant delays when routed with the greedy algorithm, while the average number of the packets in the network is large. Especially the average (total) size of injection buffers when the greedy algorithm is used is more than twice the average size of injection buffers when any of the other three algorithms is used. In our experiments on the 200 × 200 torus, the network throughput observed was about 0, 0276 (69%) for the greedy algorithm, 0.0254 (63.5%) for algorithm A1 0.0265 (66.25%) for algorithm KKR, and 0.029 (72.5%) for algorithm A2.
Experimental Evaluation of Hot–Potato Routing Algorithms Average delay λ = 0.005 λ = 0.01 λ = 0.02 Greedy 0.76 2.547 10.932 A1 0.336 0.943 4.664 KKR 0.475 1.304 6.456 A2 0.444 0.88 4.444
881
Average size of injection buffers λ = 0.005 λ = 0.01 λ = 0.02 189 1,026 9,088 156 386 3,835 78 543 5,383 46 486 3,720
Table 2. Average delay of packets and average total size of injection buffers. The average on the delay was taken among all packets that reached their destinations within 2, 500 steps of execution. The average size of injection buffers was computed for the steps of execution 1000–2500.
References 1. P. Baran. On Distributed Communication Networks. IEEE Transactions on Communications, pp. 1–9, 1964. 2. A. Bar–Noy, P. Raghavan, B. Shieber, and H. Tamaki. Fast Deflection Routing for Packets and Worms. In Proc. of the 12th Annual ACM Symposium on Principles of Distributed Computing, pp. 75–86, 1993. 3. I. Ben–Aroya, T. Eilam, and Schuster. Greedy Hot–Potato Routing on the Two– Dimensional Mesh. Distributed Computing, 9(1):3–19, 1995. 4. A. Borodin and J. Hopcroft. Routing, Merging, and Sorting on Parallel Models of Computation. Journal of Computer and System Sciences, 30:130–145, 1985. 5. A. Broder and E. Upfal. Dynamic Deflection Routing on Arrays. In Proc. of the 28th Annual ACM Symposium on the Theory of Computing, pp. 348–358, 1996. 6. C. Busch, M. Herlihy, and R. Wattenhofer. Randomized Greedy Hot–Potato Routing. In Proc. of the 11th Annual ACM/SIAM Symposium on Discrete Algorithms (SODA ’00), pp. 458–466, 2000. 7. U. Feige and P. Raghavan. Exact Analysis of Hot–Potato Routing. In Proc. of the 33rd Annual IEEE Symposium on Foundations of Computer Science, pp. 553–562, 1992. 8. B. Hajek. Bounds on Evacuation Time for Deflection Routing. Distributed Computing, 5:1–6, 1991. 9. C. Kaklamanis, D. Krizanc, and S. Rao. Hot–Potato Routing on Processor Arrays. In Proc. of the 5th Annual ACM Symposium on Parallel Algorithms and Architectures, pp. 273–282, 1993. 10. M. Kaufmann, H. Lauer, and H. Schroder. Fast Deterministic Hot–Potato Routing on Meshes. In Proc. of the 5th International Symposium on Algorithms and Computation, LNCS 834, Springer–Verlag, pp. 333-341, 1994. 11. F.T. Leighton. Average Case Analysis of Greedy Routing Algorithm on Arrays. In Proc. of the 2nd Annual ACM Symposium on Parallel Algorithms and Architectures, pp. 2–10, 1990. 12. I. Newman and A. Schuster. Hot–Potato Algorithms for Permutation Routing. IEEE Transactions on Parallel and Distributed Systems, 6(11): 1168–1176, 1995. 13. R. Prager. An Algorithm for Routing in Hypercube Networks. Master’s thesis, University of Toronto, 1986.
Improving the Up∗ /Down∗ Routing Scheme for Networks of Workstations Jos´e Carlos Sancho and Antonio Robles Departamento de Inform´atica de Sistemas y Computadores Universidad Polit´ecnica de Valencia P.O.B. 22012,46071 - Valencia, SPAIN {jcsancho,arobles}@gap.upv.es
Abstract. Networks of workstations (NOWs) are being considered as a costeffective alternative to parallel computers. Many NOWs are arranged as a switchbased network with irregular topology, which makes routing and deadlock avoidance quite complicated. Current proposals use the up∗ /down∗ routing algorithm to remove cyclic dependencies between channels and avoid deadlock. Recently, a simple and effective methodology to compute up∗ /down∗ routing tables has been proposed by us. The resulting up∗ /down∗ routing scheme makes use of a different link direction assignment to compute routing tables. Assignment of link direction is based on generating an underlying acyclic connected graph from the network graph. In this paper, we propose and evaluate new heuristic rules to compute the underlying graph. Moreover, we propose a traffic balancing algorithm to obtain more efficient up∗ /down∗ routing tables when source routing is used. Evaluation results show that the routing algorithm based on the new methodology increases throughput by a factor of up to 2.8 in large networks, also reducing latency significantly. Keywords: Networks of workstations, irregular topologies, routing algorithms, deadlock avoidance.
1 Introduction NOWs are arranged as a switch-based network with irregular topology which provides the wiring flexibility, scalability, and incremental expansion capability required in this environment. Routing in irregular topologies can be based on either source or distributed routing. In the former case, routing tables are used at each host to obtain the port sequence to be used at intermediate switches to reach the destination. In order to achieve high bandwidth and low latencies, NOWs are often connected using gigabit local area network technologies. There are recent proposals for NOW interconnects like Autonet [8], Myrinet [1], Servernet II [4], and Gigabit Ethernet [9]. Several deadlock-free routing algorithms have been proposed for NOWs, such as up∗ /down∗ routing [8], adaptive-trail routing [5], minimal adaptive routing [7], and smart-routing [2]. However, we will focus on up∗ /down∗ routing because it is the most popular routing scheme currently used in commercial networks, like Myrinet [1].
This work was supported by the Spanish CICYT under Grant TIC97-0897-C04-01.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 882–889, 2000. c Springer-Verlag Berlin Heidelberg 2000
Improving the Up∗ /Down∗ Routing Scheme for Networks of Workstations
"up"
883
d
a
b
e
a
h
g
i
f
c
7
8
(b) b
c
d
e
5 0 f
g
h
(a)
1
2
3
4
6
i
(c)
Fig. 1. (a) Generated BFS spanning tree for a 9-switch network with assignment of direction to links. (b) Generated DFS spanning tree, and (c) assignment of direction to links for the same 9-switch network. In this paper, we propose and evaluate new heuristic rules and a new traffic balancing algorithm that improve the methods to compute up∗ /down∗ routing tables in a NOW environment when source routing is used. Evaluation results show that the routing algorithm based on the new methodology increases throughput by a factor of up to 2.8 for large networks, also reducing latency significantly. The rest of the paper is organized as follows. In Section 2, the up∗ /down∗ routing scheme and the methodologies to compute its routing tables are described. In Section 3, new heuristic rules to compute the up∗ /down∗ routing scheme are proposed. Section 4 describes the proposed traffic balancing algorithm when using source routing. Section 5 shows performance evaluation results. Finally, in Section 6 some conclusions are drawn.
2 Up /Down Routing U p∗ /down∗ is the most popular routing scheme currently used in commercial networks. In order to compute up∗ /down∗ routing tables, different methodologies can be applied. These methodologies are based on an assignment of direction (“up” or “down”) to the operational links in the network by building a spanning tree. These methodologies differ in the type of graph to be built. One methodology is based on a BFS spanning tree, such as it was proposed in Autonet [8], whereas another methodology is based on a DFS spanning tree, as it has been recently proposed in [6]. In networks without virtual channels, the only practical way of avoiding deadlocks consists of restricting routing in such a way that cyclic channel dependencies 1 are 1
There is a channel dependency from a channel ci to a channel cj if a message can hold ci and request cj . In other words, the routing algorithm allows the use of cj after reserving ci . Also, there is a routing restriction when there is no channel denpendency.
884
Jos´e Carlos Sancho and Antonio Robles
avoided [3]. To avoid deadlocks while still allowing all links to be used, up∗ /down∗ routing uses the following rule: a legal route must traverse zero or more links in “up” direction followed by zero or more links in “down” direction. Thus, cyclic channel dependencies are avoided by imposing routing restrictions, because a message cannot traverse a link along the “up” direction after having traversed one in “down” direction. Next, we describe how to compute both a BFS and DFS spanning tree, and how to assign direction to links in each graph. 2.1 Computing a BFS Spanning Tree First, to compute a BFS spanning tree, a switch must be chosen as the root. Starting from the root, the rest of the switches in the network are arranged on a single BFS spanning tree. The “up” end of each link is defined as: 1) the end whose switch is closer to the root in the spanning tree; 2) the end whose switch has the lower identifier, if both ends are at switches at the same tree level. Figure 1(a) shows the resulting link direction assignment for a 9-switch network. 2.2 Computing a DFS Spanning Tree Like in BFS spanning trees, an initial switch must be chosen as the root before starting the computation of a DFS spanning tree. The rest of the switches are added following a recursive procedure [6]. This procedure builds a path that connects all the switches in the network. Figure 1(b) shows the DFS spanning tree obtained from the same network graph used in Figure 1(a). Unlike in the BFS spanning tree, adding switches to build the path is made by using heuristic rules. We will address this issue later. Next, before assigning direction to links, switches in the network must be labeled with positive integer numbers. A different label is assigned to each switch. The “up” end of each link is defined as the end whose switch has the higher label. Figure 1(c) shows the label assigned to each switch. Note that, the DFS spanning tree achieves a lower number of routing restrictions, as can be seen by dashed lines in Figures 1(a) and 1(c) for the BFS and DFS spanning trees, respectively.
3 Applying New Heuristic Rules Several spanning trees can be computed. In order to achieve better performance, heuristics to find the suitable spanning tree are needed. For BFS spanning trees, heuristic rules can only be applied to choose the root switch. The number of different BFS spanning trees that can be computed on a network graph is limited by the number of switches in the network. However, when computing a DFS spanning tree, heuristic rules can be applied to both the selection of the root switch and the selection of the following switches of the spanning tree. Notice that the number of spanning trees that could be computed in this case is very large. We first focus on the heuristic rules for selecting the root switch. So far, two approaches have been used to select the root of a spanning tree: (R0) to select the switch with identifier equal to zero, like in DEC AN1 [8]; (R1) to select the switch with lower
Improving the Up∗ /Down∗ Routing Scheme for Networks of Workstations
885
‘up’ (a)
(b)
(c)
Fig. 2. Different link orientation patterns in a DFS spanning tree.
average topological distance to the rest of the switches, like Myrinet [1]. We propose a new heuristic rule that will be referred to as R2. The heuristic is based on computing all the spanning trees and selecting one of them based on two behavioral routing metrics, that is: (1) the average number of links in the shortest routing paths between hosts over all pairs of hosts, referred to as average distance; and (2) the maximum number of routing paths crossing through any network channel, referred to as crossing paths. We first compute the metrics for each spanning tree obtained by selecting the root among every switch in the network. Finally, the switch selected as the root will be the one that provides the lower value for the crossing paths metric. In case of tie, the switch with lower value of average distance will be selected. In short, the switch selected as the root will be the one that allows more messages to follow minimal paths and provides a better traffic balancing. The time complexity to compute the new heuristic rule is O(n3 ), where n is the number of switches. Unlike BFS spanning tree, after selecting the root switch, a DFS spanning tree still allows heuristic rules to be applied to the rest of the switches when building the spanning tree. We propose the two following heuristic rules: (H1) The switch with higher average topological distance to the rest of the switches is selected as the next switch in the spanning tree, and so on. This heuristic was proposed in [6]. (H2) The switch with a higher number of links connecting to switches that already belong to the spanning tree is selected as the next switch. In case of tie, the H1 heuristic rule is applied. The H2 heuristic reduces the number of routing restrictions by increasing the number of switches whose links exhibit the orientation patterns shown in the Figures 2(a) and 2(b), which provide a lower number of routing restrictions in the switch than that provided by the link orientation pattern shown in Figure 2(c). Table 1 shows the values of the behavioral routing metrics computed for several network2sizes using the up∗ /down∗ routing algorithm based on both BFS and DFS spanning trees, which have been obtained according to the heuristic rules proposed above3. Besides the average distance and the crossing path metrics we have also included the restrictions per switch metric that is the average number of routing restrictions per switch. As can be seen, a lower values of metrics are obtained when using the R2 and H2 heuristic rules to compute spanning trees.
2 3
For further details on topology generation, see Section 5.1. For DFS spanning trees, we assume that the R2 heuristic is used to select the root.
886
Jos´e Carlos Sancho and Antonio Robles
Table 1. Behavioral routing metrics for BFS and DFS spanning trees using different heuristics. Spanning tree
BFS
DFS
Network size 16 32 64 16 32 64
Average distance
Crossing paths
Restrictions per node R1 R2
R1
R2
R1
R2
2.208 3.102 4.013
2.133 2.871 3.787
37 173 593
23 63 238
3.375 3.562 3.281
3.125 2.937 2.875
H1
H2
H1
H2
H1
H2
2.108 2.792 3.634
2.091 2.752 3.590
23 73 204
23 43 190
3.125 2.821 2.687
2.875 2.625 2.585
4 Traf c Balancing Algorithm When a routing algorithm able to provide partial adaptivity, such as up∗ /down∗ routing, is implemented using source routing, a strategy to select a single path between each pair of hosts is needed. Different selection policies can be applied, such as random and round-robin selections. However, they do not guarantee a suitable traffic balancing in the network, which may reduce network performance. We propose a traffic balancig algorithm that tries to achieve an uniform channel utilization, avoiding that a few channels become a bottleneck in the network. First, the algorithm associates a counter to every channel in the network. Each counter is initialized to the number of routing paths crossing the channel, that is, the channel utilization. Additionally, a cost function associated to every routing path according to its channel utilization is evaluated. The procedure defined below is applied repetitively to the channel with highest value of counter. In each step, a routing path crossing the channel is selected to be removed if there is more than one routing path between the source and the destination switch of this routing path. In this way, we prevent the network to become disconnected. When a routing path is removed, the counters associated with every channel crossed by the path are updated. If there is more than one routing path able to be removed in a channel, the algorithm will first choose the routing path whose source and destination hosts have the highest number of routing paths between them. The algorithm finishes when the number of routing paths between every pair of hosts is reduced down to the unit. The time complexity to compute this traffic balancing algorithm is O(n2 ∗ diameter), where n is the number of switches. This time is much lower than that exhibited by other proposals, such as smart-routing [2].
5 Performance Evaluation In this section, we evaluate by simulation the performance of the up∗ /down∗ routing scheme when the heuristic rules and the traffic balancing algorithm proposed in Sections 3 and 4, respectively, are applied to compute the routing tables. Table 2 shows
Improving the Up∗ /Down∗ Routing Scheme for Networks of Workstations
887
the acronyms for the up∗ /down∗ routing algorithms evaluated according to type of spanning tree, heuristic, and traffic balancing algorithm used. Table 2. Acronyms used for the up∗ /down∗ routing algorithms. Routing algorithm
Root heuristic R1
Path heuristic -
Traffic balancing
U D− BF S1
Spanning tree BFS
U D− BF S2
BFS
R2
-
No
U D− BF S2b
BFS
R2
-
Yes
U D− DF S1
DFS
R2
H1
No
U D− DF S2
DFS
R2
H2
No
U D− DF S2b
DFS
R2
H2
Yes
No
5.1 Network Model Network topology is completely irregular and has been generated randomly. We have evaluated networks with 16, 32, and 64 switches. For space reasons, the results for 64 switches have not been plotted. We have generated ten different topologies for each network size analyzed. The maximum variation in throughput improvement of U D− DF S2b routing with respect to U D− BF S1 routing is not larger than 20%. Results plotted in this paper correspond to the topologies that achieve the average behavior for each network size. We assume that every switch in the network has 8 ports, using 4 ports to connect to workstations and leaving 4 ports to connect to other switches. For message length, 32-flit and 512-flit messages were considered. Different message destination distributions have been used like uniform, bit-reversal, and matrix transpose. In order to obtain realistic simulation results, we have used timing parameters for the switches taken from a commercial network. We have selected Myrinet because it is becoming increasingly popular due to having very good performance/cost ratio. According to Myrinet switches, the latency through the switch for the first flit is 150 ns, and after transmitting the first flit, the switch transfers at the link rate of 6.25 ns per flit. The clock cycle is 6.25 ns. Each switch has a crossbar whose arbiter processes one message header at a time. Flits are one byte wide and the physical channel is one flit wide. Also, source routing and wormhole switching is used like in Myrinet. 5.2 Simulation Results Figures 3(a) and 3(b) show the average message latency versus accepted traffic for networks with 16 and 32 switches, respectively. Message size is 32 flits and uniform destination distribution is used. We can observe that the new heuristic (H2) to compute the DFS spanning tree allows U D− DF S2 to reduce latency with respect to U D− DF S1
Jos´e Carlos Sancho and Antonio Robles 1500 1400 1300
1200
1100
’UD_BFS1’ ’UD_BFS2’ ’UD_BFS2b’ ’UD_DFS1’ ’UD_DFS2’ ’UD_DFS2b’
1000
900
800
Average Message Latency (ns)
Average Message Latency (ns)
888
0.005 0.01 0.015 0.02 0.025 0.03 Accepted Traffic (flits/ns/switch)
1900 1800 1700 1600 1500 1400 1300 1200 1100 1000
’UD_BFS1’ ’UD_BFS2’ ’UD_BFS2b’ ’UD_DFS1’ ’UD_DFS2’ ’UD_DFS2b’
0.02 0.015 0.01 0.005 Accepted Traffic (flits/ns/switch)
(b)
(a)
1800
1600
1400
’UD_BFS1’ ’UD_BFS2’ ’UD_BFS2b’ ’UD_DFS1’ ’UD_DFS2’ ’UD_DFS2b’
1200 1000
0.005 0.01 0.015 0.02 0.025 Accepted Traffic (flits/ns/switch)
(a)
Average Message Latency (ns)
Average Message Latency (ns)
Fig. 3. Average message latency vs accepted traffic. Network size is (a) 16 and (b) 32 switches. Message length is 32 flits. Uniform distribution.
1900 1800 1700 ’UD_BFS1’ 1600 ’UD_BFS2’ ’UD_BFS2b’ 1500 ’UD_DFS1’ 1400 ’UD_DFS2’ ’UD_DFS2b’ 1300 1200 1100 1000
0.025 0.02 0.015 0.01 0.005 Accepted Traffic (flits/ns/switch)
(b)
Fig. 4. Average message latency vs accepted traffic. Network size is 32 switches. Message length is 32 flits. (a) Bit-reversal and (b) matrix transpose message distributions.
for every value of traffic. It is due to the fact that the new heuristic introduces lower number of routing restrictions than the previous heuristic (H1), allowing more messages to follow minimal paths. Obviously, the improvement is higher in large networks because messages can profit more from following minimal paths. The improvement in throughput of U D− DF S2 with respect to U D− BF S1 achieves a factor of up to 2.8 for a 64-switch network. Also, the new heuristic (R2) to select the root significantly improves the performance of the up∗ /down∗ routing scheme based on BFS spanning tree with respect to the R1 heuristic. The improvement in throughput of U D− BF S2 with respect to U D− BF S1 ranges from 20% for small networks to 60% for large networks. Moreover, the traffic balancing algorithm only contributes to improve throughput for small network sizes, especially when up∗ /down∗ routing is based on DFS spanning tree. It is due to the fact that channel utilization is higher than for large networks. As a consequence, an algorithm to balance traffic will achieve more benefits. The improvement in throughput of U D− DF S2b with respect to U D− DF S2 is about 16% in 16-switch networks.
Improving the Up∗ /Down∗ Routing Scheme for Networks of Workstations
889
For space reasons, the results for long message (512-flits) are not plotted. The improvement in performance of the up∗ /down∗ routing schemes based on a DFS spanning tree with respect to those based on BFS spanning tree decreases slightly with respect to the one achieved with short messages. Similar results were obtained in [6]. Figures 4(a) and 4(b) show the results for a 32-switch network when using message distributions with temporal locality, such as bit-reversal and matrix transpose. The improvement in performance of U D− DF S2 with respect to U D− DF S1 is noticeably decreased, although the latency reduction is still significant for the entire range of traffic. Notice that U D− DF S2b increases throughput with respect to U D− BF S1 up to a factor of 2.5.
6 Conclusions In this paper, we have proposed new heuristics to obtain the underlying graph used by the up∗ /down∗ routing scheme to compute the routing tables. Moreover, an algorithm to balance the traffic in the network using source routing has been proposed in order to avoid that some channels become a bottleneck in the network. The main contribution of these techniques is that they are able to improve network performance without adding resources to the network that would increase its cost. Simply, routing tables have to be updated. The simulation results modeling a Myrinet network show that the up∗ /down∗ routing algorithm based on DFS spanning tree, when the new heuristic to compute the spanning tree and the traffic balancing algorithm are applied, almost triples the throughput in large networks with respect to the up∗ /down∗ routing algorithm based on BFS spanning tree currently used in commercial networks. For smaller networks, performance improvement is also smaller but the proposed heuristic rules always improve latency and throughput.
References 1. N. J. Boden et al., Myrinet - A gigabit per second local area network, IEEE Micro, vol. 15, Feb. 1995. 2. L. Cherkasova, V. Kotov, and T. Rockicki, Fibre channel fabrics: Evaluation and design, 29th Hawaii International Conference on System Sciences, Feb. 1995. 3. W. J. Dally and C. L. Seitz, Deadlock-free message routing in multiprocessors interconnection networks, IEEE Transactions on Computers, vol. C-36, no. 5, pp. 547-553, May. 1987. 4. D. Garc´ia and W. Watson, Servernet II, in Proceedings of the 1997 Parallel Computer, Routing, and Communication Workshop, Jun 1997. 5. W. Qiao and L. M. Ni., Adaptive routing in irregular networks using cut-trough switches, in Proc. of the 1996 Int. Conf. on Parallel Processing, Aug. 1996. 6. J.C. Sancho, A. Robles, and J. Duato, New Methodology to Compute Deadlock-Free Routing Tables for Irregular Networks, in in Proc. of CANPC’00, Jan. 2000. 7. F. Silla and J. Duato, Improving the Efficiency of Adaptive Routing in Networks with Irregular Topology, in 1997 Int. Conference on High Performance Computing, Dec. 1997. 8. M. D. Schroeder et al., Autonet: A high-speed, self-configuring local area network using point-to-point links, SRC research report 59, DEC, Apr. 1990. 9. R. Sheifert, Gigabit Ethernet, Addison-Wesley, April 1998.
Deadlock Avoidance for Wormhole Based Switches Ingebjørg Theiss and Olav Lysne Department of Informatics, University of Oslo [email protected], [email protected]
Abstract This paper considers the architecture of switches. In particular we study how virtual cut-through and wormhole networks can be used as the switch internal interconnect. Introducing such switches into a deadlock free interconnect may give rise to new deadlocks. Previously, to reason that no deadlocks are created, the resulting system was considered globally, that is, the interconnects of the switches themselves was considered as part of the system. Using flow control across the switch will eliminate the possibility of creating new deadlocks, and further global reasoning will not be necessary.
1
Introduction
Wormhole routing (WHR) has recently been brought into new applications, such as high speed LANs and SANs (i.e. Autonet [1], Myrinet [2] and Servernet [3]), and has been proposed as the internal fabric of switches [4, 5]. The latter is interesting for highly scalable switches. One could consider using a k-ary n-cube [6] as the internal topology when building such a switch, but the result will have an unfortunate property. Unlike a simple crossbar router, the WHR switch may block, that is, a packet may have to wait for another packet, even when the packets have different source and destination ports. Using other internal topologies will generally not change this situation. The problem addressed is that of new deadlocks appearing when crossbar routers are exchanged with blocking switches in a given configuration. This problem is previously mentioned in [5] where it is emphasized that nonblocking switches help preserve inter-switch deadlock freedom, while blocking switches are good when only intra-switch deadlock freedom is required. In [7], however, Lysne argues that if blocking switches are used, reconsidering the structure of the switch-internal configurations and adding a few virtual channels to avoid what he calls aggregated dependencies, will ensure that the blocking property cannot cause inter-switch deadlocks. Another approach is to use extra logic in the switch to avoid the blocking property. This paper considers the latter approach, where the extra logic used is a flow control protocol across the switch. In [8] we considered using a wormhole routed switch in an SCI configuration [9], and addressed the problems connected to the non-blocking property of the A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 890–899, 2000. c Springer-Verlag Berlin Heidelberg 2000
Deadlock Avoidance for Wormhole Based Switches
891
switch. The methods in this paper are not restricted to SCI. The solutions found in [8] are generalized to technologies where a switch cannot drop packets. Also, a substantial amount of simulations have been made: three routing techniques, two topologies and several traffic patterns have been tested. The outline of the following sections is as follows: First, the deadlock problem arising with blocking switches is introduced. Next, three different flow control methods removing the blocking property and thereby the deadlock are proposed. The performance of these methods are then analyzed through simulation, and finally the scalability issues are discussed, showing that very large switches can be constructed with limited buffer capacity at each port.
2
Deadlock Caused by Blocking Switches
To simplify our discussion, we introduce some definitions. We use the term chip to imply that the node under discussion is a simple one-chip routing node, such as a crossbar router. A switch is understood to be a more complex routing node, built with an internal topology of chips connected with links. Typically, a chip has 4 to 32 ports, while a switch is virtually unlimited. When a node lets data be transferred between any pair of ports independently of traffic from any other input port, as long as the output port is not occupied, and every output port also fairly arbitrates between input ports, the node is non-blocking and fair (NBF ). In the following, a chip denotes an NBF-chip.
A (1,1) B A (0,0) B
(a)
(b)
Fig.1. (a) A blocking switch (b) Deadlock in a compound configuration
As an example of a non-NBF-node, consider a switch with 4 ports and a full duplex 2-ary 1-cube as the WHR internal topology connecting the ports
892
Ingebjørg Theiss and Olav Lysne
(Fig 1 (a)). Each NBF-chip (the black squares in the figure) in this topology is connected to two ports. If both inputs at one NBF-chip wishes to send packets to the two outputs at the other NBF-chip, they are competing for the inter-chip link, even though the destinations are distinct. If the input packet which wins the competition is blocked at its output, the packet from the other input is also blocked. A distinction between internal and external configurations in configurations using switches as routing nodes is also necessary. The former denotes the internals of the switches, the latter expresses the configuration disregarding the internal configurations, as if the routing nodes were chips. Furthermore, a compound configuration regards the entire structure, both the external and the internal configurations. A compound configuration is externally deadlock free when its external configuration is deadlock free, and internally deadlock free when all its internal configurations are deadlock free. Now, consider a structure with a deadlock free external configuration. It is obvious that if the internal configurations of the switches have cyclic dependencies, the compound configuration also has cyclic dependencies. However, even if the internal configurations are deadlock free, the compound configuration may still deadlock. Inserting non-NBF deadlock free switches into an externally deadlock free configuration can induce cyclic dependencies in the compound configuration. Let our external configuration be a wormhole routed 2-ary 2-cube using dimension order routing. We use the WHR 2-ary 1-cube from Fig. 1 (a) as the internal configurations. The compound configuration is both externally and internally deadlock free, the configuration is illustrated in Fig. 1 (b). As an example of a deadlocked situation in this configuration, consider two packets, from A to A’ and from B to B’. The packets are initiated simultaneously. The packet from A fills all resources on its path to switch (1,1), thereby occupying the horizontal interconnect of switch (0,0). A similar situation is caused by the packet from B. Since the horizontal interconnect in (1,1) is occupied by the packet from B, the packet from A is blocked. The packet from B is in its turn blocked, waiting for the horizontal interconnect in (0,0) occupied by the packet from A. A circular wait is present, and the configuration is deadlocked. This problem is not limited to wormhole routed networks, it applies to both Store And Forward (SAF) and Virtual Cut-Through (VCT). This follows easily from the fact that any deadlock situation in WHR can be seen as a deadlock situation in both SAF and VCT by considering the WHR flits as SAF and VCT packets. The deadlock can also be found when combining these technologies, ie. a SAF external configuration and WHR internal configurations. To avoid inducing a cyclic dependency we give the switches sufficient properties to become NBF, allowing us to disregard the internal configurations. To achieve this, three requirements must be imposed: (i) the switches must be deadlock free, (ii) a blocked stream crossing a switch from port A to port B cannot persistently block packets crossing the switch from C to D (when C and D are disjoint from A and B, respectively), (iii) the packet scheduling within the switch must be fair.
Deadlock Avoidance for Wormhole Based Switches
893
Theorem 1. If we insert switches that fulfill requirements (i), (ii) and (iii) into an externally deadlock free configuration, the compound configuration will also be deadlock free. Proof 1. The actual proof is by contradiction – we assume our switches introduce a cyclic dependency which has lead to a deadlock situation, and show that this gives us a contradiction. The contradiction is given by the following steps: – The introduced deadlock situation must stem from a cycle of dependencies that includes some inserted switches. – For each of the participating internal configurations the following must hold: • There must be an involved packet that cannot be transmitted successfully from one end of the switch to the other. Assume that the packet needs to be transmitted from buffer A to buffer B. • The requirements (i), (ii) and (iii) implies that the block is caused by buffer B being eternally full, otherwise there is no cause for blocking. • Because this is a deadlock situation there must be a cycle of actions all waiting for each other that includes the successful transmission of our packet across the switch. • The only resource that this packet is eternally occupying is space in the buffer where it is situated. Therefore, since we have a deadlock, this buffer must be full. • This means that the blocking situation local to this switch is due to full buffers at both interfaces. – Since the ring of dependencies in the deadlock situation are due to full buffers at each end of all the participating internal configurations, this would have been a deadlock with NBF-chips as well. This contradicts the premise of an externally deadlock free configuration. To fulfill requirements (i), (ii) and (iii), we shall look at switches with deadlock free WHR configurations. The switch outputs arbitrate fairly between inputs, and in addition, the switches have flow control between inputs and outputs. The flow control will assure that when a packet is allowed to cross the switch, it is in fact removed from the switch’s internal network, allowing other packets to other destinations to use the network. From our example in Fig. 1 (b), the flow control will always free the shared horizontal interconnects, and these interconnects will be fairly shared by the two streams.
3
Flow Control Methods
In this section several methods for implementing the properties identified in the previous section are discussed. All methods uses well-known techniques to control the flow across the interconnect in such a way that the non-blocking and fairness properties are maintained. The performances of the flow control methods are investigated through simulations, and the results are discussed.
894
3.1
Ingebjørg Theiss and Olav Lysne
Source Driven Approach
The first flow control method, the source driven approach (SDA), lets the source port initiate communication by sending a data packet to the network, but the port keeps a copy in case retransmission is necessary. After transmitting, the source waits for an acknowledge packet from the destination. The acknowledge packet indicates whether the destination port accepted the data packet or not, the latter may occur if the output port’s buffers are full. In case of a positive acknowledge, the source copy can be discarded, and the next data packet waiting can be transmitted. In case of a negative acknowledge, the pending data packet must be retransmitted. This process is repeated until the acknowledge indicates acceptance.
B
port acknowledgment buffer (AB), full acknowledge generator (AG) AG to AB, X=blocked link data packet acknowledge packet wanted path blocked
A
chip
Fig.2. Deadlock in SDA using only one transport medium
The destination port returns an acknowledge packet by putting it in an acknowledge packet buffer, where it will reside until the switch internal network has capacity to transmit it. If this buffer is full, the destination port must leave the data packet in the internal network, blocking a substantial amount of resources there. If acknowledgment traffic uses the same transport medium as data traffic, a deadlock situation may occur. Consider the two ports A and B in Fig. 2. A data packet is going to A through the link connecting B and A. A’s acknowledgment buffer is full, so A has to block the data packet until there is room. The corresponding situation applies at B. The acknowledge packets at the top of the acknowledgment buffers are requesting the links occupied by the data packets, and have to wait until the links are freed. There is a circular wait, and the system is deadlocked. Using a dedicated transport network for the acknowledge packets, either as a logical or a physical network, solves this problem. To ensure fairness, SDA uses an A-B aging scheme when packets are rejected at the destination port [9]. The negative acknowledge informs the source port of how to mark retransmitted packets. From this information, the destination can guarantee any packet a maximum waiting time, that is, an upper bound on the number of other packets being served first.
Deadlock Avoidance for Wormhole Based Switches
895
Letting the source port be idle while waiting for the flow packet is an obvious subject for improvement by allowing the transmitting protocol engine to have M active buffers holding the data packets waiting for acknowledgment. For simplicity, we have used M = 1. 3.2
Destination Driven Approach
In the destination driven approach (DDA), the source port has destination associated counters (credits) indicating how many packets the source port is allowed to send to a destination port. The credit is decremented when a packet is sent to the destination port, and incremented upon receipt of a special flow control packet returned by this destination port. The destination port accepts all data packets actually sent by the source port. Since source ports cannot possibly know the credits of its neighbors, each destination must have buffer space dedicated to each source port, and the space must be large enough to store at least one data packet. When a packet in such a dedicated buffer has successfully been delivered to the receiver on the local subnet, the special flow control packet is sent to the source port to increase its credit again. When the maximum value of credits is 1, DDA will degenerate to a stop and wait-protocol. The required buffer space limits the scalability of DDA to some extent. Using a simple technique, where each source has room for one data packet in each destination, the total buffer space needed is #sources × #destinations × maximumpacketsize which is of order O(#sources2 ), since a source is normally also a destination. It could be possible to timeshare buffer space by some fair strategy, but this method will not scale well either, neither in time dimension nor in buffer cost. The internal deadlock problem occuring in SDA is not relevant in DDA. No packet is sent unless it will be accepted by the destination, so all packets will eventually be removed from the internal network, making room for the flow control packets. Fairness is guaranteed when the selected strategy for submitting packets from the destination buffer onto the local network segment is fair. In our simulations, we use a round robin strategy. 3.3
Draining Network Approach
A third method, the draining network approach (DNA), is a modified version of SDA; DNA needs just a single network. To avoid the deadlock problem SDA solves by using two networks, the destination in DNA guarantees that all packets will be removed from the transport medium upon arrival. Consequently, if no new data packets are sent, the transport medium will eventually be emptied (drained), which implies that no packet can block acknowledge packets eternally. Some strategy must be used to solve the full acknowledgment buffer problem, otherwise the switch will drop packets. Let each source port in DNA provide
896
Ingebjørg Theiss and Olav Lysne
buffer space large enough to keep the highest number of acknowledge packets needed at one time. As for SDA, the number of pending packets at each source is set to 1 (M = 1). This means that each destination buffer must have room for #sources acknowledgments, and total buffer space needed in the switch is therefore #sources × #destinations × size of acknowledge packet. This order is similar to the non-scalable situation of DDA, but we exploit the characteristics of the worst case, and relax the strict FIFO strategy used for flow packets by SDA and DDA, thereby saving a respectable amount of buffer space. Instead of storing the entire acknowledge packet, each destination port holds an array with one index for each source. One array location is as small as 3 bits. A null value in a location indicates that no acknowledge packet is pending for this source port, while the other values can store the status of the latest received packet from this source port. A round robin strategy is used to send the acknowledge packets to the internal network. In all three approaches, we give acknowledge packets the highest priority, that is, when choosing between sending an acknowledge packet and a data packet on the internal network, we will always send the acknowledge packet, regardless of how long the data packet has been waiting. 3.4
Simulation
Our simulations of one switch comprise three different configurations, all using WHR. Two topologies are used, a 6 × 6 2-dimensional grid and a 4 × 4 Clos network [10]. Using the grid, we have simulated two different routing algorithms, the XYrouting algorithm [11] and the West First algorithm [12]. XY-routing is a deadlock free, dimension ordered, deterministic routing protocol. West First is a partially adaptive, deadlock free routing algorithm based on the Turn Model. These configurations have been tested with four traffic patterns: uniform, hot-spot, matrix transpose and matrix merge south. The uniform and the hotspot patterns let sources communicate with different destinations. Each time a packet is transmitted, its destination is randomly picked. As the names suggest, with uniform traffic all destinations have the same probability of being picked, with hot-spot, destinations have a higher chance of being picked the closer they are to the hot-spot. In both matrix patterns, each source port picks a destination at startup, and never sends to anyone else. In matrix transpose each destination port receives packets from one source port only, while in merge south all source ports send southwards, and each destination receives packets from two sources. The routing algorithm simulated with the Clos network is very simple: all packets are routed adaptively into the middle layer, and then deterministically to their destinations. The Clos configuration has also been tested with four traffic patterns, uniform and hot-spot as in the grid, and reflection and split-down, which resembles the matrix patterns in that each source picks one destination, and in the latter, each destination has two sources.
Deadlock Avoidance for Wormhole Based Switches
XY: Latency (matrix transpose) 4500
XY: Throughput (matrix transpose)
Source driven approach Destination driven approach Drained network approach
3500 3000 Clockticks
Source driven approach Destination driven approach Drained network approach (x+5)
280 Number of packets accepted per 2000 clockticks
4000
897
2500 2000 1500 1000 500
260 240 220 200 180 160 140 120
0
120
140
160
180
200
220
240
260
280
120
140
Number of packets offered per 2000 clockticks
160
(a) All matrix pattern latencies are similar to this plot.
Number of packets accepted per 2000 clockticks
Clockticks
25000
20000
15000
10000
5000
40
60
80
100
280
25
20
15
10
5
120
20
40
60
80
100
120
Number of packets offered per 2000 clockticks
(c) All hot spot latencies are similar, but in West First, SDA crosses DNA
(d) All hot spot throughputs are similar to this plot
West First: Latency (uniform)
West First: Throughput (uniform) 240
Source driven approach Destination driven approach Drained network approach Number of packets accepted per 2000 clockticks
6000
5000
Clockticks
260
30
Number of packets offered per 2000 clockticks
7000
240
Source driven approach Destination driven approach Drained network approach (x+5)
35
30000
20
220
Clos: Throughput (hotspot)
Source driven approach Destination driven approach Drained network approach
0
200
(b) All matrix pattern throughputs are similar to this plot, but scales vary
Clos: Latency (hotspot) 35000
180
Number of packets offered per 2000 clockticks
4000
3000
2000
1000
Source driven approach Destination driven approach Drained network approach (x+5)
220
200
180
160
140
120
100 0
120
140
160
180
200
220
240
260
280
Number of packets offered per 2000 clockticks
(e) Unlike the other uniform latency plots, West First has higher latency and DDA crosses DNA
120
140
160
180
200
220
240
260
280
Number of packets offered per 2000 clockticks
(f) Unlike the other uniform throughputs, DNA is better than DDA in West First
Fig.3. Selected simulation results
898
Ingebjørg Theiss and Olav Lysne
Packets arrive from the network to the source port at uniformly distributed random time intervals. When the average time interval decreases, the network load increases. The graphs below show the latency and throughput of each flow control method with increasing load, the complete set of results can be found in [13]. Packets are 90 flits long. The most prominent result is that the SDA performs better than both DDA and DNA, regardless of workload, traffic pattern, topology and routing algorithm. (A couple of exceptions: latency is slightly worse in SDA than DNA using West First routing and the hot-spot pattern, and throughput in XY shortly drops below DDA and DNA using the matrix merge south pattern.) With the uniform traffic pattern, SDA is significantly better. Using XY the throughput graphs show that SDA handles traffic up to app. 220 packets per 2000 clock ticks, while MDM and TTM only handle ca. 190, SDM is approximately 15% “better”. Using West First SDA is approximately 14% “better”, using the Clos network approximately 25%. Generally, SDA is much better, but using the matrix patterns in XY, hot-spot in West First and reflection for the Clos network the differences are relatively small. The results from comparing DDA with DNA variates more. Using the uniform traffic pattern, DDA performs better with XY and particularly in the Clos network, while DNA works best with West First. However, even though DNA is logically similar to SDA, it performs very similar to DNA. For uniform traffic, DDA has better throughput than DNA when M = 1, but not by an extensive amount. Latency is much better, though. This similarity in performance is possibly due to the fact that the sources send to the same destination. For the uniform traffic, DNA does not have to wait for an acknowledgment from the destination before sending the next packet to a new destination port, since it is credit based. This is no advantage when the destination port is static. With the grid topology and XY routing, ports at the edge of the network have higher latency and lower throughput than ports in the middle. Using the adaptive routing algorithm, the eastern nodes have a lower service rate. Using the symmetric Clos network, no nodes are disfavoured.
4
Conclusion
We have seen how an end to end flow control method across a switch can hide the blocking property and allow switches to be used as routing nodes in an interconnect such as if they were non-blocking. There are several ways of how to implement such a flow control, we proposed both source driven and destination driven approaches. By simulation, we found the source driven approach to have better performance than the destination driven approach, but at a cost of an extra internal network. To avoid the extra network, a third approach, also source driven, was suggested, and found to have lower costs than the destination driven approach, but basically the same performance.
Deadlock Avoidance for Wormhole Based Switches
899
Further work in this area could be to add more topologies and routing techniques, or to find out how the configurations behave if the network segments at the output ports becomes saturated. Another issue is to lower the costs of the drained network approach further. By implementing an additional timeout strategy and allowing a very small probability for dropped packets, the costs could be strongly reduced. A spin-off of our investigation is the Clos vs. mesh issue. Apparently, our Clos network performs better than our meshes. This is not very surprising, considering that the hop distance in the Clos network is only two. The chances of a header flit wanting an occupied out-link are therefore far less than in a mesh, where the average hop distance is much higher. The incentive for using wormhole routing in Clos is for the same reason less, though. On the cost issue; our small Clos network is more cost effective when counting chips and links, but it doesn’t scale as well as mesh networks do.
References [1] M. D. Schroder et.al. Autonet: a high-speed, self-configuring local area network using point-to-point links. SRC Research Report 59, Digital Equipment Corporation, 1990. [2] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik, C. L. Seitz, J. N. Seizovic, and Wen-King Su. Myrinet: A Gigabit-per-Second Local-Area Network. IEEE Micro, 15(1):29–36, 1995. [3] Robert W. Horst and David Garcia. ServerNet SAN I/O Architecture. Hot Interconnects V, 1997. [4] Vibhavasu Vuppala and Lionel M. Ni. Design of A Scalable IP Router. In Hot Interconnects V, 1997. [5] Lionel M. Ni, Wenjian Qiao, and Mingyao Yang. Switches and Switch Interconnects. In Proceedings of the Fourth International Conference on Massively Parallel Processing Using Optical Interconnections, pages 122–129, 1997. [6] Jos´e Duato, Sudhakar Yalamanchili, and Lionel M. Ni. Interconnection Networks: an Engineering Approach. IEEE Computer Society, 1997. [7] Olav Lysne. Deadlock Avoidance for Switches based on Wormhole Networks. In Proceedings of the 1999 International Conference on Parallel Processing, AizuWakamatsu (Japan), pages 68–74. IEEE Computer Society Press, 1999. ISBN 0-7695-0350-0. [8] Geir Horn, Ingebjørg Theiss, Olav Lysne, and Tor Skeie. Switched SCI Systems. In Scalable Coherent Interface: Technology and Applications, Proceedings of SCI Europe ’98, pages 13–24. Cheshire Henbury, 1998. [9] IEEE 1596-1992. The Scalable Coherent Interface (SCI), 1992. [10] Charles Clos. A Study of Non-Blocking Switching Networks. Bell Syst. Tech. J., 32:406–424, 1953. [11] Lionel M. Ni and Philip K. McKinley. A Survey of Wormhole Routing Techniques in Direct Networks. Computer, 1993. [12] Christopher J. Glass and Lionel M. Ni. The Turn Model for Adaptive Routing. Journal of the Association for Computing Machinery, 1994. [13] Ingebjørg Theiss and Olav Lysne. Simulation results for deadlock free wormhole based switches. Research Report 284, University of Oslo, Department. of Informatics, June 2000. ISBN 82-7368-232-3 ISSN 0806-3036.
An Analytical Model of Adaptive Wormhole Routing with Deadlock Recovery Mohamed Ould-Khaoua and Ahmad Khonsari Computer Science Department, Strathclyde University, Glasgow, U.K. {mohamed, ak}@cs.strath.ac.uk
Abstract: This paper proposes a new analytical model to predict the mean message latency in k-ary n-cubes with Compressionless routing, a recovery-based fully-adaptive routing proposed by Kim et al [2].
1 Introduction Deadlock recovery as a viable alternative to deadlock avoidance has recently gained consideration in the scientific community. It has been shown that deadlocks are quite rare except when the network is close to saturation [2]. Thus the hardware dedicated for deadlock avoidance is not necessary most of the time. This consideration has motivated the authors in [2] to introduce Compressionless Routing as a framework for developing recovery-based fully-adaptive routing algorithms. This paper proposes the first analytical model of compressionless routing in k-ary n-cubes.
2 The Analytical Model Compresionless routing and the router structure are described in detail in [2]. The model is based on the following assumptions [3]. i) Message destinations are uniformly distributed. Nodes generate traffic independently of each other, following a Poisson process with a mean rate of λ g messages/cycle. The message length is m flits, where m is a random variable with a mean m . A flit requires one-cycle transmission time across a physical channel. ii) The timeout period is τ cycles. The probability of timeout at a channel is independent of the subsequent channels. When a transmission failure occurs due to timeout, a message is transmitted after a time gap of V cycles, where V may be any random variable with a mean V . Successive message re-transmissions are independent of the each other. iii) L ( L ≥ 1) virtual channels are used per physical channel. The mean message latency is composed of the mean network latency, t r , and the mean waiting time, ws , in the source node. However, to capture the effects of virtual channels multiplexing, the mean message latency has to be scaled by a factor, l , representing the average degree of virtual channels multiplexing. Therefore, we can write [3] Latency = (t r + w s )l (1) A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 900-903, 2000. Springer-Verlag Berlin Heidelberg 2000
An Analytical Model of Adaptive Wormhole Routing with Deadlock Recovery
901
The average number of channels that a message crosses along each of the n dimensions and across the network, k and d respectively, are given by [3]
k = (k − 1) / 2
(2)
d = nk (3) Consider a message that has was transmitted successfully through the network. Since a message crosses, on average, d channels, the distribution of the network latency in the case of a successful transmission can be written as TS ( x) = Prob(m + d + d ⋅ w ≤ x) (4) where w denoting the waiting time at a channel. Since the waiting times at two successive channels are independent of each other and since the Laplace-Stieltjes Transform (LST) of the sum of two independent random variables is equal to the product of their LSTs [1], the LST of TS (x) is given by ∞ − sx
∫0 e
T S* ( s ) =
dTS ( x) = M * ( s )e − sd W * ( s ) d
(5)
Consider a message that experiences a timeout at the i-th hop channel. The LST of the network latency in the case of a transmission failure due to a timeout at the i-th hop channel can be written as
TF*i ( s) = e − s ( 2(i −1) +τ )W * ( s ) (i −1)
(6)
Let ψ i be the number of channels that a message can select at its i-th hop, and, Pt be the probability that message experiences a timeout at channel. If Pl denotes the probability that l virtual channels at a physical channel are busy, the probability, PFi , that a message suffers a timeout at the i-th hop channel can be expressed as ψ
ψj
∏ (1 − Pt
ψ
PFi = Pt i PL i
j =1..i −1
ψ
PL j )
(7)
Given that a transmission failure can be caused by a timeout at any of the d channels along the message path, the LST of the network latency in case of a transmission failure is given by T F* ( s ) =
∑ PF TF* (s)
i =1..d
i
(8)
i
The probability of a successful transmission, PS , and that of transmission failure, PF , are simply given by PS =
∏ (1 − Ptψ PLψ ) i
i =1..d
PF = 1 − PS = 1 −
i
(9)
∏ (1 − Ptψ PLψ ) i
i
(10)
i =1..d
The network latency of a single transmission try, taking into the network latency in case of a transmission success and transmission failure, can be written as T * ( s ) = PS TS* ( s ) +
∑ PF TF* (s)
i =1..d
i
i
(11)
902
Mohamed Ould-Khaoua and Ahmad Khonsari
Let t r be a random variable denoting the network latency seen by a message after r re-transmission attempts. Sine a new re-transmission is delayed for a time gap of v cycles and it is independent of the previous re-transmissions, we can write Tr* ( s ) =
∑ PS PFr −1T * (s) r V * (s) r −1 = PS T * (s) /(1 − PF T * (s)V * (s)) (12)
r =1..∞
The number of channels that a message can use at its i-th hop is found to be [3]
ψi =
∑ lPi j
jk + 1 ≤ i ≤ ( j + 1)k
l =0..j
()
j Pi = nj N 0k −1 (i − 1 − jk , n-j ) /
N pp + q −1 (r , m)
(0 ≤ j ≤ n − 1)
∑ (ln )N 0k −1 (i − 1 − lk , n − l ) (14)
l =0.. j
(
∑
(13)
(−1) l ( lm ) r − mpm−−lq1+ m −1 = l =0..m
)
(15)
Modelling a channel as an M/M/1 queue with impatient customers and with deterministic impatient time yields a simple and practical model that exhibits a reasonable degree of accuracy. The rate of traffic on each channel is given by
λc =
ψj
∑ ∏ (1 − Pt
i =0..d −1 j =1..i
ψ
PL j )λ 0 , where λ o = λ g / nPS
(16)
Following the suggestions proposed of [4], the mean waiting time and probability of timeout at a given channel can be approximated as
w=
t λc t 2 − + λ c t τ e (1 − λ c t ) 1 − λ c t −(1−λc t )τ 2 2 t 1 − λ c t e
− (1−λc t )τ t
− Pt τ (17)
(1 − Pt )
−(1−λc t )τ
− (1− λc t )τ
t t (18) Pt = [(1 − λ c t )λ c t e ] /[1 − λ 2c t 2 e ] Modelling the local queue in the source as an M/M/1 queue with a mean arrival rate λ g / L and a mean service time t r yields the mean waiting time as [1]
ws = (λ g / L)t r2 /(1 − (λ g / L)t r )
(19)
The probability, Pl , that l virtual channels at a given physical channel are busy is determined using a Markovian model (see [3] for a more detailed discussion). In the steady state, the model yields the following probabilities. l=0 1 (20) q l = q l −1λ c t 0
An Analytical Model of Adaptive Wormhole Routing with Deadlock Recovery
903
∑
1 / ql l=0 l = 0 .. L (21) Pl = Pl −1λ c t 0
∑ i 2 Pi / ∑ iPi
i =1..L
(22)
i =1..L
There are inter-dependencies between the different variables of the model, and these are solved iteratively [1]. Fig. 1 depicts latency results predicted by the model and simulation in the 2-D torus (with N=64 nodes; M=32 and 64 flits; L=1 and 2 virtual channels; τ =32 and 64 cycles; V=32 and 64 cycles). The model predicts latency with a good accuracy under light and moderate traffic. However, its accuracy degrades near the saturation point due to the approximations that were used to develop the model, e.g. assumption (ii). Model, M=32
Model, M=32
Latency Model, M=64 (cycles)
Latency Model,M =64 (cycles)
Simulation
Simulation
(b)
(a) 0.001
0.002
Traffic (messages/cycle)
0.0025
0.001
0.002
Traffic (messages/cycle)
0.003
Fig. 1: Latency from the model and simulation in the 2-D torus. a) L=1, b) L=2.
3 Conclusion This paper has presented a new analytical model for fully-adaptive routing with deadlock-recovery, based on the Compressionless routing framework [2], to predict message latency in wormhole-routed k-ary n-cubes. Simulation experiments have revealed that the analytical model predicts latency with a good degree of accuracy.
References [1] [2] [3] [4]
L. Kleinrock, Queueing Systems: Theory, vol. 1, J. Wiley, New York, 1975. J. Kim, A. Chien and Z. Liu, Compressionless routing: A framework for adaptive and fault-tolerant routing, IEEE TPDS 8(3), 1997, pp. 229- 244. M. Ould-Khaoua, An analytical model of Duato’s fully-adaptive routing in k-ary ncubes, IEEE Trans. Computers, 44(12), 1999, pp 1-8. H.C. Tijms, Stochastic modelling and analysis: A computational approach, J. Wiley, 1986.
Analysis of Pipelined Circuit Switching in Cube Networks Geyong Min and Mohamed Ould-Khaoua Department of Computer Science, University of Strathclyde, Glasgow G1 1XH, U.K. Email: {geyong , mohamed}@cs.strath.ac.uk Abstract. This paper proposes the first analytical model of pipelined circuit switching (PCS) in cube networks. The model uses a Markov chain to analyse the backtracking actions of the header flit during the path set-up phase in PCS. One of the main features of the present model is its ability to capture the effects of using virtual channels. The validity of the model is demonstrated by comparing analytical results to those obtained through simulation experiments.
1 Introduction Several recent studies have revealed that pipelined circuit switching (PCS) can provide superior performance characteristics over wormhole switching because it combines the advantages of both circuit switching and wormhole switching [2], [4]. In PCS, a reserved path from the source to the destination is set up prior to the transmission of the data as in circuit switching. However, PCS differs in the way that paths are established. When the header cannot progress because all the required virtual channels are busy, it releases the last reserved virtual channel by backtracking to the preceding node, then continues its search from the node to find an alternative path to the destination. Since seized channels are released when blocking occurs, deadlock cannot emerge during message routing in PCS. Thus, unlike in wormhole switching, fully adaptive routing can be cheaply implemented in PCS. This paper presents the first analytical model of PCS in hypercubes (or cubes for short). The model uses a Markov chain to calculate the mean time to set up a path, and M/G/1 queueing systems to compute the mean waiting time that a message experiences at a source before entering the network. Results from simulation show close agreement with those predicted by the model.
2 Analysis The model is based on the following assumptions: (1) Message destinations are uniformly distributed across the network nodes. (2) Nodes generate traffic independently of each other, which follows a Poisson process with a mean rate of λ g messages/cycle. (3) Message length is M flits, each of which requires one cycle to cross from one router to the next. (4) The local queue in the source node has infinite capacity. Moreover, messages at the destination node are transferred to the local processing element as soon as they arrive at their destinations. (5) Each physical channel is divided into V (V ≥ 1) virtual channels. (6) Exhaustive Profitable Backtracking (EPB) [4] routing protocol is used. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 904-908, 2000. Springer-Verlag Berlin Heidelberg 2000
Analysis of Pipelined Circuit Switching in Cube Networks
905
The mean message latency is composed of the mean network latency, S , that is the time to cross the network, and the mean waiting time seen by the message in the source node, Ws . However, to model the effects of virtual channels timemultiplexing, the mean message latency has to be scaled by a factor, V , representing the average degree of virtual channels multiplexing at a given physical channel. Therefore, we can write [6] Latency = ( S + Ws )V (1) Under the uniform traffic pattern, the message whose destination is i (1 ≤ i ≤ n)
()
hops away, can reach in nodes out of a total of ( N − 1) nodes in the network. The average number of channels, d , that a message visits to reach its destination is therefore given by n n ∑ i i =1 i n N = (2) d= N −1 2 N −1 The mean network latency, S , consists of two parts: the mean time to set up a path, C , and the actual message transmission time. Thus, S can be written as S =C+d +M (3) In order to calculate the mean path set-up time, C , we use a Markov chain [3] to model the header actions to establish the path. State π i (0 ≤ i ≤ d ) in the Markov chain corresponds to the case where the header is at the intermediate node that is i hops away from the source node. Let Ci denote the expected duration to reach state
π d starting from state π i . A transition out of state π i to π i −1 implies that the header has encountered blocking and has to backtrack to the preceding node. The residual duration becomes Ci −1 . The transition rate is the probability, Pbi , that the header is blocked in the node corresponding to state π i . However, the transition out of state π i to π i +1 denotes that the header succeeds in reserving the required virtual channel and advances one hop closer to its destination. The remaining duration is Ci +1 . The transition rate is the probability 1 − Pbi . Given that the header requires one cycle to move from one node to the next, the above argument reveals that the expected duration Ci satisfies the difference equations
(
)
Ci = 1 − Pbi Ci +1 + Pbi Ci −1 + 1
(1 ≤ i ≤ d-1)
(4)
where the states π 0 and π d satisfy the following boundary conditions
(
)
C0 = 1 − Pb0 (C1 + 1) + Pb0 C0
(5)
Cd = 0
(6)
Solving the above equations (4~6) yields the expected duration time, C 0 , to reach state π d starting from state π 0 . The mean path set-up time can be written as C = C0 + d
(7)
906
Geyong Min and Mohamed Ould-Khaoua
where the term d accounts for the d cycles that are required to send the acknowledgement flit back to the source. On average, C channels are visited before a path is set up. Half of the visits occur to reserve the virtual channels in the direction leading to the destination node and another half take place in the opposite direction using the reserved channels. Since a router has n output channels and the local node generates λ g messages per cycle, the mean arrival rate on a channel, λc , can be approximated
λc =
λgC
(8)
2n The probability of blocking, Pbi , depends on the header's current network position. A header is blocked at the intermediate node that is i hops away from the source if all possible virtual channels at the remaining (d − i ) dimensions are busy. Let PV denote the probability that V virtual channels at a given physical channel are busy ( PV is determined below). The probability, Pbi , can be written as
Pbi = (PV
)d −i
(0 ≤ i ≤ d − 1)
(9)
The probability, Pt (0 ≤ t ≤ V ) , that t virtual channels at a given physical channel are busy, can be determined using a Markovian model [1]. In the steady state, the model yields the following probabilities. 1 q t = q t −1λ c S q λ (1 / S − λ ) c t −1 c
t =0 0 < t
(10)
t =V
1 / ∑Vl=0 q l t =0 (11) Pt = Pt −1 λ c S 0 < t
Analysis of Pipelined Circuit Switching in Cube Networks
907
As a result, the mean waiting time becomes Ws =
(λ g / V )S 2 (1 + (S − M − 3d )2 / S 2 ) 2(1 − (λ g / V )S )
(14)
3 Model Validation The above model has been validated by means of a discrete-event simulator. Fig. 1 depicts mean message latency results predicted by the above model plotted against those provided by the simulator in the 64 and 256 nodes cube networks. The figure reveals that the simulation results closely match those predicted by the analytical model in the steady state regions.
400
2-ary 8-cube, V=9 400
Model, M=32
Latency (cycles)
Latency (cycles)
2-ary 6-cube, V=7
Simulation
300
Model, M=48
200 100 0 0
0.01
0.02
0.03
Traffic Rate (messages/cycle)
0.04
Model, M=32 Simulation
300
Model, M=48
200 100 0 0
0.01
0.02
0.03
0.04
Traffic Rate (messages/cycle)
Fig. 1: Latency predicted by the model and simulation, n=6 and 8, V=7 and 9.
4 Conclusion This paper has presented an analytical model of PCS in cube networks augmented with virtual channels. The simplicity of the model makes it a practical and costeffective evaluation tool. The next step in our work is to develop a model for a highradix k-ary n-cube.
References 1. 2. 3.
Dally, W.J.: Virtual channel flow control. IEEE Trans. Parallel & Distributed System. 2 (1992), 194-205 Duato, J., Yalamanchili, S., Ni, L.: Interconnection networks: An engineering approach. IEEE Computer Society Press (1997) Feller, W.: An introduction to probability theory and its applications, Vol. 1, John Wiley, New York, (1967)
908 4. 5. 6.
Geyong Min and Mohamed Ould-Khaoua Gaughan, P.T., Yalamanchili, S.: A family of fault-tolerant routing protocols for direct multiprocessor networks. IEEE Trans. Parallel & Distributed System. 5 (1995), 482-497 Kleinrock, L.: Queueing Systems Vol. 1, John Wiley, New York (1975) Ould-Khaoua, M.: A performance model for Duato’s fully adaptive routing algorithm in k-ary n-cubes. IEEE Trans. Computers. 12 (1999), 1-8
A New Reliability Model for Interconnection Networks1 Vicente Chirivella, Rosa Alcover Department of Statistics and Operation Research, Polytechnic University of Valencia, Camino de Vera s/n, 46020 Valencia, Spain {vchirive, ralcover}@eio.upv.es
Abstract. The traditional approach to study fault-tolerance in multicomputer interconnection networks consists of determining the worst possible combination of faulty components that causes a network failure, and then assuming that this will occur. But the worst possible combination does not always occur, and the routing algorithm allows the network to work in the presence of a greater number of failures. Thus network reliability parameters computed according to the traditional approach will be underestimated. In this paper we propose a new methodology to compute accurately the reliability function. The reliability parameters have been computed for an interconnection network with mesh topology, taking into account size, routing algorithm, failure and repair rates of the network channels and coverage.
1 Introduction Nowadays the growing necessity of computing power has led to computer engineers to design multicomputers with a large number of processing units. As the number of components in a multicomputer increases, the probability of one or more component faults also increases. Therefore, along with performance, dependability characterization of multicomputers is essential to evaluate their effectiveness for commercial, scientific and critical-mission applications. Dependability is a generic term used to address reliability, availability, security, maintainability and other related issues [1]. The interconnection network, the subsystem that supports the messagepassing mechanism, becomes a key issue in determining such dependability. With the purpose of being able to guarantee system dependability, the objective is to design an interconnection network that works in the presence of faulty components. The designs of fault-tolerant interconnection networks can be divided into two categories: dynamic and static. Dynamic designs have redundant components and switches that allow the reconfiguration of the network and the preservation of the original topology [2]. Obviously, this solution leads to an excessively high cost as the number of nodes increases. The static design does not require the use of additional network components. The static approach takes advantage of the alternative paths existing in the network by using fault-tolerant routing algorithms [3], [4]. These algorithms bypass faulty components in the network. However, the maximum number of faults supported by these algorithms is bounded [3]. Static design has been 1
This work was supported by the Spanish CICYT under Grant TIC97-0897.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 909-917, 2000. Springer-Verlag Berlin Heidelberg 2000
910
Vicente Chirivella and Rosa Alcover
traditionally preferred by designers. Thus, we have evaluated the interconnection network reliability when using a static design. Many authors have designed fault-tolerant routing algorithms. These studies obtain the worst possible combination of failures that the routing algorithm can support, and then they assume that it will occur [4]. This is the traditional approach. However, the worst possible combination does not always occur, and the routing algorithm is usually able to route in the presence of a larger number of failures. In this paper we measure the differences between approaches, the traditional one and a more accurate one. With this objective, in section 2 we propose a new methodology for reliability prediction of interconnection networks based on a very useful statistical tool: the continuous-parameter Markov chains [5]. This methodology takes into account topology, network size and routing algorithm used, and allows us to measure the effect of the routing algorithm on the network reliability. Then, in section 3 we apply our methodology to a network with 2D mesh topology. In this section we also propose a model to compute the reliability function and the mean time to network failure, and compare the obtained results with both approaches. Finally, in section 4 some conclusions are drawn. We will show that network reliability parameters obtained with the traditional approach are always underestimated.
2 A Methodology to Evaluate Reliability Based on Markov Chains The proposed methodology is based on Markov chains. They provide very flexible, powerful, and efficient means for the description and analysis of dynamic system properties. The necessary tasks to apply the reliability methodology in the field of interconnection networks can be summarized in the following steps: 1- Define the Interconnection Network Fault Model. This step requires the interconnection network selection and the hypothesis assumed establishment on its operation. As it is well-known, a network is defined when its topology, flow control mechanism and routing algorithm are specified. On the other hand we must establish the hypothesis assumed in the network operation when a failure occurs. 2 - Select the network dependability parameters. In this step, we must select the dependability parameters that will quantify the reliability characteristic that we want to study. There is a large group of statistical parameters to evaluate reliability [6]. Some of these parameters are adequate to measure the reliability characteristics of gracefully degraded systems. Gracefully degrading systems react to a detected failure by reconfiguring to a state that may have a decreased level of performance. 3 - Define the network states and the transition rates. Now, we must specify the Markov chains used for model the network functioning. For this, it is necessary to define the states that represent the network operation and to establish the transitions between them. The network states are defined taking into account the reliability parameters and according to the number of faulty channels in the network. 4 - Compute the values of the network reliability parameters. In this step we must solve the system of differential equations that govern the Markov chain [5]. It provides the state probabilities. Thus, the expressions of the network reliability parameters selected can be obtained analytically. 5 - Analyze the results. Finally the results must be analyzed.
A New Reliability Model for Interconnection Networks
911
3 Applying the Reliability Methodology In this section we apply the analytical methodology summarized in section 2. First, in section 3.1 we propose the network fault model (step1). Then, in section 3.2 we propose two models to compute the reliability function and the mean time to failure for an interconnection network with mesh topology. One model is proposed from the traditional point of view and the other under our approach (steps 2 to 4). Finally in section 3.3 we obtain and analyze the results (step5). 3.1 Fault Model Step1. Wormhole switching is the prevalent technique in the current generation of message passing multicomputers [7]. When wormhole is used, low-dimensional meshes and k-ary n-cubes achieve a higher performance than other topologies as hypercubes [8]. Many recent experimental and commercial multicomputers, as Intel Paragon and MIT Reliable Router [9], use a mesh. A mesh is a two dimensional grid with k-nodes in each dimension. In this paper we study the reliability of a network with 16x16-mesh topology, and wormhole as flow control mechanism. Concerning to the routing algorithm, a good fault-tolerant routing algorithm should be simple, use few virtual channels, support maximum adaptivity in routing and use minimal paths when possible. Other desirable features are deadlock-freedom, good performance under no-fault scenarios, and the ability to handle a large number of faulty components [10]. Duato’s Double East Last West Last fault-tolerant routing algorithm [3] possesses most of these desirable characteristics, and hence, is the routing algorithm studied in this paper. Concerning to the assumed hypotheses on network operation, we suppose that a network fails when one or more nodes cannot communicate with each other, either because there are no physical links between them, or because the routing algorithm cannot select a route to reach the destination node. We also assume that if there is a selectable path between two nodes, the path will be selected. Nodes are reliable, and only the failures of channels are studied in this paper. Some faulty channel combinations may disconnect the network. Thus, if the network is disconnected, the network fails. A failed channel simply ceases to work and the nodes at the end of a malfunctioning channel stop using that channel. Fault-tolerant routing algorithms automatically recover the system from the occurrence of some channel failures during normal operation. The recovery consists of the detection of the fault, the identification of the faulty component, the correction of the errors induced by the fault, and notification to the neighboring nodes that there is a faulty component in the network. However, fault detection mechanisms are not perfect, and they can fail with a certain probability. The probability of system recovery when a fault occurs is called coverage, C [11]. In this work, this probability is assumed to be constant. Finally, the network is repairable. When the interconnection network is repaired, it is completely repaired, replacing all the faulty components. On the other hand, the modeling of the interconnection network has been based on continuous-parameter Markov chains. For this, we have considered exponential distribution with parameter λ for the channel failure times, exponential distribution
912
Vicente Chirivella and Rosa Alcover
with µ parameter for the network reparation times, and a uniform distribution of message destinations. The chosen values for these parameters are the following. The mean time to channel failure (1/λ) is measured in months, its typical value being between 1 and 6 months. The lowest value corresponds to multicomputers assembled in cabinets, while the highest one corresponds to machines mounted in rooms and wired externally. The mean time to network repair (1/µ) is measured in hours. We have considered 3, 6, 12, 24, 48 and 72 hours. The lowest value corresponds to military applications, while the highest value correspond to non-critical applications. Finally, the chosen values for coverage (C) are 0’95, 0’99 and 0’999. For instance, 0'95 means that the probability of failure recover when a channel fails is 0'95. These parameters are used in the following steps. 3.2 Computing Reliability Parameters Step2. For the sake of clarity, in this paper we have selected a reduced set of reliability parameters: the reliability function and the mean time to network failure. The reliability function (R(t)) at time t provides the probability that the network works correctly until this time t, given that it was operational at time 0. The mean time to network failure (MTTF) is the mean time at which the network first fails, and can be obtained integrating the reliability function. Step3. Now, we must specify the Markov chains used for modeling the two approaches proposed in this paper. First, we specify the traditional approach and its transition rates. Then, we will do the same with our approach, and will pay special attention to the expressions of the transition rates between states. In our paper, we define the network states taking into account the MTTF and according to the number of network faulty channels. The following states have been defined: correct state, when there is no faulty component; degraded state, when there is a faulty channel, but the fault detection mechanisms have detected the fault and the routing algorithm can transmit messages between any pair of nodes; and failed state, when the failure of the network occurs, whatever its cause is. With these states, we propose two models (Fig.1) to compute the network reliability function. The model on the left corresponds to the traditional approach, while the other one fits to our approach. The traditional approach to fault tolerant routing on a 2D mesh assumes that the routing algorithm can support a single failure; that is, the network fails with the failure of the second channel (worst combination). This is the case when the two channels connected to a node in the mesh border fail. Thus, the transition rates among states depend on the size of the mesh but not on the algorithm used for routing. As it is considered that the algorithm cannot route after the second failure, there are no differences among the fault-tolerant routing algorithms proposed by different authors. The model on the left, shown in Fig.1, allows us to obtain the reliability function of the interconnection network. From the correct state C.S., the network changes to the degraded state D.S.1 (transition rate a1) if a failure occurs and the fault is covered, or to the failed state F.S. (transition rate a2) if the fault is not covered. In the degraded state, the network changes to the correct state if repaired (µ) or to the failed state when the next failure occurs (b1). For example, in a 16x16 mesh, the transition rate a1 is the product of the number of channels (480), the coverage and the failure rate of a channel (a1=480Cλ). The transition rate a2 is the product of the number of
A New Reliability Model for Interconnection Networks
913
channels, the probability of failure in the fault detection mechanism and the failure rate of a channel (a2=480(1-C)λ). The transition rate b1 is the product of the number of remaining non-faulty channels and the failure rate of a channel (b1=479λ). This rate is not a function of coverage because the network fails with the second failure, regardless of whether the fault is covered o not. It must me noticed that the interconnection network can work with more faulty components than the ones allowed in the worst case. This is the new approach we propose in this paper. In order to limit the model complexity while keeping a high accuracy, we assume that after the fifth failure the performance of the network is too low and the multicomputer is turned off. Therefore, we only include the degraded states from 1 to 4 these values being the number of failed channels at each moment.
Correct State
Correct State
a1
µ D.S.1
b1 Failed State
a2
a2
a1
µ
µ
b1
c2
D.S.2
µ
Failed State
b2
D.S.1
e1 d2
c1 D.S.3
d1
D.S.4
µ
Fig. 1. Reliability model for traditional (left) and new approach (right). The transition rates depend on the size of the mesh and on the routing algorithm used. Effectively, as the algorithm can route messages if certain combinations of failure locations occur, the transition probabilities will depend on the fault-tolerant routing algorithm chosen. The state diagram on the right in Fig.1 shows the network states and the transitions between them when our approach is used. The network starts in the correct state and evolves to the degraded state when a fault occurs if: (1) the failure is covered and, (2) the routing algorithm can still establish communication between any pair of nodes. The network changes to the failed state if at least one of the two conditions fail. The difference with the reliability model for the traditional approach is that from the degraded state D.S.1, the network can reach another degraded state, D.S.2. The transition rate between the two states depends on the ability of the routing algorithm to maintain communication after a new failure. Once the failure combinations have been obtained, the transition rates between states can be computed. In our model, the expression of the transition rate qi,i+1 from the degraded state i (i≥0) to the next degraded state i+1 is given by:
914
Vicente Chirivella and Rosa Alcover
qi ,i +1 = ( N − i )rpi +1Cλ ,
(1)
and the transition rate from the degraded state i to the failed state is given by
qi ,i +1 = ( N − i )(1 − rpi +1C )λ ,
(2)
with rp i +1 = nfc i +1 (ncf i ( N − i )) where C is coverage; λ is the channel failure rate; N is the number of channels; rpi+1 is the probability of network working when a new failure takes place, knowing that i failures have already taken place in an operational network; and nfci+1 is the number of combinations of i+1 faulty channels that do not cause the network failure, knowing that such combinations are obtained from the combinations of i faulty channels that did not cause the network failure, with the positions of a new faulty channel. The transition rates for each state diagram in Fig.1 are shown in Table 1. They are computed according both traditional approach and our approach, for a 16x16 mesh and the fault tolerant routing algorithm DELWL. Table 1. Transition rates for the traditional and the new approach. a1=480 C 8 a2=480 (1-C) 8 a1=480 C 8 a2=480 (1-C) 8 b1=478'625C 8 b2=(479-478'625C) 8
Traditional Approach b1=479 8 New approach c1=477'098 C 8 c2=(478-477'098 C) 8 d1=475'9433 C 8 d2=(477-475'9433C) 8
e1=476 8
Step4. The network reliability function, R(t), is the sum of the probabilities of being in one operational state at each instant of time [6], and the mean time to failure can be obtained integrating this function. The reliability function for the traditional approach is the sum of the probabilities of being in the correct state, P(C.S.), or in the degraded state, P(D.S.1) R (t) = P(C.S.) + P (D.S. 1) while in our approach the reliability function is the sum of the probabilities of being in the degraded states, numbered from 1 to 4, or in the correct state: R (t) = P(C.S.) + P (D.S.1) + P (D.S.2) + P (D.S.3) + P (D.S.4) With the values of λ, µ, and C, the reliability function and the MTTF has been computed and the results are used to compute the ratios shown in Fig.2.
A New Reliability Model for Interconnection Networks
915
3.3 Results Step5. To compare the mean time to network failure, we use the ratio of the mean time to failure obtained with our approach and the mean time to failure obtained with the traditional approach. The results appear in Fig.2, represented as a function of channel failure rate, network repair rate, and coverage. As shown in the plots, the MTTF ratio is always larger than one. Therefore, the MTTF obtained with our approach is always larger than the MTTF obtained with the traditional approach. The ratio is in the range 1'4 to 49, that is, the network MTTF obtained with our approach can be up to 49 times larger than the value obtained with the traditional approach. As our reliability model is much more accurate than the traditional one, the MTTF obtained with the traditional approach is largely underestimated. The differences between both approaches increase with failure coverage. This fact can be observed in the sequence of plots in Fig.2. This occurs because the network tends to transit to a degraded state instead of to the failed state, and then there are more opportunities to network repair. The difference between both approaches also increases when the channel failure rate diminishes, and the repair rate and coverage increases. The effects of coverage and network repair diminish as the failure rate increases. This occurs because the network state evolves quickly to the failed state, and the importance of the possibility of network repair diminishes. Only when coverage is high and the repair time is low, the degraded states are important and they mark the difference between both approaches.
MTTF ratio
MTTF ratio Coverage 0’95 - 16x16 mesh 5,0
Coverage 0’99 - 16x16 mesh 16,0 14,0
4,0
12,0 10,0
3,0
8,0
2,0
6,0
1,0
4,0
0,0
0,0
2,0
0,0002 0,0004 0,0006 0,0008 0,0010 0,0012 0,0014 channel failure rate (1/hours)
0,0002 0,0004 0,0006 0,0008 0,0010 0,0012 0,0014 channel failure rate (1/hours)
MTTF ratio Coverage 0’999 - 16x16 mesh 50,0 40,0 30,0 20,0
Mean Time to Repair (hours) 3 12 48 6 24 72
10,0 0,0 0,0002 0,0004 0,0006 0,0008 0,0010 0,0012 0,0014 channel failure rate (1/hours)
Fig. 2. MTTF ratio for a 16x16 mesh, as a function of channel failure and repair rates and coverage
916
Vicente Chirivella and Rosa Alcover
We can see in the first plot of Fig.2 that the MTTF ratio presents a maximum. This is due to the effects of the degraded states on the network reliability. Those effects are due to the new operational states, which allow more opportunities to network repair. The importance of those effects can be reinforced or diluted by particular values of the failure and repair rates, and this is the cause for the ratio evolution. Finally, a more detailed study of the mean time to failure, including the network size, and another dependability parameter, the steady-state availability, is available as a technical report at the GAP web server [12].
4 Conclusions In this paper we have proposed a methodology for computing dependability in interconnection networks based on Markov chains. The Markov chains have allowed us to model the effects of the routing algorithm and the failure locations in the network. State transitions in Markov chain has been determined by considering the topology (a 16x16 mesh), the routing algorithm (DELWL) and the number and locations of faulty channels. This study has take into account that networks may work when the number of faulty channels is larger than the number of faulty channels supported by the routing algorithm. We have determined that network reliability parameters computed using the traditional method are too conservative. It must be emphasized that the mean time to failure is always larger if it is computed according to our approach instead of the traditional approach. The differences between mean times to failure are up to 49 times larger, according to the size of the mesh. The differences grow as the coverage increases, the mean time to repair is short and mean time to failure is long. We have shown that the coverage of failures is the most important parameter on determining the mean time to network failure. Finally, our model is close to reality and consequently provides more realistic values of network reliability parameters.
References 1. Bolch, G., Greiner, S., de Meer H. and Trivedi, K.S.: Queueing Networks and Markov Chains, Wiley-Interscience, (1998). 2. Tsai, J. and Kuo, S.: Constructions of Link-Fault-Tolerant q-ary n–cube Networks, Electronics Letters (1997), vol.33, no.12, 1025-1026. 3. Duato, J.: A Theory of Fault Tolerant Routing in Wormhole Networks, IEEE T. on Par. and Distr. Systems (1997), vol.8, no.8, 790-802. 4. Gaughan P.T. and Yalamanchili S.: A Family of Fault-Tolerant Routing Protocols for Direct Multiprocessor Networks, IEEE T. on Par. and Distr. Systems (1995), vol.6, no.5, 482-497. 5. Trivedi K.S. Probability and Statistics with Reliability, Queuing, and Computer Science Applications, Prentice-Hall (1992). 6. Beaudry M.D.: Performance-Related Reliability Measures for Computing Systems, IEEE T. on Computers (1978), vol.27, no.6, 540-547. 7. Ni, L. and McKinley, P.: A Survey of Wormhole Routing Techniques in Direct Networks, Computer (1993), vol. 26, no. 2, 62-67.
A New Reliability Model for Interconnection Networks
917
8. A. Agarwal, "Limits on interconnection network performance", IEEE T. on Par. and Distr. Systems (1991), vol. 2, no. 4, pp. 398-412 9. Dally, W.J., Dennison, L.R., Harris, D., Kan, K. and Xanthopoulus T.: The Reliable Router: A Reliable and High-performance Communication Substrate for Parallel Computers, Proc. of the Workshop on Parallel Computer Routing and Communication (1994), 241-255. 10. Vaidya, A.S., Das, R.C. and Sivasubramaniam A.: A Testbed for the Evaluation of FaultTolerant Routing in Multiprocessor Interconnection Networks, IEEE T. on Par. and Distr. Systems (1999), vol.10, no.10, 1052-1081. 11. Dugan, J.B. and Trivedi, K.S.: Coverage Modeling for Dependability Analysis of FaultTolerant Systems, IEEE T. on Computers (1989), vol.38, no.6, 775-787. 12. Chirivella V. and Alcover R.: Improving the accuracy of reliability models for interconnection networks, Technical report (1999), http://www.gap.upv.es/index_eng.html.
A Bandwidth Latency Tradeoff for Broadcast and Reduction Peter Sanders and Jop F. Sibeyn Max-Planck-Institut f¨ ur Informatik Im Stadtwald, 66123 Saarbr¨ ucken, Germany. {sanders,jopsi}@mpi-sb.mpg.de. http://www.mpi-sb.mpg.de/{∼sanders,∼jopsi}
Abstract. The “fractional tree” algorithm for broadcasting and reduction is introduced. Its communication pattern interpolates between two well known patterns — sequential pipeline and pipelined binary tree. The speedup over the best of these simple methods can approach two for large systems and messages of intermediate size. For networks which are not very densely connected the new algorithm seems to be the best known method for the important case that each processor has only a single (possibly bidirectional) channel into the communication network.
1
Introduction
Consider P processing units, PUs, of a parallel machine. Broadcasting, the operation in which one processor has to send a message M to all other PUs, is a crucial building block for many parallel algorithms. Since it can be implemented once and for all in communication libraries such as MPI [9], it makes sense to invest into algorithms which are close to optimal for all P and all message lengths k. Since broadcasting is sometimes a bottleneck operation, even constant factors should be considered. In addition, by reversing the direction of communication, broadcasting algorithms can usually be turned into reduction algorithms. Reduction is the task to compute a generalized sum i
Partially supported by the IST Programme of the EU under contract number IST1999-14186 (ALCOM-FT). For very short messages, different algorithms based on trees with large degree near the root are better, also a synchronous communication model is less attractive.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 918–926, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Bandwidth Latency Tradeoff for Broadcast and Reduction
919
time t + k to transfer a message of size k regardless which PUs are involved. This is realistic on many modern machines where network latencies are small compared to the start-up overhead t. Both sender and receiver have to cooperate in transmitting a message. We are considering two variants. Our default is the duplex model where a PU can concurrently send a message to one partner and receive a message from a possibly different partner. We use the name send|recv to denote this parallel operation in pseudo-code. The more restrictive simplex model permits only one communication direction per processor. The broadcasting time for simplex is at most twice that for duplex communication for half as many PUs.2 We note the cases where we can do better. We begin our description in Sec. 2 by reviewing simple results on pipelined broadcasting algorithms. By arranging the PUs in a simple chain, execution time ∗ =k 1+O tP/k + O(tP ) (1) T∞ can be achieved. Except for very long messages, a better approach is to arrange the PUs into a binary tree. This approach achieves broadcasting time3 T1∗ = k 2 + O( t log(P )/k) + O(t log P ) (2) (replace “2” by “3” for the simplex model). We also give lower bounds. The main contribution of this paper is the fractional tree algorithm described in Sec. 3. It is a generalization of the two above algorithms and achieves an execution time of 1/3 t log P ∗ T∗ = k 1 + O + O(t log P ), (3) k i.e., it combines the advantage of the chain algorithm to have a (1 + o(1)) factor in the k dependent term with the advantage of the binary tree algorithm to have a logarithmic dependence on P in the t dependent term of the execution time. For large P and medium k the improvement over both simple algorithms approaches a factor two (3/2 for the simplex model). For some powerful network topologies, somewhat better algorithms are known. For Hypercubes, there is an elegant and fast algorithm which runs in time ∗ = k(1 + t log(P )/k)2 = k(1 + O( t log(P )/k)) + O(t log P )) [1, 4]. HowTHC ever, no similarly good algorithm was known for networks with low bisection4 bandwidth, e.g., meshes. Even for fully connected networks the best known algorithms for arbitrary P are quite complicated [2, 8]. The fractional tree algorithm does not have this problem. In Sec. 4 we explain how it can be adapted to several sparse topologies like hierarchical networks and meshes. 2
3 4
A couple of simplex PUs emulate each communication of a duplex PU in two substeps. In the first substep one partner acts as a sender and the other as a receiver for communicating with other couples. In the second substep the previously received data is forwarded to the partner. Throughout this paper log x stands for log 2 x. The bisection width of a network is the smallest number of connections one needs to cut in order to produce two disconnected components of size P/2 and P/2.
920
2
Peter Sanders and Jop F. Sibeyn
Basic Results on Broadcasting Long Messages
Lower Bounds. All non-source PUs must receive the k data elements, and the whole broadcasting takes at least log P steps. Thus in the duplex model there is a lower bound of (4) Tlower = k + t · log P . In the simplex model all non-source PUs must receive the k data elements. Hence the communication volume is at least (P − 1) · k. Even if all PUs are communicating all the time this implies a time bound of 2(1 − 1/P )k. In the full paper we additionally exploit that that it takes time until PUs can start to send useful data and show a bound of Tlower, simplex 2 · (1 − 1/P ) · k + t · (log P − 4).
(5)
These lower bounds hold in full generality. For a large and natural class of algorithms, we can prove a stronger bound though. Consider algorithms that divide the total data set of size k in s packets of size k/s each. All PUs operate synchronously, and in every step they send or receive at most one packet. So, until step s − 1, there is still at least one packet known only to the source. Thus, for given s, at least s − 1 + log P steps are required in the duplex model. Each step takes k/s + t time. For given k, t and P , the minimum is assumed for s = k · t/ log P : 2 ∗ = k 1 + t log(P )/k . (6) Tlower Two Simple Pipelined Algorithms. For k t, a central idea for fast broadcasting is to chop the message into s packets of size k/s and to forward these packets in a pipelined fashion. The simplest pipelined algorithm arranges all PUs into a chain of length P − 1. The head of the chain feeds packets downward. Interior PUs receive one packet in the first step and then in each step receive the next packet and forward the previously received packet. Fig. 1-d gives an s := (P − 2 + example. It is easy to see that one getsan execution time of T∞ k s s) · (t + s ). The optimal choice for s is k(P − 2)/t. Substituting this into T∞ 2 ∗ yields T∞ := k 1 + t · (P − 2)/k = k 1 + O tP/k + O(tP ) . For k tP the performance of this algorithm is quite close to the lower bound (4). However, since t is usually a large constant, on systems with large P we only get good performance for messages which are extremely large. We can reduce the dependence on P by arranging the PUs into a binary tree. Now every interior node forwards every packet to both successors. This needs two steps per packet. The execution time is T1s := (d + 2s) · (t + ks ) where d is the time step just before the last leaf receives the first packet; d is defined by the recurrence Pi = 1 + Pi−1 + Pi−2 , P0 = 1, P1 = 2. We have d = min {i : Pi ≥ P } − 1 ≈ log1.62 P . For our purposes it is sufficient to note that d = O(log P ). Fig. 1-a shows the tree with P5 = 18 PUs. Choosing s = 2k · d/t, one gets T1∗ :=
A Bandwidth Latency Tradeoff for Broadcast and Reduction
921
√ 2 k 2 + d · t/k = k 2 + O( t log(P )/k) + O(t log P ) . (For the simplex model replace the two by a three.) For small and medium k this is much better than the chain algorithm, yet for large k it is almost two times slower.
3
Fractional Tree Broadcasting
The starting point for this paper was the question whether we could find a communication pattern which allows a more flexible tradeoff between the high bandwidth of a chain (i.e., a tree with degree one) and the low latency of a binary tree. We give a family of communication pattern we call fractional trees which have this property. Here we describe the algorithm for the duplex model in detail. As already outlined in the introduction, the duplex algorithm can be translated into a simplex algorithm running in double time. In the full paper, we explain a faster direct implementation which is able to forward a run of r 1 times slower than on packets in 2r + 1 steps and hence is only a factor 2 − r+1 the duplex model. It turns out to be nontrivial to translate a parallel send|recv into a sequences “send, recv” or “recv, send” such that no delays or deadlocks occur. The idea for fractional trees is to replace the node of a binary tree by a group of r PUs forming a chain. The input is fed into the head of this chain. The data is passed down the chain and on to a successor group as in the single chain algorithm. In addition, the PUs of the group cooperate in feeding the data to the head of a second successor group. Fig. 1 shows the structure of a group and several examples.
step 0
r=1
r=2
r=3
r=
1
1
2 r+1 2.. .. r+2 . . r1 r ... r 2r
step 1 step 2 step 3 step 4 step 5
0
a
b
c
d
e
Fig. 1. Examples for fractional trees with r ∈ {1, 2, 3, ∞} where the last PU receives its first packet after 5 steps. The case r = 1 corresponds to plain binary trees and pipelines can be considered the case r = ∞. Part e) shows the communication pattern in a group of r PUs which cooperate to supply two successor nodes with all the data they have to receive. Edges are labeled with the first time step when they are active.
922
Peter Sanders and Jop F. Sibeyn
Procedure broadcastFT(r, s, 0 ≤ i < r:Integer; var D[0..s − 1]:Packet) recv(D[0]) – – wait for first packet pipeDown(r, 0, D) – – First phase for k := r to s − r step r do – – Remaining phases sendRight|Recv(D[k − r + i], D[k]) pipeDown(r, k, D) sendRight(D[s − r + i]) (* send down packets D[k..k + r − 1] and receive packets D[k + 1..k + r − 1] *) Procedure pipeDown(r, k:Integer; var D[..]:Packet) for j := k to k + r − 2 do sendDown|Recv(D[j], D[j + 1]) sendDown(D[k + r − 1])
Fig. 2. Pseudocode executed on each PU for fractional tree broadcasting, where i is the index of the PU within its group, s is a multiple of r, and the array D is the input on the root and the output on the other PUs. For the root PU, receiving is a no-op. For the top PU of a group receiving means receiving from any PU in the predecessor group. For the other PUs it means receiving from the predecessor in the group. Sending down means sending to the next PU in the group respectively sending to the top PU of the successor group. Sending right means sending to the top PU of the right successor group. If the successor defined by this convention does not exist, sending is a no-op.
All PUs execute the same code shown in Fig. 2. All timing considerations are naturally handled by the synchronization implicit in synchronous point-to-point communication. The input is conceptually subdivided into s/r runs of r packets each. The only nontrivial point is that the i-th member of a group is responsible for passing the i-th packet of every run of r packets to the right. The effect is that every r+1 steps the head of the right successor gets a run of r packets in the right order. The pause after this run is used to pass the last packet downward. Packets are passed right while the next run arrives. As in the special case of binary trees (r = 1), the right successors receive data one step later than the downward successors. Therefore, optimal tree layouts are somewhat skewed. The number of nodes reachable within d + 1 steps is governed by the recurrence Pi = r + Pi−r + Pi−r−1 (Pi = i + 1 for i ≤ r) so that d = min {i : Pi ≥ P } − 1. This implies d = O(r log(P/r)). Using this recurrence each processor can find its place in the tree in time O(d) and without any communication. Performance Analysis. Having established a smooth timing of the algorithm the analysis can proceed analogously to that of the simple algorithms from the introduction. Every communication step takes time (t + k/s) and d + s · (1 + 1/r) steps are needed until all s/r runs have reached the last leaf group. We get a total time of 1 k Trs := d + s 1 + t+ . (7) r s
A Bandwidth Latency Tradeoff for Broadcast and Reduction
923
Using calculus one gets s = kdr/(t(r + 1)) as an optimal choice for the number of packets. Substituting this into Eq. (7) yields
drt 2 rt log P 1 1 ∗ ) + O(rt log P ). (8) Tr := k 1 + r 1+ k(r+1) = k 1+ r +O( k Since d depends on r in a complicated way, there seems to be no closed form formula for an optimal r. But we get a close to optimal value for r by setting d = d ·(r +1) and ignoring that d depends on r.5 We get r ≈ (k(r +1)/(d·t))1/3 . After rounding, these values make sense for k(r + 1) ≥ d · t. For smaller k one should use r = 1 or even a non-pipelined algorithm. Substituting r and s into Trs we get a broadcasting algorithm with execution time T∗∗
≤ k 1+
d·t k(r + 1)
1/3 3
=k 1+O
t log P k
1/3 + O(t log P ) .
For k t log P algorithm performs quite close to the lower bound (4). Performance Examples. How does the algorithm compare to the two simple algorithms? For example, for P = 1024 and k/t = 4096 we choose d = d/(r + 1) ≈ log P − 1 = 9 and get r ≈ (k/(d · t))1/3 ≈ 8. This yields d = 57 and we get s = 4096 · 57 · 8/9 ≈ 456. With these values Trs ≈ 1.389k. These choices are quite robust. For example, a better approximation of the optimal r yields r = 10 and s = 503 but the resulting Trs ≈ 1.387k is less than 0.2 % better. Fig. 3 plots the achievable speedup for three different machine sizes. Even for a medium size parallel computer (P = 64) an improvement of up to a factor 1.29 can occur. For very large machines (P = 16384) the improvements reach up to factor of 1.8 and a significant improvement is observed over a large range of message lengths. Our conclusion is that fractional tree broadcasting yields a small improvement for “everyday parallel computing” yet is a significant contribution to the difficult task of exploiting high end machines such as the ones currently build in the ASCI program. For example, Compaq plans to achieve 100TFlops with 16384 Alpha processors by the year 2004 [3].
4
Sparse Interconnection Networks
Hierarchies of Crossbars. Compaq’s above mentioned 16384 PU system is expected to consist of 256 SMP modules with 64 PUs each. We view it as unlikely that it will get an interconnection network with enough bisection width to efficiently implement the hypercube algorithm. Rather, each module will only have a limited number of channels to other modules. We call such a system a 256 × 64 hierarchical crossbar. Systems with similar properties are currently build by several companies. 5
In the program one can efficiently solve the equations numerically, e.g., using golden section search [7].
924
Peter Sanders and Jop F. Sibeyn 1.8
P=64 P=1024 P=16384
improvement min(T*1,T*∞)/T**
1.7 1.6 1.5 1.4 1.3 1.2 1.1 1
10
100
1000
k/t
10000
100000
1e+06
Fig. 3. Improvement of fractional tree broadcasting over the best of pipelined binary tree and sequential pipeline algorithm as a function of k/t.
We now explain how a fractional tree with group size r can be embedded into an a × b hierarchical crossbar if b ≥ r and if each module supports at least two incoming and outcoming channels to arbitrary other modules. A generalization to more than two levels of hierarchy is also possible. First, one group in each module is connected to form a global binary tree with a nodes. Next, the b − r remaining PUs in each module are connected to a form local fractional tree. What remains to be done is to connect the local trees by the global tree. Groups in the global tree with degree one can directly link with their local tree. Leaf groups in the global tree use one of their free links to connect to their local tree. The remaining free links are used to connect to the local trees of modules with a group in the global tree of degree two. There will be one remaining unused link which can be used to further optimize the structure. By accepting an additional depth of (r + 1) log b, we can work with one less connection per module: Use two global groups per module. The first one links to the second one and one other module. The second one links to the local tree and possibly to one other module. Meshes. We only outline a simple case. Generalizations which are sufficient in practice should be relatively easy. A completely general treatment might turn out to be rather complicated. Assume we are given an a × b mesh and r = r1 · r2 such that a/r1 = b/r2 is a power of two. We partition the mesh into submeshes of size r1 × r2 each forming a group of the fractional tree. Now we can embed the binary tree of groups exploiting the substantial work on embedding binary trees into meshes (e.g., [10, 6]). Inside the group, the PUs are arranged in a snakelike fashion. In this way one gets an embedding with constant edge congestion. Often
A Bandwidth Latency Tradeoff for Broadcast and Reduction
925
it is even possible to achieve edge congestion one for bidirectional meshes. Fig. 4 gives an example where it is exploited that H-trees yield a complete binary tree with one leaf in every 2 × 2 submesh.
Fig. 4. Embedding of a fractional tree with r = 2 into an 8 × 16 mesh. Broadcasting on it gives edge congestion one even with x-y-routing.
References [1] V. Bala, J. Bruck, R. Cypher, P. Elustondo, A. Ho, C. Ho, S. Kipnis, and M. Snir. CCL: A portable and tunable collective communication library for scalable parallel computers. IEEE Transactions on Parallel and Distributed Systems, 6(2):154–164, 1995. [2] A. Bar-Noy and S. Kipnis. Broadcasting multiple messages in simultaneous send/receive systems. In 5th IEEE Symp. Parallel, Distributed Processing, pages 344–347, 1993. [3] Compaq. AlphaServer SC series product brochure, 1999. http://www.digital.com/hpc/news/news_sc_launch.html. [4] S. L. Johnsson and C. T. Ho. Optimum broadcasting and personalized communication in hypercubes. IEEE Transactions on Computers, 38(9):1249–1268, 1989. [5] V. Kumar, A. Grama, A. Gupta, and G. Karypis. Introduction to Parallel Computing. Design and Analysis of Algorithms. Benjamin/Cummings, 1994. [6] J. Opatrny and D. Sotteau. Embeddings of complete binary trees into grids and extended grids with total vertex-congestion 1. Discrete Applied Mathematics, 98:237–254, 2000. [7] W. H. Press, S. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C. Cambridge University Press, 2. edition, 1992. [8] Santos. Optimal and near-optimal algorithms for k-item broadcast. JPDC: Journal of Parallel and Distributed Computing, 57:121–139, 1999. [9] M. Snir, S. W. Otto, S. Huss-Lederman, D. W. Walker, and J. Dongarra. MPI – the Complete Reference. MIT Press, 1996.
926
Peter Sanders and Jop F. Sibeyn
[10] P. Zienicke. Embedding of treelike graphs into 2-dimensional meshes. In Graph Theoretic Concepts in Computer Science, volume 484 of LNCS, pages 182–192. Springer, 1990.
Optimal Broadcasting in Even Tori with Dynamic Faults Stefan Dobrev and Imrich Vrt’o Institute of Mathematics, Slovak Academy of Sciences Department of Informatics, P.O.Box 56, 840 00 Bratislava, Slovak Republic {kaifdobr,vrto}@savba.sk
Abstract. We consider a broadcasting problem in the n-dimensional k-ary even torus in the shouting communication mode, i.e. any node of a network can inform all its neighbours in one time step. In addition, during any time step a number of links of the network can be faulty. Moreover the faults are dynamic. The problem is to determine the minimum broadcasting time if at most 2n−1 faults are allowed in any step. In [3], it was shown that the broadcasting time is at most diameter+O(1), provided that k is limited by a polynomial of n. In our paper we drop this additional assumption and prove that the broadcasting can be always done in time diameter +2. The bound is the best possible.
1
Introduction
Broadcasting is the standard communication problem in interconnection networks when a node has to send a message to all other nodes. There are many applications of the broadcasting problem in parallel and distributed computing [6,8,9]. Recently, a lot of attention has been paid to fault-tolerant dissemination of information in networks [10]. In this paper we consider the shouting communication mode in which any node can inform all its neighbours in one time step. In addition, we assume that during any time step a number of links of the network can be faulty. This model was introduced by Santoro and Widmayer [11]. The problem is to determine the minimum broadcasting time if at most f faults are allowed in any time step, where f stands for the edge connectivity minus 1 of the network. For the hypercube, the problem was studied in [3,7,10] and completely solved in [5]. For the n-dimensional general torus, Chlebus, Diks and Pelc [2] proved an upper bound on the brodcasting time O(diameter). De Marco and Rescigno [4] showed for the n-dimensional k-ary even torus that the broadcasting time is at most diameter+O(1), provided that k is limited by a polynomial of n. In our paper we drop this additional assumption and prove that the broadcasting can be always done in time diameter +2. The bound is the best possible. Our method previously used in [5] is related to the isoperimetric problem in graphs and can be applied to other networks.
Supported by the VEGA grant No. 2/7007/20.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 927–930, 2000. c Springer-Verlag Berlin Heidelberg 2000
928
2
Stefan Dobrev and Imrich Vrt’o
Model and Basic Facts
Let Cn,k be a network of processors connected as the n-dimensional k-ary torus defined as the cartesian product of n cycles Ck , for k even. The network Cn,k has k n nodes, is regular of degree 2n, with edge connectivity 2n − 1. Its diameter equals nk/2. The links are bidirectional. The computation is synchronous. In one time step a node is able to send its message to all its neighbours. This is called the shouting mode. In each time step at most 2n − 1 links are faulty, i.e., the message transmitted along the faulty link is not delivered. The faults are dynamic in the sense that the set of faulty links can change during the execution of the broadcast. Initially, a node of Cn,k knows a message. This message needs to be sent to all other nodes. Our problem is to determine the minimum time to broadcast the message in the torus Cn,k , provided that the faults are distributed in the worst possible manner. The vertices of Cn,k are {−k/2, −k/2 + 1, ..., k/2 − 1}, and two vertices u and v are adjacent if |u − v| (mod k) = 1. The vertices of Cn,k are n-tuples (x1 , x2 , ..., xn ), where −k/2 ≤ xi ≤ k/2 − 1, for i = 1, 2, ..., n. Define o = (0, 0, ..., 0). Let d(u, v) be the distance between u and v in Cn,k . Denote S(r) = {v ∈ Cn,k |d(v, o) = r} and V (r) = {v ∈ Tn,k |d(v, o) ≤ r} = ∪ri=0 S(r). Clearly, |S(r)| is the number of integer solutions of the equation |x1 |+ |x2 |+ ...+ |xn | = r, where −k/2 ≤ xr ≤ k/2 − 1. Since k is even we have |S(r)| = |S(nk/2 − r)|, which implies |V (r)| + |V (nk/2 − r − 1)| = k n . Moreover, |S(0)| = 1, |S(1)| = 2n and |S(2)| = 2n2 , for k ≥ 6. Let A be a subset of vertices of Cn,k . Let ∂(A) denote the set of all vertices as and Leader [1] proved: in distance at most 1 from A in Cn,k . Bollob´ Lemma 1. Let Cn,k be the n-dimensional k-ary torus with k even, and let A be a nonempty subset of vertices of Cn,k . If |A| = |V (r)| + α|S(r + 1)|, for some r and 0 ≤ α < 1, then |∂(A)| ≥ |V (r + 1)| + α|S(r + 2)|.
3
Optimal Upper Bound on the Broadcasting Time
In this Section we use Lemma 1 to bound the broadcasting time. First we state a technical lemma, proof of which will appear in the full version. Lemma 2. Denote
nk
4 4n 3 . X =1− − n |S(r)| r=3
The value X is nonnegative for k ≥ 6 and n ≥ k + 4 or k ≥ n ≥ 10. Theorem 1. Assume that k ≥ 6 and n ≥ k + 4 or k ≥ n ≥ 10. The minimum broadcasting time T in the n-dimensional k-ary torus, for k even, with 2n − 1 dynamic link faults satisfies T ≤ nk/2 + 2. The bound is the best possible.
Optimal Broadcasting in Even Tori with Dynamic Faults
929
Proof. The broadcasting scheme is simple. Initially, the node o contains the message to be disseminated. In each time step each node sends the message to all its neighbours. The analysis follows. By Am we will denote suitable sets of nodes which know the message after the m-th time step. Observe that the number of nodes that know the message after the m-th step is at least |∂(Am−1 )|− (2n− 1). Clearly, there exist sets A0 and A1 and A2 s.t. |A0 | = 1, |A1 | = 2 and |A2 | = 1 + 2n = |V (1)|. According to Lemma 1 |∂(A2 )| ≥ V (2)|. Because of the 2n − 1 faulty links, we have |∂(A2 )| − (2n − 1) ≥ 2n2 + 2 = |V (1)| + α3 |S(2)|, where α3 = 1 − 1/n. Define A3 to be a subset of nodes that know the message after the 3-rd step and satisfies |A3 | = |V (1)| + α3 |S(2)|. Similarly, |∂(A3 )| ≥ |V (2)| + α3 |S(3)|. Because of the 2n − 1 faulty links, we have |∂(A3 )| − (2n − 1) = |V (2)| + α3 |S(3)| − (2n − 1) ≥ |V (2)| + α4 |S(3)|, 2n where α4 = α3 − |S(3)| . Define A4 to be a subset of nodes that know the message after the 4-th step and satisfies |A4 | = |V (2)| + α4 |S(3|. We prove by induction that for 4 ≤ m ≤ nk 2 −1
|Am | = |V (m − 2)| + αm |S(m − 1)|, where αm = 1 −
2n 2n 2n 1 − − − ... − . n |S(3)| |S(4)| |S(m − 1)|
Assume the claim holds for some 4 ≤ m − 1 ≤
nk 2
− 2, i.e.
|Am−1 | = |V (m − 3)| + αm−1 |S(m − 2)|, where αm−1 = 1 −
2n 2n 2n 1 − − − ... − . n |S(3)| |S(4)| |S(m − 2)|
Lemma 1 implies |∂(Am−1 )| − (2n − 1) = |V (m − 2)| + αm−1 |S(m − 1)| − (2n − 1) ≥ |V (m − 2)| + αm |S(m − 1)|, 2n where αm = αm−1 − |S(m−1)| . Define Am to be a subset of nodes that know the message after the m-th step and satisfies |Am | = |V (m − 2)| + αm |S(m − 1)|. Then m−1 2n 1 . αm = 1 − − n |S(i)| i=3
Now we use a dual argument. By Bm we will denote suitable subsets of nodes, which do not know the message after the m-th step. Assume that after
930
Stefan Dobrev and Imrich Vrt’o
the (nk/2 + 2)-nd step there exists at least one node which does not know the message. Observe that the number of nodes that do not know the message after the (m − 1)-st step is at least |∂(Bm )| − (2n − 1). Clearly there exist sets B nk +2 , 2 B nk +1 and B nk such that |B nk +2 | = 1, |B nk +1 | = 2 and |B nk | = 1 + 2n. 2 2 2 2 2 Similarly as for A3 we determine |B nk −1 | = 2n2 + 2. 2 Now compute nk
2 −2 1 nk 2n nk − 3)| + (1 − − )|S( − 2)|+ 2n2 +2 |A nk −1 | + |B nk −1 | = |V ( 2 2 2 n |S(r)| 2 r=3 nk
4 4n 3 n 2 ≥ k + 1 + 2n (1 − − ). n |S(r)| r=3
By Lemma 2 the expression in the brackets is positive which implies a contradiction. Finally, we show that there is a distribution of faults in each step which forces the brodcasting time nk/2 + 2. Consider vertices u, v such that d(u, v) = nk/2. Let u (v ) be a neighbor of u (v). Initially, let u knows the message. In the first step we place faults on all edges adjacent to u , except for u u. From now on we place faults on all edges adjacent to v , except for v v. After nk/2 + 1 steps, the message reaches the vertex v and one additional step is necessary to complete the broadcasting.
References 1. Bollob´ as, B., Leader, I., An isoperimetric inequality on the discrete torus, SIAM J. on Discrete Mathematics 3 (1990), 32-37. 2. Chlebus, B., Diks, K., Pelc, A., Broadcasting in synchronous networks with dynamic faults, Networks 27 (1996), 309-318. 3. De Marco, G., Vaccaro, U., Broadcasting in hypercubes and star graphs with dynamic faults, Information Processing Letters 66 (1998), 321-326. 4. De Marco, G., Rescigno, A.A., Tighter bounds on broadcasting in torus networks in presence of dynamic faults, Parallel Processing Letters, to appear. 5. Dobrev, S., Vrt’o, I., Optimal broadcasting in hypercubes with dynamic faults, Information Processing Letters 71 (1999), 81-85. 6. Fraigniaud, P., Lazard, E., Methods and problems of communication in usual networks, Discrete Applied Mathematics 53 (1994), 79-133. 7. Fraigniaud, P., Peyrat, C., Broadcasting in a hypercube when some calls fail, Information Processing Letters 39 (1991), 115-119. 8. Hedetniemi, S.M., Hedetniemi, S.T., and Liestman, A., A survey of gossiping and broadcasting in communication networks, Networks 18 (1986), 319-349. 9. Hromkovic, J., Klasing, R., Monien, B., Paine, R., Dissemination of information in interconnection networks (broadcasting and gossiping), in: Combinatorial Network Theory, (D.-Z. Du, D. F. Hsu, eds.), Kluwer Academic Publishers, 1995, 125-212. 10. Pelc, A., Fault tolerant broadcasting an gossiping in communication networks, Networks 26 (1996), 143-156. 11. Santoro, N., Widmayer, P., Distributed function evaluation in the presence of transmission faults, in: SIGAL’90, LNCS 450, Springer Verlag, Berlin, 1990, 358-369.
Broadcasting in All-Port Wormhole 3-D Meshes of Trees Petr Salinger and Pavel Tvrd´ık Department of Computer Science and Engineering Czech Technical University, Karlovo n´ am. 13 121 35 Prague, Czech Republic {salinger,tvrdik}@sun.felk.cvut.cz
Abstract. In this paper, we show that 3-D meshes of trees allow an elegant and optimal one-to-all broadcast algorithm supposing routers with distance-insensitive switching and all-output-port capability.
1
Preliminaries
Let B = {0, 1} be the binary alphabet and let Bn = {xn−1 . . . x0 ; xi ∈ B}, n ≥ 1, be the set of all n-bit strings. For integer i ≥ 1, the i-fold concatenation of bit α is denoted by αi . The empty string is α0 = ε. Let B0 = {ε}. If x ∈ Bi , then |x| = i denotes its length. An undirected graph G consists of nodes N (G) and edges E(G). An edge joining 2 adjacent nodes u and v is denoted by u, v. Given integer n ≥ 1, the complete binary tree of height n, CBT n , is defined as follows: N (CBT n ) = ∪ni=0 Bi and E(CBT n ) = {x, xa ; |x| < n, a ∈ B}. We assume the standard tree node labeling (see Figure 1(a)). The 3-D mesh of trees of height n, M3T n , is defined as follows: N (M3T n ) = (Bn × Bn × ∪ni=0 Bi ) ∪ (Bn × ∪ni=0 Bi × Bn ) ∪ (∪ni=0 Bi × Bn × Bn ), E(M3T n ) = {(x, y, z), (xa, y, z) ; |x| < n, |y| = |z| = n, a ∈ B} ∪ {(x, y, z), (x, ya, z) ; |y| < n, |x| = |z| = n, a ∈ B} ∪ {(x, y, z), (x, y, za) ; |x| = |y| = n, |z| < n, a ∈ B}. Hence, |N (M3T n )| = 4 · 23n − 3 · 22n and |E(M3T n )| = 6 · 23n − 6 · 22n . Also, M3T n ⊂ CBT n × CBT n × CBT n . M3T n can be viewed as a 3-D mesh 2n × 2n × 2n whose each node is a leaf of exactly 1 x-tree, 1 y-tree, and 1 z-tree (see Figure 1(c)). M3T n can be canonically decomposed into 23i 3-D submeshes of trees, denoted by subM3T n−i (x, y, z), where x, y, z ∈ Bi . The 2-D mesh of trees (M2T n ) is defined similarly (see Figure 1(b)). Given a connected network G and a designated node s possessing a packet, the task of a one-to-all broadcast (OAB) in G with the source s is to deliver the packet from s to every other node in G. We assume that each node has a router capable of distance-insensitive switching, such as wormhole or circuit switching. We denote such networks as WH networks. In each round, all paths used by
ˇ This research has been supported by MSMT research program #J04/98:212300014, ˇ ˇ grant #580/2000. by CVUT grant #300009203, and by FRVS
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 931–934, 2000. c Springer-Verlag Berlin Heidelberg 2000
932
Petr Salinger and Pavel Tvrd´ık ε 0
1
00
01
10
(00,00,11)
11
(0,00,11)
(ε ,00,11)
(1,00,11) (11,00,11)
(00,00,1) 000
001
010 011
100 101
110
(00,00,ε )
111
(a) (ε ,00)
x
(0,00)
(00,00) (00,0)
y
ε
(00,00,0)
z
(1,00) (01,00)
(10,00)
(11,00)
(00,00,00)
y x
(00,0,00)
(00,01)
(01,01)
(10,01)
(11,01) (00,ε ,00)
(00,ε )
(00,10)
(01,10)
(10,10)
(11,10) (00,1,00)
(00,1)
(00,11)
(01,11)
(10,11)
(11,11)
(11,00,00)
(b)
(11,01,00)
(11,10,00)
(11,11,00)
(c)
Fig. 1. Node labeling in (a) CBT 3 , (b) M2T 2 , and (c) M3T 2 . copies of the packet are pairwise link-disjoint. Links can be half- or full-duplex . We assume all-output-port capability, i.e., in one round a router can inject copies of the packet from the node into the network via all output links simultaneously.
2
Previous and Related Work
Several OAB algorithms have been developed for all-port WH 2-D meshes and tori, e.g., [2,5]. 2-D meshes of trees have been studied in [1,3,4]. Paper [1] gives algorithms for OAB and AAB in 1-port store-and-forward full-duplex combining model. An optimal OAB algorithm and asymptotically optimal algorithms for AAB, in 1-port WH node-disjoint and both half-duplex and full-duplex combining models are presented in [3]. An OAB algorithm in all-output-port WH model is described in [4]. It achieves the optimal number of rounds if the source of the OAB is a mesh node, level-1 tree node, or a root. This represents 67% of nodes. In the remaining cases, it needs 1 additional round.
3
The Main Result Algorithm OAB(n, sj ): if Even(n) then { if j = n then /* sj is not a mesh node */ sj sends the packet to (0n , 0n , 0n ); apply A(n); apply C(n); } else { /* n is odd */ apply B(n, sj ); for all x, y, z ∈ B do in parallel apply A(n − 1) in subM3T n−1 (x, y, z); /* in all 8 submeshes */ apply C(n); };
Broadcasting in All-Port Wormhole 3-D Meshes of Trees
933
The OAB algorithm uses the standard shortest-path routing. Without loss of generality, we assume source node sj = (0j , 0n , 0n ). Procedure A(n). Assumptions: n is even and the source is mesh node s = (0n , 0n , 0n ). In the first three rounds of procedure A(n), 1 mesh node in each of 64 subM3T n−2 (x, y, z), x, y, z ∈ B2 is informed. In further 3 rounds, each of these 64 nodes informs other 64 nodes in its 64 subM3T n−4 ’s, and so on, up to the granularity of individual mesh nodes. Hence, after completion of procedure A(n), all the mesh nodes are informed. procedure A(n): for i = n − 2 downto 0 by 2 do for all x, y, z ∈ Bn−2−i do in parallel { (x000i , y000i , z000i ) sends the packet to (x110i , y110i , z110i ) via (x110i , y, z000i ) and to (x100i , y100i , z100i ) via (x000i , y100i , z) and to (x010i , y010i , z010i ) via (x0, y000i , z010i ); for all u, v ∈ B do in parallel (xuv0i , yuv0i , zuv0i ) sends the packet to (x¯ uv0i , yuv0i , zuv0i ) and to (xuv0i , y u ¯v0i , zuv0i ) and to (xuv0i , yuv0i , z u ¯ v0i ); for all u, v, w, t ∈ B do in parallel (xut0i , yvt0i , zwt0i ) sends the packet to (xut¯0i , yvt0i , zwt0i ) and to (xut0i , yv t¯0i , zwt0i ) and to (xut0i , yvt0i , zw t¯0i );};
sj 00 y
00 00
01
01
11
y
11
00 00 y
10
11
11
11
z= 00 x 01 10
11
00 00
01
y
z= 01 x 01 10
11
01
00 00 y
10
10
11
11
11
x 01 10
z= 01 11
00 00
01
y
x 01 10
01
11
00 00
y
y
10
11
11
11
x 01 10
11
z= 11 x 01 10
11
01 10 11
z= 10 x 01 10
11
00 00 y
01 10 11
x 01 10
01
10
z= 01
00 00
z= 10
10
z= 00
11
01
10
00 00
x 01 10
01
10
z= 00
y
x 01 10
10
00 00 y
x 01 10
00
z= 11 11
00 00 y
x 01 10
11
01 10 11
z= 10
z= 11
Procedure B(n, sj ). The source node is sj = (0j , 0n , 0n ), 0 ≤ j ≤ n. Procedure B(n, sj ) performs 2 rounds and informs 1 mesh node in each subM3T n−1 (x, y, z), x, y, z ∈ B. procedure B(n, sj ): /* Round 1 */ sj sends the packet to (00n−1 , 10n−1 , 10n−1 ) and to (10n−1 , 10n−1 , 10n−1 ); /* Round 2 */ parbegin sj sends the packet to (00n−1 , 00n−1 , 00n−1 ) and to (10n−1 , 00n−1 , 00n−1 ); (00n−1 , 10n−1 , 10n−1 ) sends the packet to (00n−1 , 10n−1 , 00n−1 ) and to (00n−1 , 00n−1 , 10n−1 ); (10n−1 , 10n−1 , 10n−1 ) sends the packet to (10n−1 , 10n−1 , 00n−1 ) and to (10n−1 , 00n−1 , 10n−1 ); parend;
sj
x
y
z= 0
z= 1
x
y
z= 0
z= 1
i Definition 1. For any a ∈ ∪∞ i=1 B and b ∈ B, define function κ(ab) = del trailing(a, b), where del trailing(a , b) if a = a b, del trailing(a, b) = a otherwise;
934
Petr Salinger and Pavel Tvrd´ık
Procedure C(n). It is applied in the situation where all the mesh nodes have been informed. Since each mesh node is a leaf of exactly 1 x-tree, 1 y-tree, and 1 z-tree, all remaining tree nodes in all x-, y-, and z-trees are informed in a single round. Since the function κ() is injective on nonzero strings with length n, every internal tree node w = (x, y, z), |x| < n, |y| = |z| = n, of an x-tree, is informed by exactly one mesh node. v = (κ−1 (x), y, z). Similarly, for y-trees and z-trees. ε procedure C(n): for all x, y, z ∈ Bn do in parallel (x, y, z) sends the packet to (κ(x), y, z) if x =n0 and to (x, κ(y), z) if y =n0 and to (x, y, κ(z)) if z =n0;
0
1
00
000
01
001
010
011
10
100
11
101
110
111
Lemma 2. Let b2 (n) (b3 (n)) denote the lower bound on the number of rounds of an all-output-port OAB in M3T n if the source has degree 2 (3, resp.). Then b3 (n) = b2 (n) = 32 (n + 1) for odd n, b3 (n) = 32 n + 1 and b2 (n) = b3 (n) + 1 = 32 n + 2 for even n. Theorem 3. Algorithm OAB(n, s) performs the OAB in all-output-port WH M3T n using the shortest-path routing. Its number of rounds equals the lowerbound if n is odd or if n is even and sj is a mesh node or tree root. In the remaining cases, it needs one additional round. We have proposed a simple OAB algorithm for all-output-port WH 3-D meshes of trees. Depending on the parity of n and the position of the source node, its complexity matches the lower bound or one more round is needed. It is interesting to compare this algorithm with the OAB algorithm in all-output-port WH 2-D meshes of trees [4]. The 2-D case is more complicated. It is an interesting research problem to generalize this result in several ways: (1) to rectangular 3-D meshes of trees; (2) to k-ary 3-D meshes of trees, k ≥ 3; (3) to h-dimensional meshes of trees, h ≥ 4.
References 1. D. Barth. An algorithm of broadcasting in the mesh of trees. In CONPAR 92– VAPP, volume 634 of LNCS, pages 843–844. Springer-Verlag, 1992. 2. J.-Y. L. Park and H.-A. Choi. Circuit-switched broadcasting in torus and mesh networks. IEEE Trans. on Parallel and Distributed Systems, 7(2):184–190, 1996. 3. P. Salinger and P. Tvrd´ık. Optimal broadcasting and gossiping in one-port wormhole meshes of trees. In PDCS’99, volume 2, pages 713–718. Acta Press, 1999. 4. P. Salinger and P. Tvrd´ık. Optimal broadcasting in all-port meshes of trees with distance-insensitive routing. In IPDPS’00, pages 353–358. IEEE CS Press, 2000. 5. Y.-C. Tseng. A dilated-diagonal-based scheme for broadcast in a wormhole-routed 2D torus. IEEE Trans. on Computers, 46(8):947–952, 1997.
Probability-Based Fault-Tolerant Routing in Hypercubes Jehad Al-Sadi1, Khaled Day2, Mohamed Ould-Khaoua1 1
Computer Science Department, Strathclyde University, Glasgow, U.K. {jehad, mohamed}@cs.strath.ac.uk 2 Computer Science Department, Sultan Qaboos University, Muscat, Sultanate of Oman [email protected] Abstract. This paper describes a new fault-tolerant routing algorithm for the hypercube, using the concept of probability vectors. Each node calculates a probability vector whose kth element represents the probability that a destination node at distance k cannot be minimally reached due to a faulty node or link. A performance comparison with the recently-proposed safety vectors algorithm, through extensive simulation, shows that the new algorithm exhibits superior performance in terms of routing distances and percentage of reachability.
1 Introduction Efficient interprocessor communication is the key to good system performance in interconnection networks. As the network size scales up the probability of processor and link failure also increases. It is therefore essential to design fault-tolerant routing algorithms that allow messages to be routed between non-faulty nodes in the presence of faulty links and nodes. Several fault-tolerant routing strategies for the hypercube have been proposed in the literature. The main challenge is to devise a simple and effective way of representing limited global fault information that allows optimal or near-optimal routing. Recently, there has been a number of attempts to design limited-global-information-based algorithms for the hypercube [1], [2], [3], [4]. This paper proposes a new limited-globalinformation-based routing algorithm for the hypercube. Routing at each node A is based on a calculated probability vector P A = ( P1A ,...., PnA ) . PkA represents the probability that a destination node at distance k cannot be reached from node A using a minimal path due to a faulty node or link along the path. A performance comparison against the safety vectors algorithm through extensive simulation experiments is then presented. The results reveal that the new routing algorithm outperforms the safety vectors algorithm in terms of routing distances and percentage of reachability.
2 The Proposed Fault-Tolerant Routing Algorithm The label of a node A in n-cube is written anan-1…a1, where ai ∈ {0, 1} is the bit at ith dimension. The neighbour of a node A along the i-th dimension is denoted A (i ) . A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 935-938, 2000. Springer-Verlag Berlin Heidelberg 2000
936
Jehad Al-Sadi, Khaled Day, and Mohamed Ould-Khaoua
The Hamming distance between a node A and a node B is denoted H(A,B). A path between two nodes A and B is a minimal path if its length is equal to H(A,B). With respect to a given destination node D, a neighbour A(i ) of node A is called a preferred neighbour for the routing from A to D if the i-th bit of A ⊕ D is 1. We say in this case that i is a preferred dimension. Neighbours other than preferred neighbours are called spare neighbours. Routing through a spare neighbour increases the routing distance by two over the minimum distance. The basic idea of this algorithm is for each node A to determine its faulty set FA of faulty or unreachable neighbours, to use this unsafety set to calculate its
probability vector P A , and then perform fault-tolerant routing using P A . Definition 1: The faulty set FA of a node A is defined as FA
=
i ! fA
1< i < n
(i ) , where f Ai is given by f Ai = { A }
if A (i ) is faulty Otherwise
φ
The kth element PkA of P A denotes the probability that a destination at distance k from A is not minimally reachable from A. Since node A has FA faulty or unreachable immediate neighbours, and only one of the n edges incident from A constitutes a minimal path to a specific destination at distance one, the probability P1A is: P1A =
(1)
FA n (i )
In order to compute the other elements PkA , k≥ 2, let RkA
be the probability that a
destination at distance k from A is minimally reachable via its neighbour probability PkA , k ≥ 2 , can then be expressed as n
(i )
PkA = ∏ (1 − R kA i =1
(i )
RkA
),
where
0 if node A(i ) is faulty = k ( i ) A (1 − Pk −1 ) otherwise n
A (i ) .
The (2)
(3)
When a node A has to forward a message M towards its destination D it applies the probability-based routing algorithm PB_Routing, outlined in Fig 1. If we can route through a preferred neighbour, A (i ) , then the associated least expected routing distance is calculated as follows: Pr = h(1 − PhA−(1i ) ) + ( h + 2) PhA−(1i )
(4)
Probability-Based Fault-Tolerant Routing in Hypercubes
937
(i )
where PhA−1 is the probability that a minimal path via the preferred neighbour A (i ) to a destination at distance h is faulty. On the other hand, if we route through a spare neighbour, A ( j ) , then the least expected routing distance is calculated as follows: (5)
Sp = ( h + 2)(1 − PhA+(1j ) ) + ( h + 4) PhA+(1j ) Algorithm PB_Routing (M: message; A,D: node) /* called by node A to route the message M towards its destination node D */
if D is a reachable neighbour then deliver M to D; exit; /* destination reached */ h = Hamming distance between A and D Let
A (i )
be a reachable preferred neighbour with least PhA−(1i ) value;
Pr = h(1 − PhA−(1i ) ) + ( h + 2) PhA−(1i ) ; /* least expected distance through Let
A( j )
A(i )
*/
be a reachable spare neighbour with least PhA+(1j ) value;
Sp= ( h + 2)(1 − PhA+(1j ) ) + ( h + 4) PhA+(1j ) ; /* least expected distance through A( j ) */ if ∃ A(i ) and ( (∃ A( j ) and Pr ≤ Sp) or (∼∃ A( j ) ) ) then send M to A(i ) ; else if ∃
A( j )
and ( (∃
else failure
and Pr > Sp) or (∼ ∃ A(i ) ) ) then send M to /* destination unreachable */ A(i )
A( j ) ;
Fig. 1. Outline of the proposed probability-based fault-tolerant routing algorithm
3 Performance Comparison This section reports results from simulation experiments comparing the performance of the proposed routing algorithm to that of the safety vectors algorithm [3]. To this end, a simulation study has been conducted for both the proposed PB_Routing algorithm and safety vectors approaches over an 8-dimensional hypercube (256 nodes) with different random distributions of faulty nodes. We started with a nonfaulty hypercube. The number of faulty nodes was then increased gradually up to 75% of the hypercube size with random fault distributions. A total of 10,000 sourcedestination pairs were selected randomly during each run. Let Total be the total number of generated messages, Delivered be the number of delivered messages, and FailCount be the number of routing failure cases. We propose the following three performance measures as the basis for comparing the safety vectors and the probability vectors algorithms. - Percentage of unreachability =
Fail _ Count × 100 Total
- Average deviation from optimality
=
1 Routing _ Distance − Hamming_Distance ∑ Delivered Hamming_Distance
938
Jehad Al-Sadi, Khaled Day, and Mohamed Ould-Khaoua
% of Unreachability
P B _ R o u tin g
S a fe t y V e c t o r s
100 50 0 0
150
300
450
600
750
N o o f F a u lty N o d e s
Fig. 2. Percentage of Unreachability of PB_Routing and Safety Vectors Algorithms
Average Deviation
A R _ R o u t in g
S a fe ty V e c to rs
100 50 0 0
150
300
450
600
750
N o o f F a u lt y N o d e s
Fig. 3. Average Deviation from optimality of PB_Routing and Safety Vectors algorithms
The percentage of unreachability measures the percentage of messages that the algorithm fails to deliver to destination due to faulty components. The average deviation from optimality indicates how close the achieved routing is to the minimal distance routing. The obtained results (Fig.2, 3) reveal that PB_Routing achieves much higher reachability with low to moderate deviation from optimality.
4 Conclusion We have proposed a new probability-based fault-tolerant routing algorithm for the hypercube. Each node A calculates a numeric probability vector PA and uses it in performing fault-tolerant routing. The performance of the proposed algorithm has been compared to that of the recently proposed safety vectors algorithm through extensive simulation. The new algorithm outperforms the safety vectors algorithm in terms of reachability and deviation from optimality.
References 1. Chiu, G.M., Chen, K.-S.: Fault-tolerant routing strategy using routing capability in hypercube multicomputers, Proc. Int’l Conf. Parallel and Distributed Systems, pp. 396-403, 1996. 2. Lee, T.C.: J.P. Hayes, A fault-tolerant communication scheme for hypercube computers, IEEE Trans. Computers, vol. 41, no. 10, pp. 1,242-1,256, Oct. 1992. 3. Wu, J.: Adaptive fault-tolerant routing in cube-based multicomputers using safety vectors, IEEE Trans. Parallel and Distributed Systems, vol. 9, no. 4, pp. 321-334, April 1998. 4. Wu J., Fernandez, E.B.: Broadcasting in faulty hypercubes, Proc. 11th Symp. Reliable Distributed Systems, pp. 122-129, 1992.
Topic 14 Instruction-Level Parallelism and Processor Architecture Kemal Ebcioglu Global Chair
This year, the Euro-Par conference is being held in beautiful Munich, Germany. I am very honored to welcome you to the instruction level parallelism and processor architecture sessions of Euro-Par 2000! Instruction level parallelism (ILP) and processor architecture are important and growing fields. ILP research aims to extract very fine-grained parallelism not only from scientific code, but also from the irregular, general code that pervades applications and operating systems. Thus, successful ILP techniques can have a paramount effect on the performance of the entire computer system. ILP researchers in the academia and industry have continuously been designing leading-edge techniques to increase processor performance, such as microarchitecture enhancements, memory latency tolerance, and aggressive compiler algorithms. Samples of these techniques can be seen among the present collection of papers. This year, 29 papers were submitted to the ILP and processor architecture topic. The selection process was very competitive, and difficult for the topic organizers. At the end, 4 submissions were accepted as regular papers, and 8 were accepted as short papers. I would like to thank the other members of the organizing committee of the present topic, namely Prof. Theo Ungerer (the Local Chair), Prof. MariaGiovanna Sami, and Prof. Nader Bagherzadeh (the Vice-Chairs), who painstakingly reviewed many papers and provided insightful remarks. Prof. Ungerer took the lead in the publicity effort, set up an excellent Web-based review database for the papers, and at the end represented us during the Euro-Par program committee meeting. I am also grateful to our referees for lending us their expertise and providing rigorous reviews. I would finally like to thank the Euro-Par 2000 conference organizers for their continued support.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 939–939, 2000. c Springer-Verlag Berlin Heidelberg 2000
On the Performance of Fetch Engines Running DSS Workloads Carlos Navarro, Alex Ram´ırez, Josep-L. Larriba-Pey, and Mateo Valero Universitat Polit`ecnica de Catalunya Jordi Girona 1–3, D6 08034 Barcelona (Spain) {cnavarro, aramirez, larri, mateo}@ac.upc.es
Abstract This paper examines the behavior of current and next generation microprocessors’ fetch engines while running Decision Support Systems (DSS) workloads. We analyze the effect of the latency of instructions being fetched, their quality and the number of instructions that the fetch engine provides per access. Our study reveals that a well dimensioned fetch engine is of great importance for DSS performance, showing gains over 100% between a conventional fetch engine and a perfect one. We have found that, in many cases, the I-cache size bounds the benefits that one might expect from a better branch prediction. The second part of our study focuses on the performance benefits of a code reordering technique for the Database Management System (DBMS) that runs our DSS workload. Our results show that the reordering has a positive effect on the three parameters and can speed-up the DSS execution by 21% for a 4 issue processor, and 27% for an 8 issue one.
1
Introduction
Fetch engine performance is characterized by three different factors: the latency of instructions being fetched from memory, the number of instructions fetched per access, and the quality of those instructions. Instruction latency is caused by the speed gap between processing and memory. While the pipeline needs new instructions every cycle, the memory can only supply them at a ratio of tens or even hundreds of cycles. Reducing the I-cache misses has been addressed using software and hardware techniques. On the software side we have code reordering techniques, like [7,11]. On the hardware side we have set associative caches, hardware instruction prefetching [17], and victim caches [8], among others. The number of instructions fetched per cycle, fetch bandwidth, is a problem when more than one basic block has to be provided per cycle. Bandwidth is limited by the execution of non-contiguous instructions and the hardware width
This research has been supported by CICYT grant TIC-0511-98, the Spanish Ministry of education grant PN98 43443683-1 (Carlos Navarro), the Generalitat de Catalunya grant 1998FI-003060-26 (Alex Ramirez) and CEPBA.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 940–949, 2000. c Springer-Verlag Berlin Heidelberg 2000
On the Performance of Fetch Engines Running DSS Workloads
941
of the fetch engine. The execution of non-contiguous basic blocks has been addressed also, both in hardware and software. The hardware solutions comprise the branch address cache [20], the collapsing buffer [5], and the trace cache [10,15]. Also, in our previous work [12,13] we have addressed this problem with a software code reordering technique, the Software Trace Cache (STC), and the interaction between the software and hardware trace cache [14]. The quality of instructions is determined by branch prediction accuracy. Dynamic branch prediction schemes have been used during the last years to avoid stopping the fetch until branch resolution [16]. But the use of this prediction mechanisms also introduces wrong path instructions in the pipeline whenever a branch is mispredicted. A lot of work has been done during the last years to increase branch prediction accuracy [6,16,21]. DBMSs are highly structured codes, perform many procedure calls and have plenty of control statements to handle all types of error situations. This causes a great level of control flow activity in the code and larger instruction working sets when compared with other common integer codes[1,2]. Recent work has shown that those instruction working sets can be a problem for the instruction cache sizes used in current generation microprocessors [1]. Our results show that for DSS workloads, the fetch engine has a large influence on the IPC. In Figure 1 we compare the performance of a conventional fetch engine against the best fetch engine possible for 4 and 8 issue superscalars. This perfect fetch engine returns the full width of the next correct path instructions every cycle. That is, it does not have latency problems, it uses all the possible bandwidth, and it uses a perfect branch prediction. We can see that the perfect fetch engine shows speedups of 98% and 59% over the 16KB and 32KB engines for the 4 issue processor and of 116% and 70% for the 8 issue processor. With all this evidence, we study how DBMSs exercise the different parts of the current processors’ fetch engines, and how different improvements on the fetch engine can help their execution. We analyze the fetch impact on the overall performance, the isolated behavior of the three main fetch parameters, and their effect on performance. We also analyze the effects of a code reordering technique on each fetch engine component and its effect on performance. The rest of the paper is structured as follows. Section 2 presents our experimental setup. Sections 3, 4, and 5 study the isolated effect on performance of fetch latency, quality and bandwidth. Section 6 addresses the effects of applying a code reordering scheme to our workload. Finally we conclude in Section 7.
2
Experimental Setup
Our DSS workload runs on top of the PostgresSQL 6.3 DBMS [18]. Our workload is modeled with a session that executes queries 3 and 17 of the Transaction processing council TPC-D benchmark [19]. We set up the Alpha version of the Postgres 6.3 database on digital Unix v4.0 compiled with the -O3 flags of the Digitals’s C compiler. A TPC-D database is used with scale factor 0.1 (100MB of raw data).
Carlos Navarro et al. 3.44
942
2.66
3
2.03
IPC
4_issue 8_issue
1.67
1.34
1.59
2
16KB
32KB
Perfect
Fetch_configurations
Fig.1. Comparison between two conventional fetch engines and the perfect one for 4 and 8 issue superscalar processors. The experiment shows results for cache sizes of 16KB and 32KB. The conventional fetch engine uses a 2048 entries Bimodal predictor. The simulator used in this study is based on the sim-outorder timing simulator included in the Simplescalar-Alpha v3.0 tool set [4]. Table 1 shows the configuration we have used for our baseline architecture. The I-cache and branch prediction setups are varied across the study. Since the complete execution requires over a billion instructions, we have used sampling in order to reduce simulation times. Simulations are performed using the following sequence: 50 Million detailed simulation sample followed by a 150 Million fast simulation sample where we just emulate the ISA. We model a branch misprediction in the following way. When a branch misprediction is detected, the execution engine spends a fixed number of cycles arranging the window and other internals before re-starting the fetch in the correct path. This number of cycles is what we call misprediction recovery penalty and it is a simulation parameter. After those cycles, fetch is re-started and the issue of correct path instructions restarts 2 cycles later[9]. In this work we are using two different processor widths, 4 and 8. The fetch engine for the 4 issue processor predicts a single branch per cycle, while for the 8 issue processor it can predict a second branch if the first was not taken.
3
Effect of Instruction Latency on Performance
The first parameter we explore is the latency of the instructions being fetched from memory, and the impact that it has on the overall performance. Our analysis will be structured as follows: First we will show the perceived latency for Parameters
Values
Parameters
Values
Core Misprediction recovery penalty 1 Issue width 4,8 Memory Window size 128 Inst L1 several sizes/ 32 byte/ 2 way L/S queue size 128 Data L1 64K/ 32 byte/ 2 way FUs Unbound fully pipelined Combined L2 2MB/ 64 byte/ 2 way Branch prediction Latencies 1/ 7 / 80 Direction Several schemes Instruction TLB 16 4K entries/ 4 way Address BTB 4K entries/ 4 way Data TLB 32 4K entries/ 4way RAS 128 entries
Table1. Baseline processor
On the Performance of Fetch Engines Running DSS Workloads
943
several cache sizes, and finally we will show how those latencies affect the overall execution performance. Given that with the current technology trends our L2 latency could be quite aggressive, we will also provide results for a L2 latency of 15 cycles. This latency is more feasible for future processor clock speeds, even with integration of L2 in the processor die. For example, in [3], where a 1GHz processor with integrated 2MB L2 cache is evaluated, the latencies for L2 are larger than 15 cycles. Our metric for the perceived instruction latency, will be the Average Memory Access Time for instructions (AMATi ). This metric will show the average latency that our fetch engine perceives per instruction. AMATi is computed as: Icache miss rate ∗ L2 latency + (Icache miss rate ∗ L2 instruction miss rate) ∗ memory latency. With an AMATi equal to 1, the fetch engine would never block waiting for instructions. Figure 2(a) shows the AMATi for several I-cache sizes when we use both 7 and 15 cycles L2 latencies. We observe that with an 8KB I-cache the fetch engine must wait more than 1.5 cycles per instruction or even 2.08 cycles for a 15 cycles L2 latency. We can also see that for 15 cycles, even a 64KB I-cache shows a latency that still is far from the desired 1. Our results show that the working set starts to fit in caches with sizes of 128KB or larger. This is not strange because the instruction footprint of our queries is larger than 230KB. Figure 2(b) shows the IPC for different I-cache sizes using perfect branch prediction for 7 cycles L2 latency. The perfect I-cache shown in that Figure is used as an upper bound. Results are shown for both 4 and 8 issue processors. We can see that the decrease in performance for I-caches smaller than 128KB is important. This is due to the increase in exposed latency and is even more evident if we focus on the 8 issue processor. For a 4 issue processor and caches of 8 and 16KB it is better to double I-cache size than to double the issue width. Figure 2(c) shows results for a 15 cycles L2 latency. We can see that the slope for both 4 and 8 issue is steeper than the one of Figure 2(a). In this case, the drops in IPC are significant even for large caches. For instance, we have speedups of 18% and 23% in 4 and 8 issue processors when we go from a 64KB to a 128KB I-cache.
4
Effect of Instruction Quality on Performance
In this section we want to quantify if the branch prediction accuracy has the same importance as the I-cache performance. In order to inspect that, we have tested the behavior of several branch predictors. In particular we have used Bimodal and Gshare branch predictors of 10, 12, 14, 16, and 18 bits and Hybrid branch predictors of 12 and 14 bits. The branch prediction accuracy of those predictors is between 98.34% and 88.55%. In general, the branch prediction accuracy is good, compared to the prediction rates obtained on several SpecInt95 benchmarks [6]. Figure 3(a) shows the IPC we can expect from a 4 issue processor running the branch predictors we have tested. Results show an interesting effect. On one
Carlos Navarro et al. 2.08
944
1.51
15_cycles_L2 7_cycles_L2
8KB
16KB
32KB
64KB
128KB
1.00
1.0
1.00
1.02
1.01
1.12
1.05
1.14
1.30
1.5
1.30
AMATi
1.64
2.0
256KB
(a) 3.16
3.21
3.23 3.04
3
3
3.14
3.18
2.36
2.38
2.86
2.38
2.35
2.38
2.47
2.39
2.29
2 1.77
IPC
IPC
2.17 1.86
2
1.94
8_issue 4_issue
1.75 1.45
1.45 1.28 1.10 0.99
1.10 1
1
0.72 0.66 8KB
16KB
32KB
64KB
128KB
256KB
Perfect
8KB
16KB
32KB
64KB
128KB
I-cache_configurations
I-cache_configurations
7_cycles_L2_latency
15_cycles_L2_latency
(b)
256KB
Perfect
(c)
Fig.2. (a) AMATi in cycles for different L1 I-cache sizes and L2 latencies. Charts (b),(c): IPC for several I-cache configurations. Chart (b) shows results for 7 cycle L2 latency. Chart (c) shows results for 15 cycle L2 latency. hand, the difference in branch prediction accuracy between the worst performing branch predictor (G10) and the oracle one are clearly reflected in the IPC for the results with a perfect I-cache. That is, the quality of instructions directly enhances the overall performance. On the other hand, for the system with an 8KB I-cache, the benefits of the same increase in instruction quality are very small. We can see that the impact on IPC of the increasing branch prediction accuracy grows when we increase the I-cache size. So the perceived latency is hiding the benefits of the increase in quality. Thus, for our workloads, spending a lot of hardware in complex branch predictors is not cost effective unless AMATi is small enough. The results we have seen so far are for simulations where the misprediction recovery penalty was set to 1. This leads to small branch misprediction penalties. One question that could arise is, how the effect of latency over instruction quality is modified by an increasing branch misprediction penalty. Figure 3(b) shows results for branch misprediction recovery cycles equal to 6 leading to penalties of at least 8 cycles. In that Figure, we can see that, the effect observed in the last section still exists when we have larger branch misprediction penalties. For branch misprediction recovery of 6 cycles it is still better to duplicate an 8KB I-cache than to have a perfect branch predictor. If we have a design with a misprediction recovery time of 12 cycles (chart found at [9]), it is always better to go to the perfect branch prediction than to duplicate the cache size. However, it is interesting to note that I-cache size is still
On the Performance of Fetch Engines Running DSS Workloads 2.5
2.5 2.39
2.39 2.16
2.14
2.17
2.15
2.12
2.09
2.01
2.0 1.86
1.10
2.08
2.07
2.04
1.71
1.73
1.72
1.70
1.68
1.68
1.67
1.66
1.35
1.34
1.34
1.05
1.05
1.04
1.53 1.38
1.06
1.37
1.06
1.38
1.06
1.38
1.06
1.36
1.32
1.06
1.04
1.0 Perfect H14
H12
G18
G16
G14
G12
1.5 1.35
1.36
1.26
1.00
G10
1.99
2.04
2.01
1.95
1.86 1.68
1.64
2.03
2.0
IPC
IPC
1.45
2.09
1.85 1.73
1.5
945
1.05
B18
1.05
B16
B14
B12
B10
1.64
1.62
1.33
1.31
1.33
1.32
1.30
1.03
1.02
1.04
1.03
1.02
1.64
1.60 1.49
1.45
1.10 1.0
1.91
1.91
1.90
1.54 1.32
1.23
1.83
1.57
1.57
1.56
1.56
1.28
1.28
1.28
1.27
1.25
1.01
1.01
1.01
1.00
0.99
1.52
Perfect 32KB 16KB 8KB
1.12
Perfect H14
H12
G18
G16
G14
0.98
G12
0.92 G10
B18
B16
B14
branch_predictors
branch_predictors
1_cycle_misprediction_recovery_penalty
6_cycles_misprediction_recovery_penalty
(a)
1.89
1.78 1.65
B12
B10
(b)
Fig.3. IPC for each of the branch predictors tested. Charts (a) is for misprediction recovery penalties of 1 cycles, while chart (b) is for misprediction recovery penalties of 6 cycles. All simulations are for a 4 issue processor and L2 latency of 7 cycles. having a bounding effect on the performance benefits of better branch prediction accuracy.
5
Effect of Fetch Bandwidth on Performance
Instruction fetch bandwidth will be a major limiting factor to high performance in next generation microprocessors. Nevertheless, fetch bandwidth could be also a problem for 4 or 8 issue processors. Several experiments have been performed to characterize the impact that fetch bandwidth has on the performance. In particular we have tested the impact of increasing the fetch bandwidth for a 4 issue processor to 8 instructions per cycle. The complete analysis can be found at [9] and has not been shown here due to space limitations. Our results show that for 4 and 8 issue processors, large increases in bandwidth do not reflect in such improvements in IPC. Thus, for our workload and issue widths, increasing the fetch bandwidth is not critical. Instead, its is more important to fetch the right path instructions.
6
Code Reordering
In the previous sections we have shown the high impact that the fetch engine design can have in the overall execution performance of our DSS workload. From this evidence, it seems that for our workloads, it can be interesting to use a supplementary technique in order to help I-caches. This technique can be a code reordering scheme. Code reordering techniques have been used for a long time in order to reduce the I-cache misses [11]. In our previous work, [12,13,14] we have shown that DSS workloads are a good target for these techniques. In particular we presented a reordering technique, the Software Trace Cache (STC), aimed at increasing the
946
Carlos Navarro et al.
No_STC 1.1386
1.2063
1.0028
128KB
256KB
96.59 95.35
96.57 96.81
96.31 96.75
No_STC STC
92
90
1.0 8KB
96.57 96.81
94
88.55
1.0091
1.0497
STC 1.0524
1.2
Hit_rate
94.19
1.2989
96
96.57 96.83
97.15
97.05 97.87
98.34 98.33
95.91
1.3793
97.99 98.17
98.18 98.18
1.5066
AMATi (in cycles)
1.4
98
97.69 97.85
fetch bandwidth for future aggressive wide superscalars. This technique has been proven as efficient increasing the number of non taken branches and reducing the I-cache misses. In this section we want to characterize how the application of the STC reordering affects each of the fetch parameters and how the overall performance can benefit from it. In order to do so, we have used the same PC translation mechanisms used in [13] to simulate the reordering, and we have performed simulations for three different I-cache sizes: 8, 16 and 32 KB. First, we will study the impact of the STC reordering on the latency. Figure 4(a) shows the AMATi for both reordered and non reordered workloads. We can see that for the same I-cache size, the reordered workload always shows a lower latency. We can see that for a 32KB I-cache, the reordered workload has an AMATi of the same order as a 64KB one with a non reordered workload.
16KB
32KB
64KB
H14
I-cache_configurations
H12
G18
G16
G14
G12
G10
B18
B16
B14
B12
B10
Branch_predictors
(a) 5.07
5.67
(b)
3
3.12
4
3.03
FIPA
5
2 Base_4_issue
Base_4_issue_+_STC
Base_8_issue
Base_8_issue_+_STC
Configurations
(c)
Fig.4. (a) AMATi for both reordered and non reordered workload. Results for STC are shown up to 32KB I-cache. (b) Branch prediction hit rates for both reordered and non reordered workload. The branch predictors are the same used in Section 4. (c)Fetched Instructions per Access (FIPA) for both reordered and non reordered workload. The 2 leftmost bars are for 4 issue, while the right ones are for 8 issue. The second aspect we want to analyze is the impact that the STC has in the quality of the fetched instructions. Figure 4(b), shows the branch prediction hit rate for both reordered and non reordered workloads, for the same branch predictors used in Section 4. Results show that the branch prediction accuracy improves always. This is not strange because the STC reorders the basic blocks in
On the Performance of Fetch Engines Running DSS Workloads
947
2.42
2.66
2.09
2.02
2.14
2.13
2.10
IPC
1
1.62
1.22 1.54
2.08
1.96
2
1.71
1.37 1.64
1.06 1.29
IPC
2
2.62
3 2.66
3
2.67
order to leave the biased branches in a not taken form. This leads to a reduction in the destructive conflicts in the branch prediction tables. The third parameter we want to analyze is the effect that the reordering has in the effective fetch bandwidth. This effect has been shown in [12,13] for aggressive superscalar processors. Here we want to study it for our 4 and 8 issue superscalars. Figure 4(c) shows the number of fetched instructions per access to the fetch engine (FIPA) for the baseline engine on the 4 and 8 issue superscalar processors. In this simulations the branch prediction scheme used is the H12 and the I-cache evaluated is 32KB. The results show that, while the increase in bandwidth for 4 issue superscalar is small, the increase for 8 issue is bigger (about 12%). This fact, coupled with the branch prediction accuracy results, means that we are not only fetching more instructions but more accurate instructions. Finally we want to see the overall effect that our STC technique has over the execution performance. Figure 5 shows the performance increases that a conventional fetch engine would expect if the code was reordered with the STC. Results show that for the three configurations we have tested with STC, we always have similar performance to a system with double I-cache size.
H12+STC H12
1
0
0 8KB
16KB
32KB
64KB
128KB
256KB
Perfect
8KB
16KB
32KB
64KB
128KB
I-cache_configurations
I-cache_configurations
4_issue_superscalar
8_Issue_superscalar
(a)
256KB
Perfect
(b)
Fig.5. Comparison between the IPC of a non-reordered and a STC reordered workload. Chart (a) show results for 4 issue while chart(b) shows results for 8 issue. Bottom line shows the IPC for the baseline issue processor with the H12 branch predictor and several cache sizes. Upper line shows the IPC of the same configuration if the code is reordered using the STC.
7
Concluding Remarks
In this paper we have analyzed the high performance impact that the fetch engine behavior has on the overall performance of a current superscalar processor while running DSS workloads. We have seen that a bad tuning of the engine can lead to high penalties in the IPC. The impact of the three main parameters of the fetch engine has been studied. In particular we have seen that, due to the size of the working set of our workload,
948
Carlos Navarro et al.
the impact of I-cache misses is the most important. The fetch engine can not tolerate the latencies of L2 caches. This effect can even hide the benefits of a better branch predictor. So, for these workloads it is more important to spend chip area in hiding L2 cache latency (bigger I-cache for example) than adding a more complex branch predictor. Finally, we have shown how a code reordering technique (STC) affects the behavior of the different fetch parameters. It is specially interesting the fact that the branch prediction accuracy is improved without extra hardware cost. Also, we have seen that the STC can speed-up the DSS execution in more than 21% for a 4 issue processor, and 27% for an 8 issue one.
References 1. A. Ailamaki, D. Dewitt, M. Hill, and D. Wood. Dbms on modern processors: Where does time go? Proc. of the 25th Int. Conf. on Very Large Data Bases, 1999. 2. L. Barroso, K. Gharachorloo, and E. Bugnion. Memory system characterization of commercial workloads. Proc. of the 25th Intl. Symp. on Comp. Architecture, 1998 3. L. Barroso, K. Gharachorloo, A. Nowatzyk, and B. Verghese. Impact of chip-level integration on performance of oltp workloads. Proc. of the 6th Intl. Conf. on High Performance Comp. Architecture, January 2000. 4. D. Burger, T. M. Austin, and S. Bennett. Evaluating future microprocessors: The simplescalar tool set. Technical Report TR-1308, Comp. Sciences Dept., Univ. of Wisconsin-Madison, 1996. 5. T. Conte, K. Menezes, P. Mills, and B. Patell. Optimization of instruction fetch mechanism for high issue rates. Proc. of the 22th Intl. Symp. on Comp. Architecture, pages 333–344, June 1995. 6. T . Heil, Z. Smith, and J. E. Smith. Improving branch predictors by correlating on data values. Proc. of the 32th Intl. Symp. on Microarchitecture, 1999. 7. W. Hwu and P. Chang. Achieving high instruction cache performance with an optimizing compiler. Proc. of the 16th Intl. Symp. on Comp. Architecture, 1989. 8. N. J. Jouppi. Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers. Proc. of the 17th Intl. Symp. on Comp. Architecture, pages 364–373, June 1990. 9. C. Navarro, A. Ram´ırez, J. Larriba-Pey, and M. Valero. Fetch Engine Design Decissions for DSS Workloads Tech. report UPC-DAC-2000-9 , Feb. 2000. 10. A. Peleg and U. Weiser. Dynamic flow instruction cache memory organized arround trace segments independent of virtual address line. U.S. Pat. 5.381.533, Jan. 1995. 11. K. Pettis and R. Hansen. Profile guided code positioning. Proc. ACM SIGPLAN’99 Conf. on Programming Language Design and Implementation, pages 16–27, 1990. 12. A. Ram´ırez, J. Larriba-Pey, C. Navarro, X. Serrano, J. Torrellas, and M. Valero. Code reordering of decision support systems for optimized instruction fetch. Proc. of the Intl. Conf. on Parallel Processing, pages 238–245, September 1999. 13. A. Ram´ırez, J. Larriba-Pey, C. Navarro, J. Torrellas, and M. Valero. Software trace cache. Proc. of the 13th Intl. Conf. on Supercomputing, pages 119–126, 1999. 14. A. Ramirez, J. Larriba-Pey, and M. Valero. Trace cache redundancy: Red & blue traces. Proc. of the 6th Intl. Conf. on High Performance Comp. Architecture, 2000. 15. E. Rottenberg, S. Benett, and J. E. Smith. Trace cache: a low latency aprroach to high bandwith instruction fetching. Proc. of the 29th Intl. Symp. on Microarchitecture, pages 24–34, December 1996.
On the Performance of Fetch Engines Running DSS Workloads
949
16. J. E. Smith. A study of branch prediction strategies. Proc. of the 8th Intl. Symp. on Comp. Architecture, 1981. 17. J. E. Smith and W.-C. Hsu. Prefetching in supercomputer instruction caches. Supercomputing 92, pages 588–597, November 1992. 18. M. Stonebreaker and G. Kemnitz. The postgres next generation database management system. Communications of the ACM, October 1991. 19. Transaction Processing Performance Council (TPC). Tpc benchmark d (decision support) http://www.tpc.org. Standard Specification, Revision 1. 2. 3, 1993–1997. 20. T. Y. Yeh, D. T. Marr, and Y. N. Patt. Increasing the instruction fetch rate via multiple branch prediction and a branch address cache. Proc. of the 7th Intl. Conf. on Supercomputing, pages 67–76, July 1993. 21. T. Y. Yeh and Y. N. Patt. Two-level adaptive branch prediction. Proc. of the 24th Intl. Symp. on Microarchitecture, pages 51–61, 1991.
Cost-Efficient Branch Target Buffers Jan Hoogerbrugge Philips Research Laboratories, Prof. Holstlaan 4, 5656 AA Eindhoven, The Netherlands [email protected]
Abstract. Branch target buffers (BTBs) are caches in which branch information is stored that is used for branch prediction by the fetch stage of the instruction pipeline. A typical BTB requires a few kbyte of storage which makes it rather large and, because it is accessed every cycle, rather power consuming. Partial resolution has in the past been proposed to reduce the size of a BTB. A partial resolution BTB stores not all tag bits that would be required to do an exact lookup. The result is a smaller BTB at the price of slightly less accurate branch prediction. This paper proposes to make use of branch locality to reduce the size of a BTB. Short-distance branches need fewer BTB bits than long-distance branches that are less frequent. Two BTB organisations are presented that use branch locality. Simulation results are given that demonstrate the effectiveness of the described techniques.
1 Introduction Branch target buffers (BTBs) play an important role in many pipelined processors in which branch prediction is employed to provide a steady flow of instructions in the presence of branch instructions [1]. A BTB is a cache in which branch targets are stored. In its most elementary form, a BTB is a table indexed by the lower part of the pc (program counter) with entries consisting of a tag and a branch target. In the fetch stage of the pipeline the BTB is indexed by the lower part of the pc. The tag of the returned entry is compared with the remaining bits of the pc. If these pc bits match, the instruction that is fetched from the instruction cache will be a branch instruction. In that case the target returned by the BTB will be assigned to the pc so that instructions will be fetched from that address in the next cycle. This scheme works fairly well, since most branches are taken (65-70% on average) and branches are stored in the BTB when they are taken. A first improvement on this scheme is to make the BTB set-associative to reduce conflicts between branches mapping onto the same entry. Instead of indexing one entry from the BTB, a set of n entries is indexed and n parallel tag compares determine the presence of a branch and, if present, which entry it contains. A further improvement is to predict the direction, i.e. taken or not taken, of a branch not by the presence of it in the BTB but in a more sophisticated way. A common method is to include a two bit saturating counter in a BTB entry [2]. This counter is incremented on a taken branch and decremented on a not-taken branch. A branch will be predicted to be taken if its corresponding counter has its highest bit set. Very often these counters are decoupled from the BTB and stored in another table which has typically more entries than the BTB and is often accessed differently to exploit branch correlation [3, 4, 5]. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 950–959, 2000. c Springer-Verlag Berlin Heidelberg 2000
Cost-Efficient Branch Target Buffers
951
Another improvement is to employ a return address stack (RAS) [6]. Function return branches are indirect jumps with varying targets. This makes that the target stored in the BTB, which is simply the target of the last execution of the branch, is a worse target predictor. A RAS can improve this by letting function call branches push the return address on it and letting function return branches pop values of it and use them as target prediction. For this to work, the fetch stage must know whether branches are function calls, function returns, or neither. This sort of type information can be stored in the BTB. If the BTB reports that the fetched instruction is a function call it pushes the address of the next sequential instruction on the RAS. If the BTB reports that the fetched instruction is a function return it pops a value of the RAS and uses it for target prediction. The objective of the work described in this paper is to improve the cost-efficiency of BTBs by using fewer bits per BTB entry. This will make BTBs smaller and reduce power dissipation. The latter can be quite high since a BTB is a table of typically a few kbyte in size that is accessed every cycle1 . Alternatively, with the same area and power budgets one can make a BTB that delivers more performance with the techniques described in this paper. We obtain a reduction in storage requirements by making the tag and target fields of a BTB entry smaller. Both reductions are shown in Fig. 1. Reduction of tag fields is known as partial resolution and has been proposed by Fagin and Russell [8]. The main contribution of this paper is target field size reduction. We make use of branch locality, i.e., most branch distances are short. Two schemes are presented that exploit branch locality with comparable performance. In one scheme BTB entries can store a short branch and two entries are required to store a long branch, which occurs less frequently. In the other scheme only one entry per set of a set-associative is able to store long branches. The other entries can only store short branches and are therefore smaller. The remainder of the paper is organised as follows. Section 2 describes the simulation environment we used for our experiments. Section 3 discusses partial resolution. Section 4 presents the two BTB organisations that exploit branch locality. Section 5 concludes the paper.
2 Simulation Environment We used the SimpleScalar v2.0 tool set for our experiments in which we modified the BTB simulation code [9]. SimpleScalar resembles the 32-bit MIPS architecture. We used a decoupled system in which the BTB detects branches and provides branch targets and type information, and a separate predictor provides direction predictions. To expose the effect of different BTB organisations, we use a fairly aggressive hybrid branch predictor consisting of a 4096 entry gshare predictor with 12 bits history, a 2048 entry bimodal predictor, and a 2048 entry meta predictor [3, 4, 5]. We used a 32 entry RAS. The BTBs used are 4-way set-associative with LRU replacement with sizes varying from 32 to 1024 entries. 1
According to Musoll [7], the Intel Pentium Pro dissipates 5% of its total power dissipation in its 512-entry BTB. This percentage becomes higher for a less aggressive processor or a processor with less complex decoding. The percentage will also be higher for 64-bit processors.
952
Jan Hoogerbrugge (a)
Reducing tag fields (b)
Reducing target fields (c)
0 0 0 1 0 0 0 0
(d)
0 0 0 0 0 1 1 0
Paired−entry BTB
Variable−size BTB
Fig. 1. Organisations of four types of BTBs. The gray parts represent tag fields while the white fields represent target and type information fields. The picture is approximately on scale. (a) shows a traditional 4-way set-associative BTB with 32 entries. (b) shows a BTB with partial resolution in which tag fields have been reduced. (c) and (d) show two organisations in which branch locality is exploited. (c) shows a paired-entry BTB (PE-BTB) in which long branches occupy two BTB entries. (d) shows a variable-size BTB (VS-BTB) in which only one BTB entry per set can store a long branch. As a benchmark set we used the eight benchmarks from SPECint95. Simulation was limited to 100 million instructions.
3 Partial Resolution BTBs with partial resolution do not store all the address bits that are not used for indexing in tags [8]. This reduces both the number of storage bits as well as the size of the comparators that perform the tag compare. The result of partial resolution is socalled false-hits; the BTB hits for an instruction which is not a branch instruction or is a branch instruction that does not correspond to the information returned by the BTB. If the branch predictor predicts taken, then the control flow is very likely to be directed in the wrong direction. This will be detected and corrected in a later pipeline stage. The effect is a cycle penalty comparable to the misprediction penalty. How often this occurs will depend on how many tag bits are used. Fagin and Russell conclude that 2 tag bits are necessary to obtain 99.9% of the accuracy of full resolution and 3 tag bits for 99.99%. Our experiments however showed that more tag bits are required. Figure 2 shows the results of these experiments. We used two benchmarks with a large branch working set, gcc and go, and two benchmarks with a small branch working set, compress and
Cost-Efficient Branch Target Buffers gcc - 128 entries
30 25
10
5
5
0
2
8
0
10
Rate (%)
5
5
4 6 Number of Tag Bits
8
0
10
compress - 128 entries
30
10
5
5
2
4 6 Number of Tag Bits
8
0
10
li - 128 entries
30 25
10
5
5
0
2
4 6 Number of Tag Bits
2
8
10
4 6 Number of Tag Bits
8
10
li - 512 entries Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
15
10
0
0
20
15
10
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
25
Rate (%)
20
8
compress - 512 entries
30
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
4 6 Number of Tag Bits
15
10
0
2
20
15
0
0
25
Rate (%)
20
10
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
30
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
25
8
15 10
2
4 6 Number of Tag Bits go - 512 entries
20
10
0
2
25
15
0
0
30
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
20 Rate (%)
4 6 Number of Tag Bits go - 128 entries
25
Rate (%)
15
10
30
Rate (%)
20
15
0
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
25
Rate (%)
Rate (%)
20
gcc - 512 entries
30
Misprediction, w/o inv. False-hit & taken, w/o inv. Misprediction, w/ inv. False-hit & taken, w/ inv.
953
0
0
2
4 6 Number of Tag Bits
8
10
Fig. 2. Misprediction and false-hit rates for gcc, go, compress, and li for 128 and 512 entry BTBs as a function of the number of tag bits. The false-hit taken rate of compress for 512 entries without invalidation is so high (more than 430%) that it lies beyond the vertical range of the graph.
954
Jan Hoogerbrugge
li. We used BTBs of 128 and 512 entries. Two policies were used to handle false-hits: invalidation and no invalidation. With invalidation a BTB entry that causes a false-hit is invalidated to prevent it continuing to cause false-hits2 . This is achieved by a special type information value in the type information field of the BTB entry. Alternatively, one can write a random value in the tag field. The graphs of Fig. 2 show two values: the misprediction rate and the false-hit taken rate. The latter is the ratio of the number of non-branch instructions that cause a false-hit that are predicted taken relative to the number of branch instructions. Obviously, this value can be more than one when only a few tag bits are used. Several conclusions can be drawn from the results shown in Fig. 2. First, both the misprediction and the false-hit taken rate decrease as the number of tag bits increases. More tag bits reduces the probability of branches being mixed up with other branches (which decreases the misprediction rate) and that of branches being mixed up with non-branch instructions (which decreases the false-hit taken rate). Invalidation reduces the false-hit taken rate significantly but increases the misprediction rate because BTB entries of branches are invalidated. Because the decrease in false-hit taken rate is much higher than the increase in misprediction rate, it is a good design decision to invalidate BTB entries that cause false-hits. The number of tag bits required for a misprediction rate close to the misprediction rate for full resolution and a false-hit taken rate close to zero depends clearly on the branch working set size of the application. For the small branch working sets of compress and li, one to four tag bits will be sufficient. For the larger working sets of gcc and go, seven to eight tag bits will be required. This is significantly more than the two to three tag bits advised by Fagin and Russell. Because gcc and go are more representative of real-world applications than compress and li, we propose to use at least eight tag bits. Note that BTBs with fewer entries will have a higher false-hit rate and will therefore need more tag bits to reduce this to close to zero.
4 Exploiting Branch Locality The tag and target fields are the largest fields of a BTB entry. Partial resolution shortens the length of tag fields. In this section we show how target fields can be shortened by making use of branch locality, i.e., the property of most programs that most branch distances are short. Branch locality is already being used for pc-relative branches in most architectures to reduce code size. Storing pc-relative offsets in the BTB is likely to be infeasible since a BTB lookup and addition would have to be performed sequentially in one clock cycle. A solution is to use page-relative addressing. Short branches are branches in which only the lowest n significant3 pc bits change. The other branches are long branches. The idea is to choose a small value for n such that most branches are short branches and to develop a BTB such that short branches are stored in fewer 2
3
Fagin and Russell do not mention invalidation in their paper [8]. It is therefore not clear whether they apply this. In many architectures instructions are aligned on 2m byte addresses. In that case the lowest m bits are always zero and therefore not significant and do not have to be stored in the BTB.
Cost-Efficient Branch Target Buffers
955
bits than longer branches. First we determine a value for n. Figure 3 shows the percentage of short branches as a function of the number of addressing bits for pc-relative and page-relative addressing for gcc. Function return branches were excluded in this measurement since their targets are stored in the RAS and not in the BTB. The measurement shows that pc-relative addressing would be a little bit more effective for our purpose than page-relative addressing. The reason is that a short-distance branch across a boundary that is a high power of two is a long branch. Nevertheless, pc-relative addressing is not applicable for the reason mentioned above. The graph of Fig. 3 starts to level off at n = 10, at which 86% of the branches are short branches. Therefore, n = 10 will be a suitable value. 100 PC-relative Page-relative
Short Jumps (%)
80
60
40
20
0
5
10 Number of Bits (n)
15
20
Fig. 3. Percentage of short branches as a function of the number of offset bits n. In pcrelative addressing, a branch from a to b is a short branch if −2n−1 ≤ b − a < 2n−1 . In page relative addressing, a branch from a to b is a short branch if only the n lowest significant bits of a and b differ in value.
4.1 Paired-Entry and Variable-Size BTBs We propose two BTB organisations for exploiting branch locality: paired-entry and variable-size BTBs. —A paired-entry BTB (PE-BTB) is a set-associative BTB in with which an entry can contain one short branch and a pair of two entries from one set is needed to store a long branch. For efficient implementation, entry pairs are required to be adjacent and aligned in the set. A pair-bit is required per entry pair to determine whether entries are paired. For optimal resource utilisation, the total storage requirements of two short branches should be equal to the storage requirements of one long branch. This limits the range of possible tag and target sizes; the sum of 2 tags and 2 short targets should be equal to 1 tag and 1 long target (ignoring type information fields). To obtain better matching, one can decide to use different tag sizes for long and short branches or for different ways.
956
Jan Hoogerbrugge
—A variable-size BTB (VS-BTB) is a set-associative BTB with which some part of the entries of each set can store both long and short branches while the remaining set entries can store only short branches. Using the target information from the BTB is straightforward. The lookup returns on a hit whether a short or a long branch is hit. In the case of a short branch and a taken prediction, only the n lowest significant bits of the pc are updated; the upper bits retain their value. In the case of a PE-BTB, the pair-bits are used in the tag comparison. If a long branch is stored in two BTB entries, the tag compare of the entry whose tag bits are used to store the long target has to be disabled. Some design options exist in updating the BTB. In the case of updating a PE-BTB with a long branch: (1) one can victimise the pair containing the least recently used entry, or (2) one can victimise the pair which has been least recently used, where a pair is used when one of its entries is used. To understand the difference, consider the following set of four entries showing the latest access times (T1 < T2 < T3 < T4 ): T1
T4
T2
T3
In the case of the first alternative, the first pair would be victimised (because it contains the first entry which is the least recently used entry), whereas in the case of the second alternative the second pair would be victimised (because this pair was used least recently). We opted for the first alternative because of its simplicity and because the differences in prediction accuracy are negligible. For VS-BTBs one has to decide whether short branches are allowed in long entries or whether long entries are reserved exclusively for long branches. We chose the first alternative, which yields a slightly better prediction accuracy. 4.2 Evaluation To evaluate the cost-efficiency of the discussed techniques we defined the four BTB types that in Table 1. All types are 4-way set-associative and use LRU replacement. In the entry size calculations we neglected the bits for the LRU administration, which are the same for all types. In the case of partial resolution, we used 8 bit tags. For short branches we used 10 bit page-relative addressing. We assumed a 32-bit RISC architecture in which instructions were 32-bit aligned, so absolute addresses were 30 bits long. Two bits of type information were used per entry, which is sufficient to distinguish four cases: (1) function call branches, (2) function return branches, (3) invalidated entries, and (4) the normal case. We varied the number of entries between 32 and 1024 in powers of two (2n , 5 ≤ n ≤ 10). Figure 4 shows BTB size vs. prediction accuracy curves for the four BTB types for the eight SPECint95 benchmarks. Figure 5 shows the average of the eight benchmarks. The curves obtained for the benchmarks corresponding to the four BTB types run in parallel until the BTB reaches a size at which it no longer constrains the prediction accuracy. The distances between the curves indicate the improved cost-efficiency of partial resolution, VS-BTBs, and PE-BTBs. The point at which the curves meet depends on the branch working set size of the benchmark. This point shifts to the right as the
cc1 Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
Misprediction Rate (%)
10 Size of BTB (kbits)
24 22 20 18
5.26 5.24
1
10 Size of BTB (kbits)
100
ijpeg Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
8.8 8.75 8.7 8.65 8.6 8.55 8.5 8.45
1
10 Size of BTB (kbits)
8.4
100
li
6
Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
5 4.5 4 3.5
10 Size of BTB (kbits)
100
m88ksim Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
12 Misprediction Rate (%)
5.5
1
13 11 10 9 8 7 6 5
1
10 Size of BTB (kbits)
4
100
perl
24 22 20 18
1
14 12 10 8
100
Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
16
16
10 Size of BTB (kbits) vortex
18
Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
Misprediction Rate (%)
Misprediction Rate (%)
5.3 5.28
8.85
16
Misprediction Rate (%)
5.32
8.9
Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
26
14 12 10 8 6 4 2
6 4
5.34
5.2
100
go
28
3
Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
5.36
5.22 1
30
14
957
compress95
5.38
Misprediction Rate (%)
30 28 26 24 22 20 18 16 14 12 10 8
Misprediction Rate (%)
Misprediction Rate (%)
Cost-Efficient Branch Target Buffers
1
10 Size of BTB (kbits)
100
0
1
10 Size of BTB (kbits)
100
Fig. 4. BTB size vs. mispredict rate for the BTB types listed in Table 1 and shown in Fig. 1.
958
Jan Hoogerbrugge Type
Resolution
Target size
Type info size Entry size Fig. 1
Traditional Full – 23-27b 4 × 30 Traditional Partial – 8b 4 × 30 PE-BTB Partial – 8b 2 × 30/4 × 10/1 × 30 + 2 × 10 VS-BTB Partial – 8b 4 × 10/1 × 30 + 3 × 10
2 2 2 2
220-236 160 82 100
(a) (b) (d) (c)
Table 1. Four types of BTBs corresponding to the types shown in Fig. 1. 15 Full resolution Partial resolution PE-BTB and partial resolution VS-BTB and partial resolution
14
Misprediction Rate (%)
13 12 11 10 9 8 7
1
10 Size of BTB (kbits)
100
Fig. 5. BTB size vs. average mispredict rate for the BTB types listed in Table 1. benchmarks become larger, and therefore more realistic. It will also shift to the right in a multi-tasking environment which puts more pressure on branch prediction resources. The results show that both VS-BTBs and PE-BTBs have a better cost-efficiency than traditional BTBs, with PE-BTBs performing a little bit better than VS-BTBs. In the case of li, perl, and vortex, VS-BTB is performing worse than the other organisations for a large number of entries. The reason is conflicts between long branches that can only be mapped in one entry per set. This is clearly a disadvantage of VS-BTBs. 4.3 Variations Several variations are possible on the VS-BTB and PE-BTB described above. First, one can use more than two target sizes. For example, 6 bits, 12 bits, and 30 bits. This could improve the utilisation of target fields. PE-BTBs can be extended to three or more entries for long branches. This could be useful to match the total storage space of multiple short branches with the storage space of one long branch, especially when target size of short branches are very small, e.g., 6 bits, or the target size of a long branch is very large, e.g., 62 bits in a 64-bit architecture. In the case of VS-BTBs, one can use several memories with a different number of entries for different branch target sizes. For example, one could use a table of 128 entries with 30 bit target fields together with a table of 512 entries with 10 bit target fields. The combination is a two-way set-associative BTB with the two ways having different numbers of entries.
Cost-Efficient Branch Target Buffers
959
5 Conclusions Two methods for reducing the size and power dissipation of BTBs have been described: partial resolution and exploitation of branch locality. Partial resolution has been proposed before while exploiting branch locality is a contribution of this work. Partial resolution reduces the number of tag bits. This reduces the size of a BTB at the cost of a slightly higher misprediction rate due to false-hits. Two BTB organisations have been proposed for exploiting branch locality. In both organisations a distinction is made between short branches and long branches, with short branches being branches with which a certain number of the highest pc bits do not change when the branch is taken. In the VS-BTB organisation, one or more entries of a set (both proposed organisations assume set-associativity) can store both long and short branches while the others can store only short branches. In the PE-BTB organisation, entries can store only a short branch and two entries are required to store a long branch. These paired entries are adjacent and aligned for simplicity. The effectiveness of partial resolution, VS-BTBs, and PE-BTBs has been demonstrated by means of simulations. The results show a clear improvement in cost-efficiency for 32-bit architectures, with PE-BTBs being sightly more efficient than VS-BTBs. For 64-bit architectures, the improvement that can be realized with the described techniques will be even greater.
References [1] Johnny K. F. Lee and Alan Jay Smith. Branch Prediction Strategies and Branch Target Buffer Design. IEEE Micro, 21(7):6–22, January 1984. [2] James E. Smith. A Study of Branch Prediction Strategies. In Proceedings of the 8th Annual International Symposium on Computer Architecture, pages 135–147, May 1981. [3] Shien-Tai Pan, Kimming So, and Joseph T. Rahmeh. Improving the Accuracy of Dynamic Branch Prediction Using Branch Correlation. In Proceedings of the Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 76–84, Boston, Massachusetts, October 12–15, 1992. [4] Tse-Yu Yeh and Yale N. Patt. Two-Level Adaptive Training Branch Prediction. In Proceedings of the 24th Annual International Symposium on Microarchitecture, pages 51–61, Albuquerque, New Mexico, November 18–20, 1991. [5] Scott McFarling. Combining Branch Predictors. Technical Report TN-36, Western Research Laboratory, Palo Alto, California, June 1993. [6] David R. Kaeli and Philip G. Emma. Branch History Table Prediction of Moving Target Branches Due to Subroutine Returns. In Proceedings of the 18th Annual International Symposium on Computer Architecture, pages 34–42, Toronto, Ontario, May 27–30, 1991. [7] Enric Musoll. Predicting the Usefulness of a Block Result: A Micro-Architectural Technique for High-Performance Low-Power Processors. In Proceedings of the 32th Annual International Symposium on Microarchitecture, pages 238–247, Haifa, Israel, November 16–18 1999. [8] Barry Fagin and Kathryn Russell. Partial Resolution in Branch Target Buffers. In Proceedings of the 28th Annual International Symposium on Microarchitecture, pages 193–198, Ann Arbor, Michigan, November 29–December 1, 1995. [9] Doug Burger and Todd M. Austin. The SimpleScalar Tool Set, Version 2.0. Technical Report 1342, University of Wisconsin-Madison, Computer Sciences Department, June 1997.
T w o -L ev e l A d d r e ss S to r a g e a n d A d d r es s P r e di c t i o n Enric Morancho, José María Llabería and Àngel Olivé Computer Architecture Department - Universitat Politècnica de Catalunya (Spain) 1 Abstract. : The amount of information recorded in the prediction tables of
the address predictors turns out to be comparable to current on-chip cache sizes. To reduce their area cost, we consider the spatial-locality property of memory references. We propose to split the addresses in two parts (high-order bits and low-order bits) and record them in different tables. This organization allows to record only once every unique high-order bits. We use it in a last-address predictor and our evaluations show that it produces significant area-cost reductions (28%-60%) without performance decreases.
1
Introduction
True-data and control dependencies are the major bottleneck for exploiting ILP. Some works [4][6][11] propose the use of prediction and speculation to overcome data dependencies. In load instructions, there is a true-data dependence between the address computation and the memory access; this dependence contributes to the large latency of the load instructions and can affect processor performance. Then, address predictors are valuable to access memory speculatively [4][10]. A typical address-prediction model is the last-address [11]. It assumes that a load instruction will compute the same address that the one computed in its previous execution. Proposals of last-address predictors [4][6] employ a direct-mapped Address Table (AT), indexed with some bits of the PC. Each AT entry contains the last address computed by a load instruction, a two-bit confidence counter, and a tag. We name this predictor Base Last-Address Predictor (BP). Last address predictors use prediction tables that record up to 4.096 64-bit addresses [2][10], that is, 32 Kbytes; that is comparable to current on-chip cache sizes. However, the previous designs do not exploit the locality of the addresses. This property has been used in other works to take different advantages [3][12][13] (a detailed description of related works can be found in [9]). In this paper, we propose a new organization of the prediction table and we apply it to a typical last-address predictor to obtain a significant area-cost reduction. This paper is organized as follows. Section 2 presents our proposal. Section 3 evaluates our proposal, and Section 4 summarizes the conclusions of this work.
1. Author’s address: Computer Architecture Department, Universitat Politècnica de Catalunya, Jordi Girona 1-3, 08034 Barcelona (Spain). E-mail: [email protected]
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 960-964, 2000. © Springer-Verlag Berlin Heidelberg 2000
Two-Level Address Storage and Address Prediction
2
961
Two-Level Address Predictor
lin k ch u lo nk_ w id _ co add co nfid res un en s t c tag er e
s re s dd _a
PC
tag
index_lat
hi gh
High-Address Table (HAT)
lin
k
co
un te
r
2.1 Basic Idea Effective addresses exhibit temporal and spatial locality; it produces redundancy in AT contents. For instance, different accesses to the same global variable produce temporal redundancy. Also, variables stored in consecutive addresses and stack accesses produce spatial redundancy. We propose an organization to record them non redundantly. AT is split in two parts: the Low-Address Table (LAT) and the High-Address Table (HAT). LAT records the low-order bits of the addresses and HAT the high-order bits; moreover, each LAT entry is linked to a HAT entry. Then, a HAT entry can be shared by several LAT entries. We apply this organization to the BP and we obtain the Two-Level Address Predictor (2LAP); Figure 1 shows its scheme. To predict a load instruction, 2LAP indexes LAT using the load-instruction PC; this access obtains the low-order address portion and a link to HAT. After that, HAT is accessed to obtain the high-order portion. This sequential access does not imply an implementation restriction because LAT can be accessed early in the pipeline and the large number of pipeline stages before issuing an instruction; for instance, 5 stages in an Alpha 21264 [1]. Moreover, we could reduce the critical path of the speculative access by recording in LAT enough bits for indexing the cache.
Low-Address Table (LAT) Predicted Address
b bits
Fig. 1.Two-Level Address Predictor. 2.2 Locality Analysis and HAT Size There is a trade-off between the number of HAT entries and the bits of the low-order portions (b). To obtain recommended sizes, we evaluate the locality in AT contents [9]; the suggestion is b=10, 12 or 14 and 64 HAT entries. We choose this HAT size because up to 96-entry HAT's can be accessed fully associatively in a single processor cycle [5]. 2.3 Prediction-Table Management 2LAP updates LAT like BP updates AT, using the always-allocate policy. Also, before recording a high-order portion in HAT, 2LAP must verify if it is yet recorded. To check it, 2LAP looks for the high-order portion in HAT. If it is found (HAT hit), both entries are linked; if not, a HAT entry is evicted. HAT Replacement and Tracing Empty HAT Entrie s To reduce the eviction from HAT of useful information, we trace HAT entries not related to any LAT entry (empty HAT entries). To detect it, we relate to every HAT entry a link counter that reflects the number of LAT entries linked to it (link_counter in Figure 1). If none empty HAT entry is found, the replacement algorithm selects
962
Enric Morancho, José María Llabería, and Àngel Olivé
randomly one HAT entry but the MRU one (no-MRU replacement algorithm). We have used this algorithm because the implementation of LRU algorithm is complex and expensive for large tables. We have evaluated the performance decrease produced by this decision: it is limited by 2.4% for b=10, 1% for b=12, and negligible for b=14. 2LAP updates the link counter on LAT replacements and on changes of the high-order portion related to a LAT entry. On HAT replacements, the LAT entries linked to the evicted HAT entry are not invalidated to simplify the design. This decision is possible because 2LAP is related to an speculative mechanism, but it produces mispredictions. Our experiments show that using three-bit link counters the performance of the 2LAP is almost saturated. These counters estimate empty HAT entries with a high correctness. Filtering-Out Some High-Order Portions in HAT 2LAP allocates unpredictable load instructions in LAT but avoids the allocation in HAT of their high-order portions, i.e., their LAT entries are not linked to any HAT entry. Moreover, the link is broken when the classification of a load instruction changes from predictable to unpredictable, and only is re-established when it changes again. The address chunk stored in low_address field of LAT is used to keep on updating the classification of the unpredictable instructions; the basic idea of this classification has been proposed in [7]. Also, the chunk is selected dynamically (chunk_id field in Figure 1) because the accuracy of 2LAP in programs with large-strided references (ijpeg) can be affected. Filtering HAT Allocations and Managing Empty HAT Entries We have evaluated the four possibilities that appear considering: a) filtering-out HAT allocations, and b) managing empty HAT entries. Our experiments show that both policies should be applied at the same time, specially in codes with a large working set of high order portions and medium-predictable instructions (gcc, go, vortex).
3
Evaluation: 2LAP versus BP
This section compares 2LAP versus BP, both using bounded prediction tables. Working-set size of static load instructions of the programs [8] justifies that the selected LAT sizes range from 256 to 4.096 entries. 3.1 Area Cost of the Predictors We evaluate the area cost of a predictor as the amount of information that it records. Following formulas show the area cost of BP and 2LAP using 64-bit logical addresses, t-bit tags, 3-bit link counters, and b-bit address chunks. BasePredictor = ( t + 64 + 2 ) × ATentries AreaCost 2LAP 64 = ( 3 + ( 64 – b ) ) × HATentrie s + log 2HATentrie s + log ------ + b + 2 + t × LATentries + log 2HATentries 2 b AreaCost
Tag length influences on predictor accuracy. In the analysed codes, [8] shows that BP accuracy saturates when the number of index and tag bits is 17; then, we compare these configurations. The area-cost reduction from a BP to a 2LAP with the same number of AT and LAT entries ranges from 37% (256 entries, b=14) to 60% (4.096 entries, b=10).
Two-Level Address Storage and Address Prediction
963
3.2 Captured Address Predictability The captured address predictability is defined as the percentage of correct predictions out of the number of executed load instructions. We will evaluate the predictability captured by several predictor configurations in the integer codes of the SPEC-95 with the largest working-set size of static load instructions (remaining programs present a similar behaviour but in a different table-size range). Our results were obtained running Alpha binaries instrumented with ATOM; programs were run until completion using reference input sets. BP
2LAP (b=10, 64 HAT entries)
50% Captured Predictability
70%
go
LAT entries
2LAP (b=12, 64 HAT entries)
2.048
4.096
40%
2LAP (b=14, 64 HAT entries)
mksim
60% 1.024 2.048
512
30%
4.096 50%
256 1.024
20% 256
10% 10000 60%
512
40% AT entries 30%
100000
Area Cost (bits)
400000
10000 60%
gcc
50%
50%
40%
40%
30%
30%
20%
100000
400000
vortex
20%
10000
100000
400000
10000
100000
400000
Fig. 2.Predictability captured by BP and 2LAP in several benchmarks. Horizontal axes stand for base-10 logarithm of predictor area cost, vertical axes stand for captured predictability.
Figure 2 shows the predictability captured by 2LAP and BP. Horizontal axes stand for area cost and vertical axes for captured predictability. Leftmost top graph is labelled with the number of AT and LAT entries of the configurations.Area-cost reduction from a 2LAP to a BP with the same number of LAT and AT entries do not represent a performance loss; for AT entries=LAT entries=4.096, a continuous oval surrounds these configurations. When LAT entries=2×AT entries (configurations surrounded by a dashed oval for LAT entries=2×AT entries=2.048), we obtain configurations with similar area cost. 2LAP outperform BP because LAT has less capacity misses than AT. 3.3 Accuracy The accuracy of a predictor is defined as the percent of correct predictions out of the number of predictions. As every misprediction could produce a penalty of several processor cycles, the 2LAP should not present a lower accuracy than the BP. Our evaluations show that for b=10, 2LAP presents a slightly lower accuracy than the BP (in the worst benchmark -gcc- the difference is limited by 0.7%). For b=14, the difference is negligible. We present in [9] a detailed accuracy comparison.
964
4
Enric Morancho, José María Llabería, and Àngel Olivé
Conclusions
We have shown that the spatial-locality property of the memory references produces redundancy in the prediction tables. We have taken advantage of this fact to reduce the area cost of the prediction tables. Our proposal splits the addresses computed by the load instructions in two parts: high-order and low-order portion. Addresses with the same high-order portion are recorded sharing one copy of the portion. Also, management of empty HAT entries, and filtering-out allocations of high-order portions related to unpredictable instructions improve the performance of the proposal. Other prediction models (stride, context and hybrid) can also take advantage of the locality of the addresses to reduce their area cost. This work proposes a new organization of the prediction table but it maintains the allocation policy, then, our proposal predicts the same instructions than the BP, and IPC speed-up is the one reported in other works [2][4][10].
Acknowledgements This work was supported by the spanish government (grant CICYT TIC98 0511-C02-01), and the CEPBA (European Centre for Parallelism of Barcelona)
References 1. Alpha 21264 MicroProcessor Data Sheet. (1999). Compaq Computer Corporation. 2. B. Black, B. Mueller, S. Postal, R. Rakvic, N. Utamaphethai and J.P. Shen. (1998). Load Execution Latency Reduction. In ICS-12, pp. 29-36 3. M. Farrens and A. Park. (1991). Dynamic Base Register Caching: A Technique for Reducing Address Bus Width. In ISCA-18, pp. 128-137. 4. J. González and A. González. (1997). Speculative Execution via Address Prediction and Data Prefetching. In ICS-11. 5. B. Jacob and T. Mudge. (1998). Virtual Memory in Contemporary Microprocessors. IEEE Micro, Vol 18(4), pp. 60-75. 6. M.H. Lipasti, C. B. Wilkerson and J.P. Shen. (1996). Value Locality and Load Value Prediction. In ASPLOS-7. 7. E. Morancho, J.M. Llabería and À. Olivé. (1998). Split Last Address Predictor. In PACT'98. 8. E. Morancho, J.M. Llabería and À. Olivé. (1999). Looking at History to Filter Allocations in Prediction Tables. In PACT99, pp. 314-319 9. E.Morancho, J.M. Llabería and À. Olivé. (1999). Two Level Address Storage and Address Prediction. Technical report UPC-DAC-99/48. 10. G. Reinman and B. Calder. (1998). Predictive Techniques for Aggresive Load Speculation. In MICRO-31. 11. Y. Sazeides and J.E. Smith. (1996). The Predictability of Data Values. In MICRO-29. 12. A. Seznec. (1994). Decoupled sectored caches: reconciliating low tag volume and low miss rate. In ISCA-21. 13. H. Wang, T. Sung and Q. Yang. (1997). Minimizing Area Cost of On-Chip Cache Memories by Caching Address Tags. IEEE Transactions on Computers, 46 (11).
Hashed Addressed Caches for Embedded Pointer Based Codes Marian Stanca, Stamatis Vassiliadis, Sorin Cotofana, and Henk Corporaal Electrical Engineering Department Delft University of Technology Delft, The Netherlands {akela, stamatis, sorin, heco}@cardit.et.tudelft.nl
Abstract. We are proposing a cache addressing scheme based on hashing intended to decrease the miss ratio of small size caches. The main intention is to improve the hit ratio for ’random’ patterns pointer memory accesses for embedded (special purpose) system applications. We introduce a hashing scheme, denoted as bit juggling, and measure the effect such a scheme has in the cache access miss ratio. It is shown, for the considered benchmark, that 3-bit bit juggling will reduce the miss ratio for up to 12%, for associative caches of maximum size of 16KBytes when compared to usual cache addressing schemes.
1
Introduction
In this paper we address the issue of lowering the cache miss ratio for pointer based codes. We assume that we have pointer based applications that are meant to be executed on an architecture that includes a cache memory. Moreover we consider that we are at liberty to change the hardware implementation of that cache memory. More in particular we address the issue of hashing the cache memory for pointer based codes. To evaluate the effectiveness of hashed caches we propose a framework which is based on a simulation tool. Our framework includes a modified cache simulation tool, based on the Simple Scalar[1]. Our experiments suggested the following: – Hashed caches can lower the cache miss ratio for certain types of applications, while worsening the performance for others. – The effectiveness of the approach can be assessed only with a priori knowledge of the memory access patterns. – The implementation cost for the necessary extra hardware is negligeable (mostly inexistent). – Improvement in miss ratio after applying the hashing scheme is up to 12%.
2
Hashing Functions and Bit Juggling Addressing
An address issued by the CPU has three different components: a tag used to identify if a line is in the cache memory, a line index to choose one of m entries A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 965–968, 2000. c Springer-Verlag Berlin Heidelberg 2000
966
Marian Stanca et al.
2 4 Mapped adress space
1
Juggled adress space
3 5
Address space
Address space
Fig. 1. Normal and hashed address mapping
Fig. 2. Tag, line index and offset extracting from issued address from the m-way set associative cache, and a line offset to choose one memory location from the cache line. The usual addressing of direct mapped caches and set associative caches are depicted in Equation 1 and Equation 2 respectively. Fhashing = (Block address) MOD (# of blocks in the cache memory)
(1)
Fhashing = (Block address) MOD (# of sets in the cache memory)
(2)
Our approach is using hashing functions in order to minimize the traffic between two consecutive memory levels. The usual hashing function (MOD) used to map main memory addresses into cache memory addresses can be seen in Figure 1, as well as a different hashing function. Certain bits used for line index or offset are extracted from the address issued by the CPU as seen in Figure 2. We are introducing a hashing technique for caches, named bit juggling (BJ). Instead of using the number of necessary bits starting with position 0 to the left for line offset we are using successive bits starting from at least position 1. The necessary tag bits will be the same. The line index bits will be the remaining ones. If, for example, we are using 1-bit BJ, the memory accesses patterns will change as follows: odd address memory accesses in the range of two line size will be stored consecutively in the same line. The line placement mechanism will remain the same except for the notion of successiveness. Successive addresses will now be, in the cache memory, those separated by 2bitjuggled addresses. Accommodating
Hashed Addressed Caches for Embedded Pointer Based Codes
967
Table 1. Data cache miss rates for various configurations Line size → Cache size 512Bytes(2) 4KBytes(2) 8KBytes(2) 16KBytes(2) 512Bytes(4) 4KBytes(4) 8KBytes(4) 16KBytes(4)
8 45.10 39.11 34.38 28.25 43.44 37.12 29.29 24.37
8 BJ3 36.00 28.76 24.77 19.12 33.17 27.19 18.31 13.28
16 16 - BJ3 43.65 35.26 37.15 27.93 33.87 22.15 27.96 18.02 40.34 32.23 35.40 26.20 26.14 16.26 21.55 11.41
32 41.43 36.72 33.15 27.03 38.52 31.21 22.46 17.67
32 BJ3 34.82 25.12 20.10 16.53 29.48 23.17 12.32 8.60
such a technique will necessitate changes to the cache buffers only. In order to simplify the approach we will assume that we may choose the main memory organization’s in banks. The number of banks is specified by the number of bits juggled. Therefore, next memory level organization’s will allow reading and writing with strides. The line identification process needs the extension of the tag, with a number of redundant bits. The extra number of bits are the number of bits to be juggled. The line replacement mechanism remains the same, according with the cache memory associativity.
3
Evaluation
We used the Simple Scalar simulation tool[1]. We have modified the cache simulator in order to accommodate our technique. The modifications performed were pertaining the addressing mode of the cache, the hit/miss handler routine, and the replacement algorithm. ’Pointer based’ and ’big enough to observe relevant statistical phenomenons’ were two key characteristics when choosing benchmarks to test our technique. The pointer based C implementation of MPEG2[2] is the only benchmark with the right characteristics. After obtaining an instrumented executable, namely the MPEG2 encoder / decoder, we simulated it with our modified tool. The results presented are for the MPEG2 decoder, the encoder shows similar results and performance improvements patterns. We simulated an MPEG2 decoder with a 476 frames mpeg stream, with 300 x 200 pixels image size. The results are presented in Table 1 for a number of cache sizes, cache associativities of 2 and 4, and for various cache line sizes. Due to implementation particularities cache miss ratio varies proportionally with the frame stream size(weak) and frame size(strong). The data stream is organized in 8 x 8Bytes objects(searched and identified in successive frames), or smaller. Due to this reason we had to keep the line size smaller than the customary 32-128Bytes in general computing in order to observe relevant patterns in memory accesses in various parts of the application. Because of the small cache line size the number of first reference cache misses was increased. The simulations were performed
968
Marian Stanca et al.
Table 2. Data cache miss rates for 64Bytes cache size, 8Bytes line size
No Bit Juggling Bit Juggling 3bits
4 entries, associativity 2 2 entries, associativity 4 34.91 32.34 25.17 21.75
without a second level cache memory. Due to the data stream organization the 3-bits BJ was our prime candidate to test our technique. The improvement in performance was up to 12%. For the direct mapped cache, a cache line can be stored in one place only and collisions are solved by replacing the old cache line with the new one. Because of this fact a change in the memory access patterns for caches with associativity of 1 will not show notable modifications in performance. Therefore we concentrate our efforts on cache memories with greater associativity. Caches smaller then 4KBytes are not likely to be found in general purpose computing but can be found in embedded applications. In our simulations we assumed small cache sizes intended for application specific inexpensive engines. For comparison reasons we have included some simulation figures for a very small cache (64Bytes) in Table 2, for a line size of 8Bytes and a MPEG stream containing only 34 frames, with 150 x 100 pixels image size. We can conclude that the MPEG2 pointer based implementation should be ran on an architecture with a hashed cache with 3 bits bit juggling technique. Array or record based codes with accesses in strides may be also used to test this technique. In this case deterministic determination of the number of bits to be juggled is possible from inspection.
4
Conclusions and Future Work
A new technique to decrease the cache miss ratio, denoted bit juggling, for pointer based codes has been introduced. Bit juggling has the characteristic of any hashing process, namely is offering improvement or worsening of performance according to the application. For a sample of pointer based code it has been shown that three bits bit juggling is offering good relative performance improvements. Using the same technique with lesser bits shows degradation in performance. The success in applicability of this technique is related with the amount of information on memory access patterns a priori known. Embedded applications are, therefore, a primary target.
References 1. D. C. Burger and T. M. Austin. The simplescalar tool set, version 2.0. Technical Report CS-TR-1997-1342, University of Wisconsin, Madison, June 1997. 2. M. S. S. Group. Mpeg-2 video codec, http://www.mpeg.org, 1996. 3. G. D. Knott. Hashing functions. The Computer Journal, 18(3):265–278, Aug. 1975.
BitValue Inference: Detecting and Exploiting Narrow Bitwidth Computations Mihai Budiu1 , Majd Sakr2 , Kip Walker1 , and Seth C. Goldstein1 1 Carnegie Mellon University {mihaib,kwalker,seth}@cs.cmu.edu 2 Pittsburgh University [email protected]
Abstract. We present a compiler algorithm called BitValue, which can discover both unused and constant bits in dusty-deck C programs. BitValue uses forward and backward dataflow analyses, generalizing constantfolding and dead-code detection at the bit-level. This algorithm enables compiler optimizations which target special processor architectures for computing on non-standard bitwidths. Using this algorithm we show that up to 31% of the computed bytes are thrown away (for programs from SpecINT95 and Mediabench). A compiler for reconfigurable hardware uses this algorithm to achieve substantial reductions (up to 20-fold) in the size of the synthesized circuits.
1
Introduction
As the natural word width of processors increases, so grows the gap between the number of bits used and those actually required for a computation. Recent architectural proposals have addressed this inefficiency by providing collections of narrow functional units or the ability to construct functional units on the fly. For example, instruction set extensions which support subword parallelism (e.g., [10]), Application-Specific Instruction-set Processors (ASIPs) (e.g., [9]), and reconfigurable devices (e.g., [11]) all allow operations to be performed on operands which are smaller than the natural word size. Reconfigurable computing devices are the most efficient at supporting arbitrary size operands because they can be programmed post-fabrication to implement functions directly as hardware circuits. In such devices, functional units are created which exactly match the bit-widths of the data values on which they compute. Using the special architectural features requires the programmer to use macro libraries or specify the bit-widths manually, a tedious and error-prone process. Furthermore, this is often impossible as there is little or no support in high-level languages for specifying arbitrary bit-widths.
This work was supported by DARPA contract DABT63-96-C-0083 and an NSF CAREER grant.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 969–979, 2000. c Springer-Verlag Berlin Heidelberg 2000
970
Mihai Budiu et al.
In this paper we present the BitValue algorithm, which enables the compilation of unannotated high-level languages to take advantage of variable size functional units. Our technique uses dataflow analysis to discover bits which are independent of the inputs to the program (constant bits) and bits which do not influence the output of the program (unused bits). By eliminating computation of both constant and unused bits the resulting program can be made more efficient. BitValue generalizes constant folding and dead-code elimination to operate on individual bits. When used on C programs, BitValue determines that a significant number of the bit operations performed are unnecessary: on average 14% of the computed bytes in programs from SpecINT95 and Mediabench are useless. Our technique also enables the programmer to use standard language constructs to pass width information to the compiler using masking operations. Narrow width information can be used to help create code for sub-word parallel functional units. It can also be used to automatically find configurations for reconfigurable devices. BitValue has been implemented in a compiler which generates configurations for reconfigurable devices, reducing circuit size by factors of three to twenty. In Section 2 we present our BitValue inference algorithm with an example. Results for the implementation in a C compiler are in Section 3 and for a reconfigurable hardware compiler in Section 4. Related work is presented in Section 5 and we conclude in Section 6.
2
The BitValue Inference Algorithm
For each bit of an arbitrary-precision integer, our algorithm determines whether (1) it has a constant value, or (2) its value does not influence the visible outputs of the program. Those two possibilities are similar to constant folding and dead code elimination, respectively. In our setting, however, these are performed at the bit-level within each word. We can cast our problem as a type-inference problem, where the type of a variable describes the possible value that each bit can have during the execution of the program. The BitValue algorithm solves this problem using dataflow analysis. We represent the bit values by one of: 0, 1, don’t know (denoted by u) and don’t care, (denoted by x). 0 1 Let us call this set of values B. Some bits are constant, U independent of the inputs and control flow of the program; such bits are labeled with their value, 0 or 1. Fig. 1. The bit values A bit is labeled x if it does not affect the output; lattice. otherwise a bit is labeled u. These bit values form a lattice, depicted in Figure 1. We write ∪ and ∩ for sup and inf in the lattice respectively. The top element of the lattice is x and the bottom is u.
X
BitValue Inference
971
The Bit String Lattice. We represent the type of each value in the program as a string of bits. We write B ∗ to denote all strings of values in B. For example, for the C statement unsigned char a = b & 0xf0, we determine that the type of a is uuuu0000, and that the type of b, assuming it is dead after this statement, is uuuuxxxx. The bitstrings also form a lattice L. Space considerations preclude us from giving the formal definition of the operations on this lattice. The ∪ and ∩ operations in L are done bitwise (i.e. ab ∪ cd = (a ∪ c)(b ∪ d)). When applied to strings of different lengths, ∪ gives a result of the shorter length, while ∩ gives a result of the bigger length. The shorter value is sign-extended in the lattice for the ∩ computation.
The Transfer Functions. In order to certify the correctness of our algorithms, we need to prove that our transfer functions are monotone and conservative. For this purpose we provide mathematical definitions for the “best” forward transfer function and for a conservative backward transfer function. We now define A, the forward transfer function of an operator in L. We define the auxiliary “expansion” function exp : L × {x, u} → 2L , which takes a bitstring s and a bit value b ∈ {x, u}, and generates a set of bistrings: all bitstrings that can be obtained from s by replacing the bits in s having the value b by all possible combinations of constant values. For example exp(0ux1x, x) = {0u010, 0u110, 0u011, 0u111}. We now define three auxiliary functions which are used to compute the transfer function of any operator. Ac : (N → N ) × {0, 1}∗ → L operates on “constant” bitstrings, i.e. bistrings containing only 0 and 1. Au : (N → N ) × {0, 1, u} → L computes the transfer function for bitstrings which comprise 0, 1 and u bits. A : (N → N ) × L → L works for bitstrings with any of the digits in B. Given a unary operation f : N → N , A(f, ·) is its associated forward transfer function in L → L. where v ∈ {0, 1}∗ Ac (f, v) = f(value(v)) Au (f, v) = y∈exp(v,u) Ac (f, y) where v ∈ {0, 1, u}∗ A(f, v) = y∈exp(v,x) Au (f, y) where v ∈ L.
The intuition behind these equations is the following: when we compute the transfer function in L for an input value, we can choose arbitrary values for the input bits which are marked x, but we must search the entire space of possibilities for the bits marked u. This definition can be easily extended to deal with n-ary operators. For example, here is what the above definition yields for the C complementation ~ operator when applied to u0x:
972
Mihai Budiu et al.
A(˜, u0x) = Au (˜, u00) ∪ Au (˜, u01) = (Ac (˜, 000) ∩ Ac (˜, 100)) ∪ (Ac (˜, 001) ∩ Ac (˜, 101)) = ((˜000 ∩ ˜100) ∪ (˜001 ∩ ˜101)) = ((111 ∩ 011) ∪ (110 ∩ 010)) = u11 ∪ u10 = u1x The backward transfer function will discover don’t care bits in the input starting from the don’t cares in the output. We do not have a closed form for the backward transfer function. We can, however, define a conservative approximation using techniques from Boolean function minimization [6]. The notion of don’t care input for a Boolean function f of n variables is well known (xi is a ∂f = 0). don’t care if ∂x i We can view an operator which computes many bits (like addition) as a vector of Boolean functions, each computing one bit of the result. An input bit is don’t care for the operator if it is a don’t care for all the functions in the vector whose result is not x. If our analysis discovers that some input bits are constant, we can use those in the backward transfer function computation, starting the computation with the restriction of f to those constant inputs. For example, let us see how the backward propagation operates on the statement c = a^b when we know already that the types of a, b and c are respectively u0, uu and xu; we expect the don’t care of c to be propagated to a and b. The two bits of c are computed by two boolean functions of 4 bits: f0 and f1 : f0 (a0 = 0, a1 , b0 , b1 ) = a0 ^b0 and f1 (a0 , a1 , b0 , b1 ) = a1 ^b1 . Because the bit 1 in the result is x, we only need to look for the don’t cares of f0 which will be the don’t cares of the input. For instance, bit a1 is a don’t care, because f0 |a1 =0 = f0 |a1 =1 , and so is b1 . So the backward propagation proves as expected that the types of the inputs are x0 and xu respectively. In this example the fact that a0 = 0 was not useful to infer more information, but if we change the operator from ^ to &, this information provides the type x for b0 . In practice the transfer functions as given by the above definitions can be expensive to compute, so we resort to using monotone conservative approximations, described fully in [5]. The Dataflow Analysis. We maintain for each value two types: the best type and the current type. The best type is initialized conservatively (⊥) and moves up in the lattice after each pass. The analysis works by alternating forward and backward dataflow passes, terminating when the best type does not change during a pass. Each pass starts by initializing the current type for all the values to , and proceeds to do the dataflow computation; during this computation the current types move down in the lattice until a fixed point is reached. At the end of each pass we update the best type: best = best ∪ current.
BitValue Inference a uuuuuuuu
c uuuuuuuu +
33 110011
uuuuuuuu
a xxuuuuuu
c xxuuuuuu +
33
xxuuxxuu &
&
unsigned char f(unsigned char c, unsigned char a) { unsigned char d; d = (c + a) & 0x33; return (d >> 4) + (d << 2); }
xxuu00uu
00uu00uu d
2 10 <<
100 4 >> 00uu
uu00uu00
xxuu00uu
d
2 <<
uu00uuuu
returned value
uuxxxx
4
>>
uu00uu00
uu +
+ r
973
r uu00uuuu returned value
Fig. 2. A C function and the associated data-flow graph. The types inferred by forward (backward) propagation are shown in the left (righ) figure. We assume that a char has 8 bits.
2.1
Example
We illustrate the algorithm on the code in Figure 2.1 The algorithm begins with the forward pass and examines the first statement. The sources for the first statement are parameters which are defined outside the procedure and thus are set to be all don’t knows, i.e. every bit is significant. c+a from Figure 2 must be computed on 9 bits. The result is truncated to 8 bits of precision because of the definition of char in the underlying language implementation. The masking operation creates a type for d with a combination of constants and don’t knows, 00uu00uu. The left shift in the return statement concatenates 0 bits at the least significant end, while the right shift generates the type 00uu. Using this information, the addition in the return statement infers that the final result has type uu00uuuu. The backward pass uses this information as a starting point. It proceeds to determine which bits of the computation are actually needed. In this example, the right shift indicates that the bottom 4 bits of d are don’t cares, and the left shift indicates that the top 2 bits are don’t cares. Since d is used in two expressions, its useful bits are represented by the ∩ of these two strings. The middle two bits of d have been found to be 0 by forward propagation, and they are not changed. From the & we deduce that the useful bits of the sum a+c are xxuuxxuu. This don’t care information propagates up through the transfer function associated with the plus operation, and the compiler deduces that for both a and c only the bottom 6 bits are significant. During the next forward pass there are no changes and the algorithm terminates. 1
We assume that all computations are carried on 8 bits; a normal C implementation would cast all values to int and back.
974
Mihai Budiu et al.
Table 1. Percent reduction in bitwidth for programs in MediaBench (left) and SpecINT95 (right). We are only counting the most significant bytes. The column labeled “bitv” indicates that only the BitValue was run, “ind” indicates that only loop induction-variable analysis was performed, and “both” indicates both analyses were performed. The results are rounded down; a zero means “less than one percent”. We were unable to profile gcc. Static % Dynamic % Benchmark ind bitv both ind bitv both adpcm e 0 19 19 0 19 19 adpcm d 0 19 19 0 24 24 g721 Q d 1 32 33 4 26 31 g721 Q e 1 32 33 4 25 29 gsm e 1 30 31 7 7 14 gsm d 1 30 31 4 24 24 epic e 0 5 5 0 0 0 epic d 0 3 3 0 4 4 mpeg2 e 0 12 13 24 4 28 mpeg2 d 0 9 10 1 7 8 jpeg e 0 4 4 1 7 9 jpeg d 0 4 4 0 11 11 pegwit e 0 14 15 0 13 13 pegwit d 0 14 15 0 16 16 mesa 0 5 5 0 5 5
3
Static % Dynamic % Benchmark ind bitv both ind bitv both 124.m88ksim 1 22 22 1 19 20 129.compres 2 11 13 0 11 12 099.go 0 6 7 0 2 2 130.li 0 14 14 0 12 12 132.ijpeg 0 5 5 1 10 11 134.perl 0 11 11 0 8 8 147.vortex 0 6 6 0 5 5 126.gcc 0 19 19 * * *
Experiments with a C Compiler
We evaluate our algorithm implemented in SUIF [15] on C programs from MediaBench [8] and SpecINT95 [12]. BitValue is implemented as a work-list based dataflow algorithm starting from def-use chains [14]. Both def-use and BitValue are local analyses. Information from alias analysis or an interprocedural BitValue analysis would improve our results, at a cost of greater compilation time. 3.1
Evaluation
In this section we compare the merits of induction variable analysis, BitValue, and the interaction between them. Induction variable analysis has been used in [13] to compute ranges of values for each variable which is used to reduce the number of necessary bits. We have used a simplified form of this analysis to analyze FORTRAN-style for loops. We only detect values which depend linearly on the loop index. We ran three experiments for each benchmark: the induction-variable analysis only, BitValue only, and both. When we ran both analyses, we first ran the induction-variable analysis, and we fed the bounds derived by it into the initial information for BitValue.
BitValue Inference 100%
100%
adpcm_e
90% 80% 70% 60% 50% 40%
975
g721_d
80%
32 24 16 12 8 4
70% 60% 50% 40%
30%
30%
20%
20%
10%
10%
0%
ijpeg
m88ksim
90%
32 24 16 12 8 4
0%
none
Ind
BitV
Both none
Ind
BitV
Both
none
Ind
BitV
Both none
Ind
BitV
Both
Fig. 3. Percentage breakdown of widths from some programs (dynamic counts).
Depending on the hardware model which exploits the narrow bitwidths, not every constant or don’t care bit can be eliminated. For instance, a constant bit in the middle of a byte cannot be discarded when using subword parallelism. To account for this, the data in Table 1 counts only the most significant bytes as useless. For example, in a 16-bit data item with inferred type x001u001 xxxuuuxx we count no saved bytes because there is a useful bit in each of the bytes. These results underestimate the performance of the algorithm but apply to a wider range of architectures. If we count all the bit savings, we obtain on average an additional 6% reduction. Most often BitValue and the induction-variable analysis complement (or even reinforce) each other: there are benchmarks (e.g., jpeg e) where the “both” count surpasses the sum of the two other counts. In Figure 3 we show the histograms of the data sizes operated upon for some selected benchmarks. The value sizes are rounded up. For each program we present four histograms: one for the original program (with no analysis), one for the induction variable analysis alone, one for BitValue analysis alone, and one for both analyses (induction followed by BitValue). For example, we can interpret the graphs for adpcm e in the following way: the first bar says that about 5% of the values in the original program are 16-bit or less. The fourth bar shows that using BitValue we discover that 16 bits are actually enough for about 30% of the values in the program. We have examined the main sources of reductions to gain insight into the effectiveness of the algorithm. The sources of reduction found by BitValue come from several patterns: (1) the use of shift, bitwise and and or, addition and multiplication by small constants are the most powerful; (2) the propagation of cast information through the backward analysis; (3) array element index computations. A preliminary evaluation of the benefits of discovering narrow values shows that the analysis is important in the context of reconfigurable functional units (RFUs). We used as a target architecture a VLIW processor augmented with an PipeRench-like [7] RFU on the data path. For example, compiling g721 e for the
976
Mihai Budiu et al.
processor+RFU combination without the analysis yields an 18% reduction in the running time. If we optimize the portion mapped to the RFU using BitValue we obtain a 26% reduction in running time. 3.2
Practical Issues
Our implementation of BitValue is fast and scales linearly in practice with program size. The space complexity is linear. We analyze on average 900 lines/second on a PIII @750Mhz, with an untuned implementation. An interesting side-effect of our analysis is that it gives a portable highlevel method for specifing widths: by using a masking operation we can seed the BitValue algorithm. For example, the statement c = c & 0x3c indicates that only the middle 4 four bits of c are useful, and this knowledge is propagated by BitValue throughout the code. Our current implementation runs the induction variable analysis only once and BitValue afterwards. Improvements can be obtained by iterating these analyses until a fixed-point is reached. Future work will investigate the possible gains.
4
Experiments with a Reconfigurable Hardware Compiler
In this section we evaluate the BitValue algorithm as it is used in the DIL compiler [4] which we developed for reconfigurable hardware. The DIL language operates on arbitrary-precision integer data types and does not require the values to be annotated with an explicit width. Because of this there is no baseline for comparing the performance of the algorithm (in C we could compare the reduced sizes with the C type-specified sizes). For evaluation purposes we artificially set the sizes of all variables to 32-bitsand then we run the algorithm to determine the reduction in size. Table 2 shows the amount of hardware required to implement kernels compiled with the DIL compiler. Note that the impact of the analysis is significant: it can decrease the silicon real-estate (and implicitly, decrease the power consumption and decrease the latency of the computation) with a factor between 3 and 20.
5
Related Work
There is a wealth of static and dynamic analyses which suggest that many of the bits computed by a program are useless. Brooks and Martonosi [3] use a simulator to show that for the programs in both SpecInt95 and MediaBench more than half of all integer computations require at most 16 bits of precision. Our compile time analysis proves statically that on average 30% of the widths are 16 bits or less for any input data. They suggest hardware techniques for creating instructions which operate on narrow widths on the fly. The work of Bondalapati and Prasanna is similar, looking
BitValue Inference
977
Table 2. The size of the circuits in bit-operations/8, for two circuit versions: one where all values are 32-bit and one with variable sizes. The percent column shows the remaining size of the circuit after optimizations (the smaller, the better). Program cordic encoder dct fir idea
Description Original Final % 12 stage implementation of Cordic vector rotations 1507 332 23 8-bit Huffman encoder with the code table hardwired 2286 578 26 1-D 8-point Discrete Cosine Transform 366 94 26 FIR filter with 20 taps and 8-bit coefficients 320 123 39 Complete 8 round key-specific International Data En2074 576 28 cryption Algorithm nqueens Evaluator for the n-queens problem on an 8x8 board 144 7 5 over Porter-Duff “over” operator 280 49 18 popcnt Count the number of “1” bits in a 16-bit word. 96 5 6
at dynamically changing functional unit sizes based on dynamically maintained width information [2]. Static techniques for inferring minimum bit-widths using don’t care detection are prevalent in the logic synthesis community, for example [6]. This approach computes satisfiability don’t care sets on a network of Boolean operators. Such an analysis operates at the bit (and not at the word level) and is significantly slower but more precise than our approach. These algorithms are exponential in complexity, and even heuristic methods cannot address benchmarks of the size we are analyzing. Our algorithm has worst-case quadratic complexity. We compared our algorithm to the Synopsis Synplify compiler, a commercial CAD tool, using the DCT benchmark from Section 4. Our analysis runs two orders of magnitude faster and generates circuits within 30% of the size obtained by Synopsis. Most similar to our work is Razdan [11]. His analysis uses a ternary logic of 0, 1 and don’t know (denoted in this paper by x); he also operates on strings of bits, and uses forward and backward analyses. Although he handles loop induction variables for loops with a statically know trip-count, he does not offer a complete solution for handling loop-carried dependences, where a lot of savings can be gained. Babb et al. [1] suggest that width analysis can be performed by determining the maximum values that can be carried on the wires, for example by examining loop bounds. This technique is further investigated by Stephenson et al. in [13]. These techniques are orthogonal to ours. Our analysis would very likely combine well with this technique, because the results of one could be used to seed the starting point of the other one, in the same way we handle induction variables.
6
Conclusions
We have presented BitValue, a compiler algorithm which infers statically the values of the bits computed by a program. Trimming constant bits or unused bits can reduce the width of the computed values, enabling the compiler to use
978
Mihai Budiu et al.
narrow width functional units, which have become available in new architectures (e.g., MMX, reconfigurable functional units, and Application-Specific Instruction Processors). BitValue can be used to analyze both C and DIL programs to significantly reduce the number of bits used to perform computations. We show that BitValue inference can determine that on average 14% of the most significant bytes (and 20% of the bits) computed are unnecessary for programs from MediaBench and SpecINT95. BitValue analysis can reduce the size of the programs synthesized for a reconfigurable architecture between three- and twenty-fold. Finally, using our algorithm we were able to increase the simulated performance of several MediaBench programs by more than 20% when run on a CPU with a reconfigurable functional unit. The algorithm we present is an essential ingredient in developing a compiler which will target sub-word parallel media extensions, low power extensions, or reconfigurable devices.
References 1. J. Babb, M. Rinard, A. Moritz, W. Lee, M. Frank, R. Barua, and S. Amarasinghe. Parallelizing applications into silicon. In IEEE/FCCM Symposium on FieldProgrammable Custom Computing Machines, Napa Valley, CA, April 1999. MIT. 2. K. Bondalapati and V.K. Prasanna. Dynamic precision management for loop computations on reconfigurable architectures. In IEEE/FCCM Symposium on Field-Programmable Custom Computing Machines, Napa Valley, CA, April 1999. Organization: University of Southern California. 3. D. Brooks and M. Martonosi. Dynamically exploiting narrow width operands to improve processor power and performance. In HPCA-5, January 1999. Princeton University. 4. M. Budiu and S.C. Goldstein. Fast compilation for pipelined reconfigurable fabrics. In ACM/FPGA Symposium on Field Programmable Gate Arrays, Monterey, CA, 1999. 5. M. Budiu and S.C. Goldstein. BitValue — Detecting and Exploiting Narrow Bitwidth Computations. Technical Report CMU-CS-00-141, Carnegie Mellon University, June 2000. 6. M. Damiani and G. de Micheli. Don’t care specifications in combinational and synchronous logic circuits. In IEEE Transactions on CAD/ICAS, pages 365–388, 1992. 7. S.C. Goldstein, H. Schmit, M. Moe, M. Budiu, S. Cadambi, R.R. Taylor, and R. Laufer. Piperench: A coprocessor for streaming multimedia acceleration. In Proceedings of the 26th Annual International Symposium on Computer Architecture, pages 28–39, May 1999. 8. C. Lee, M. Potkonjak, and W.H. Mangione-Smith. Mediabench: a tool for evaluating and synthesizing multimedia and communications systems. In Micro-30, 30th annual ACM/IEEE international symposium on Microarchitecture, pages 330–335, 1997. 9. P. Marwedel and G. Goossens, editors. Code generation for embedded processors. Kluwer Academic Press, 1995. 10. A. Peleg, S. Wilkie, and U. Weiser. Intel MMX for multimedia PCs. Communications of the ACM, 40(1):24–38, 1997.
BitValue Inference
979
11. Rahul Razdan. PRISC: Programmable reduced instruction set computers. PhD thesis, Harvard University, May 1994. 12. http://www.specbench.org/osg/cpu95/. 13. M. Stephenson, J. Babb, and S. Amarasinghe. Bitwidth analysis with application to silicon compilation. In Proceedings of the SIGPLAN conference on Programming Language Design and Implementation, June 2000. 14. E. Stoltz, M. P. Gerlek, and M. Wolfe. Extended SSA with Factored Use-Def chains to support optimization and parallelism. In Proceedings Hawaii International Conference on Systems Sciences, Maui, Hawaii, Jan. 1994. 15. R. Wilson, R. French, C. Wilson, S. Amarasinghe, J. Anderson, S. Tjiang, S.-W. Liao, C.-W. Tseng, M. Hall, M. Lam, and J. Hennessy. SUIF: An infrastructure for research on parallelizing and optimizing compilers. In ACM SIGPLAN Notices, volume 29, pages 31–37, December 1994.
General Matrix-Matrix Multiplication Using SIMD Features of the PIII
Douglas Aberdeen and Jonathan Baxter Research School of Information Sciences and Engineering Australian National University [email protected], [email protected]
Abstract. Generalised matrix-matrix multiplication forms the kernel of
many mathematical algorithms. A faster matrix-matrix multiply immediately bene ts these algorithms. In this paper we implement e cient matrix multiplication for large matrices using the oating point Intel SIMD (Single Instruction Multiple Data) architecture. A description of the issues and our solution is presented, paying attention to all levels of the memory hierarchy. Our results demonstrate an average performance of 2.09 times faster than the leading public domain matrix-matrix multiply routines.
1 Introduction A range of applications such as arti cial neural networks bene t from GEMM (generalised matrix-matrix) multiply routines that run as fast as possible. The challenge is to use the CPU's peak oating point performance when memory access is fundamentally slow. The SSE (SIMD Streaming Extensions) instructions of the Intel Pentium III chips allow four 32-bit (single precision) oating point operations to be performed simultaneously. Consequently, e cient use of the memory hierarchy is critical to being able to supply data fast enough to keep the CPU fully utilised. In this paper we focus on the implementation of an e cient algorithm for the Pentium SIMD architecture to achieve large, fast, matrix-matrix multiplies. Our code has been nicknamed Emmerald. Without resorting to the complexities associated with implementing Strassen's algorithm on deep-memory hierarchy machines [5], dense matrixmatrix multiplication requires 2M N K oating point operations where A : M ×K and B : K × N de ne the dimensions of the two matrices. Although this complexity is xed, skillful use of the memory hierarchy can dramatically reduce overheads not directly associated with oating point operations. It is the optimization of memory hierarchy combined with the SSE that gives Emmerald its performance. Emmerald implements the SGEMM interface of Level-3 BLAS, and so may be used immediately to improve the performance of single-precision libraries based on BLAS (such as LAPACK [4]). There have been several recent attempts at automatic optimization of GEMM for deep-memory hierarchy machines, most A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 980983, 2000. c Springer-Verlag Berlin Heidelberg 2000
General Matrix-Matrix Multiplication Using SIMD Features of the PIII
981
notable are PHiPAC [3] and the more recent ATLAS [6]. ATLAS in particular achieves performance close to optimized commercial GEMMs. Neither ATLAS nor PhiPAC make use of the SSE instructions on the PIII for their implementation of SGEMM. Our experiments showed that ATLAS achieves a peak of 375 MFlops/s for single-precision multiplies on a PIII @ 450 MHz, or 0.83×clock rate. Our matrixmatrix multiply using SIMD instructions achieves a peak of 890 MFlops/s, or 1.98×clock rate. We also report a price/performance ratio under USD $1/ MFlop/s for training Neural Networks for Japanese OCR. The following section will describe our novel use of the SSE for Emmerald, followed by a description of optimizations designed to improve use of the memory hierarchy. The paper concludes with a comparison between ATLAS and Emmerald.
2 SIMD Parallelization Two core strategies are employed to minimise the ratio of memory accesses to oating point operations: accumulate results in registers for as long as possible to reduce write backs and re-use values in registers as much as possible. In [2] several dot-products were performed in parallel as the innermost loop of the GEMM. We took the same approach and found experimentally that 5 dot-products in the inner loop gave the best performance. Figure 1(a) shows how these 5 dot products utilise SIMD parallelism. Four values from a row of A are loaded into an SSE register. This is re-used ve times by doing SIMD multiplication with four values from each of the ve columns of B used in the inner loop. Two SSE registers are allocated to loading values from B . Results are accumulated into the remaining ve SSE registers. When the dot-product ends each SSE result register contains four partial dotproduct sums. These are summed with each other then written back to memory. For the best e ciency, the dot product length is maximised with the constraint that all data must t into L1 cache.
3 Memory Hierarchy Optimizations A number of standard techniques are used in Emmerald to improve performance. Brie y, they include: L1 blocking : Emmerald uses matrix blocking [2, 3, 6] to ensure the inner loop is operating on data in L1 cache. Figure 1(b) shows the L1 blocking scheme. The block dimensions m and n are determined by the con guration of dotproducts in the inner loop (Section 2) and k was determined experimentally. Unrolling : The innermost loop is completely unrolled for all possible lengths of k in L1 cache blocks while avoiding over owing the instruction cache. Re-buering : Since B is large (336 × 5) compared to A (1 × 336), we deliberately buer B into L1 cache. By also re-ordering B to enforce optimal memory access patterns we minimise translation look-aside buer misses [6].
982
Douglas Aberdeen and Jonathan Baxter Iteration 1 C 5
6
7
A xmm0
xmm3 4
xmm1 2
B
1
2
1
M
Iteration 2 C 5
6
7
A xmm0
xmm3 4
B
xmm1 2
1
C
A A
C 2
B
1111111111 0000000000 m=1 0000000000 1111111111
1
k=336
N
11 00 00A 11 00 11 00 11
(a) Five dot products using eight SSE registers (xmm[0-7]). Each circle is an element in the matrix. Each dashed square represents one oating point value in an SSE register.
n=5
111 000 000 111 000 111 000 111 000 111 000 111 000 111 000 111 000k 111 000 111 000 111 000 111 000 111 000 111 000 111 000 111 000 111 000 111 000 111 buffered
000 B 111
000B 111 111 000 000 111
(b) L1 blocking for Emmerald: C ← A B where A and B are in L1 and C is accumulated in registers.
Pre-fetching : Values from A are not buered in L1. We make use of SSE pre-
fetch assembler instructions to bring A values into L1 cache when needed. L2 Blocking : E cient L2 cache blocking ensures that peak rates can be maintained as long as A, B and C t into main memory.
4 Results The performance of Emmerald was measured by timing matrix multiply calls with size M, N, K = {16, 17 . . . 700}. To ensure a conservative performance estimate we use wall clock time on an unloaded machine rather than CPU time; the stride of the matrices (which determines the separation in memory between each row of matrix data) is xed to the largest matrix size (700) and caches are ushed between calls to sgemm(). Timings were performed using a PIII 450 MHz with 128 MB RAM, 512 KB L2 cache and a 100 MHz bus. Figure 2 shows Emmerald's performance compared to ATLAS and a naive three-loop matrix multiply. The average MFlop/s rate of Emmerald after size 100 is 1.69 times the clock rate of the processor and 2.09 times faster than ATLAS. A peak rate of 890 MFlops/s is achieved when m = n = k = stride = 320. This represents 1.97 times the clock rate. The largest tested size was m = n = k = stride = 4032 on a 550 MHz machine which ran at 1016 MFlops/s. We have used Emmerald in distributed training of large Neural Networks with more than one million adjustable parameters and a similar number of training examples [1]. By distributing training over 196 Intel Pentium III 550 MHz processors, and using Emmerald as the kernel of the training procedure, we achieved a sustained performance of 152 GFlops/s for a price performance ratio of 98¢ USD/MFlop/s (single precision).
General Matrix-Matrix Multiplication Using SIMD Features of the PIII
983
Emmerald Performance 900
Mflops @ 450MHz
800
700
Emmerald ATLAS naive
600
500
400
300
200
100
0
0
100
200
400 300 Dimension
500
600
700
Fig. 2. Performance of Emmerald on a PIII running at 450 MHz compared to
ATLAS SGEMM and a naive 3-loop matrix multiply. Note that ATLAS does not make use of the PIII SSE instructions.
5 Conclusion This paper has presented Emmerald, a version of SGEMM that utilises the SIMD instructions and cache hierarchy of the Intel PIII architecture. An application demonstrating the cost-eectiveness of such work was also reported. This code and the full version of this paper is available from http://beaker.anu.edu.au/ research.html.
References [1] D. Aberdeen, J. Baxter, and R. Edwards. 98¢ /m op, ultra-large-scale neuralnetwork training on a PIII cluster. Sumbitted to SC2000, May 2000. [2] B. Greer and G. Henry. High performance software on Intel Pentium Pro processors or Micro-Ops to TeraFLOPS. Technical report, Intel, August 1997. http:// www.cs.utk.edu/ ∼ghenry/sc97/paper.htm. [3] J.Bilmes, K.Asanovic, J.Demmel, D.Lam, and C.W.Chin. PHiPAC: A portable, high-performace, ANSI C coding methodology and its application to matrix multiply. Technical report, University of Tennessee, August 1996. http://www.icsi.berkeley.edu/ ∼bilmes/phipac. [4] Netlib. Basic Linear Algebra Subroutines, November 1998. http://www.netlib.org/ blas/index.html. [5] M. Thottethodi, S. Chatterjee, and A. R. Lebeck. Tuning strassen's matrix multiplication for memory e ciency. In Super Computing, 1998. [6] R. C. Whaley, A. Petitet, and J. J. Dongarra. Automated empirical optimizations of software and the ATLAS project. Technical report, Dept. of Computer Sciences, Univ. of TN, Knoxville, March 2000. http://www.cs.utk.edu/ ∼rwhaley/ATLAS/atlas.html.
Redundant Arithmetic Optimizations Thomas Y. Yeh and Hong Wang 1Intel Corporation, Microprocessor Research Lab, 2200 Mission College Blvd., Santa Clara, CA 95052 {Thomas.Y.Yeh, Hong.Wang}@intel.com
Abstract. Redundant arithmetic can be implemented on general-purpose processors to gain significant speedup. Redundant arithmetic speeds up the execution of certain dependent instructions by removing extra work and enabling higher frequencies. However, the issues of worst case delay, scheduling and power must be resolved. We will present the ECS representation format along with initial data and analysis on these critical issues.
1 Introduction In general, redundant arithmetic speeds up dependent arithmetic computations by eliminating unnecessary conversions between intermediate and two's complement forms. Intermediate values during computations are expressed in redundant number representations that use multiple bits to indicate the value of a binary digit. By exploiting the dependency of certain operations, significant speedup can be obtained by removing sub-operations (conversions) and enabling a shorter cycle time. The carryfree characteristic of redundant arithmetic provides a scalable solution for addition of increased data path width. This optimization improves both latency and throughput of the processor [1]. 1.1 Contributions Several issues with implementation on microprocessors have been discussed in [2], [3], and [4]. In this paper, we present the novel concept of extended carry-save optimization to improve the performance of redundant arithmetic. Worst case delay, instruction scheduling and power issues are also discussed. Initial data on potential speedup with respect to in-order and out-of-order machines are presented. All operations for redundant arithmetic considered are integer operations.
2 Worst-Case Delay Redundant arithmetic can be applied to add, subtract, multiply, divide, addressing, compares, and shift left operations while conversion back to two's complement form is A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 984-988, 2000. Springer-Verlag Berlin Heidelberg 2000
Redundant Arithmetic Optimizations
985
required for other instructions including logic, store, shift right, etc. However, the worst-case latency may be longer in a machine where redundant arithmetic is always performed before the conversion. With carry-save representation, this happens when adding two conventional operands when the only dependent instruction needs conventional operands. Conventional Conventional Adder
Bypass
Redundant Processing Integer RegFile
Bypass
Integer RegFile
Redundant Adder
Conventional Adder
Fig. 1. Simple Redundant Arithmetic Implementation Block Diagram
Redundant addition is not necessary with two conventional operands, so the delay is increased. Overall speedup would require a longer chain of dependent instructions. One solution is to detect operand redundancy and optimize with the extended carrysave format (ECS). ECS allows add/subtract operations with 2 conventional operands to directly map the operands to the result bit-vectors while maintaining the same execution latency as redundant addition with 4-2 carry-save adders (CSA). This is due to the fact that a 4-2 CSA can compress 5 inputs of the same weight to a sum, a carry, and an intermediate carry. For the right-most bit position, ECS add/subtract operation with 4-2 CSA is able to compress the maximum of 7 input bits. The details of ECS operations are presented in [5]. N-1 S C
0 C0in
Fig. 2. Extended Carry-Save format (ECS)| N = operand width
To allow for fast addition, the approach is an extension of the radix-2 carry-save format. The direct mapping optimization can be used with the conventional carry-save format, but the C0in bit is needed for direct mapping of subtractions. As a result, the latency for add/subtract operations varies according to the operands' representation. Assuming the ratio of latencies between 64 bit conventional and redundant addition is two, ECS add/subtract operations result in 0 cycle for 2 conventional operands, 1 cycle for one or more redundant operands, and 2 cycles for conversion back to two's complement.
986
Thomas Y. Yeh and Hong Wang
In ECS format, the S and C bit vectors are essentially 2 two's complement bit-vectors of precision N. Conversion requires conventional carry-propagate addition of the bitvectors. To enable the ECS optimization, redundancy detection logic is needed to check the source of the operands.
3 Instruction Scheduling The scheduling of instructions determines the distance in cycle time between dependent instructions. Redundant arithmetic optimizations can potentially change both the latency and the critical path for a given program. For long distances, redundant arithmetic is ineffective because the result of the producing operation could have been converted when the dependent operation is issued. The implications of this observation are different for in-order and out-of-order machines. For out-of-order machines, the hardware dynamic scheduler can readily take advantage of latency changes for different instructions. The reorder buffer size limits the number of instructions that are examined when an instruction produces its result. Our simulation show that increasing the window size increases the percentage of windows with optimized critical paths. This shows that redundant arithmetic is effective in reducing the overall execution critical path. With in-order machines, the compiler-scheduled code can also take advantage of redundant arithmetic. Dependency, operand redundancy, and conversion information would be extra parameters to be tracked by the compiler in generating code. As shown by our previous research using SPECint95, 22-29% of the instruction pairs on the critical path can be optimized by redundant arithmetic. This gives an upper limit on the percentage of critical instructions that can be enhanced with redundant arithmetic. However, existing binaries would require dynamic scheduling capability such as dynamic compilation to realize substantial benefit with redundant arithmetic.
4 Power As discussed in [6], power consumption is a major performance-limiting factor in future processor designs. With redundant arithmetic, the number of signals increases to enable transfer of redundant data to and from the bypass, the register file, and the caches. In addition, the higher clock frequency enabled further increases the power. However, redundant arithmetic can lead to a better performance/power ratio. Redundant adders are regular simple structures compared to complex conventional adders [7]. It increases the effective processing resource with minimal increase in logic. As shown by our simulation, less than 10% of all add/subtract operations on average needs conversion. This enables designs with fewer converters than redundant adders.
Redundant Arithmetic Optimizations
987
5 Simulation Data To evaluate the effects of redundant arithmetic on the critical path, a simulator is developed to capture dependency information on dynamic instruction traces. The result representation of each operation is assigned based on instruction type and operand redundancy. Latency assignment and critical path capture utilizes a directed breadthfirst search of the dependency graph. The assumed machine model dictates the producers and consumers of redundant values. This model is based on past and current researches on redundant arithmetic including [8], [9], [10] and [11]. Producers include add/subtract and post-increment memory accesses while consumers include load/store, compare, and left shift operations. All memory accesses are assumed to hit the first level cache- completing in 1 original cycle. Logic operations complete in 1 redundant cycle.
0.5
1
% of All Instructions
% of Optimized Chains
Window sizes of 50, 100, and 300 were used along with sampling to gather data on the Perl, Go, and Compress benchmarks from the SPECint95 suite. Potential speedup is measured by comparing the longest redundant and. the longest original latencies. For optimized windows, the average speedup ranges from 30-50%.
0.8 0.6 0.4 0.2 0 2
3
4 Length
Fig. 3. Length of Optimized Chains
5+
0.4 0.3 0.2 0.1 0 Compress
Go Perl
Fig. 4. % of Optimized Instructions
Compress shows potential for speedup with the high percentages of optimized compare and memory access instructions. The low percentage of optimized windows may be an effect of the 1-cycle memory access assumption.
Thomas Y. Yeh and Hong Wang
Fig. 5. Optimized Windows
1
Compress50
0.8
Compress100
0.6
Compress300
Perl100 Perl300
Compare
Perl50
0
Ld/St
Go200
0.2
Perl
Go100
0.4
Go
Go50
Compress
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
%
% of All Windows
988
Fig. 6. % of Compare and MEM Optimized
References 1. Srinivas, H.R.; Parhi, E.K.: Computer Arithmetic Architectures with Redundant Number Systems. Conference Record of the Twenty-Eighth Asilomar Conference on Signals, Systems and Computers, Vol. 1 (1994) 182-186 2. Glew, A.: Processor with Architecture for Improved Pipelining of Arithmetic Instructions by Forwarding Redundant Intermediate Data Forms. US Patent #5619664 3. Steiss, D.: Microprocessor Arithmetic Logic Unit using Multiple Number Representations. US Patent #5815420 4. Agarwal, R.; Fleischer, B.; Gustavson, F.: Recurrent Arithmetic Computation using CarrySave Arithmetic. US Patent #5751619 5. Yeh, T.Y.: Integer Addition and Multiplication with Redundant Operands. UCLA 1999 6. Pollack, F.: New Microarchitecture Challenges in the Coming Generations of CMOS Process Technologies. MICRO 1999 Keynote 7. Naffziger, S.: A Sub-Nanosecond 0.5 micron 64b Adder Design. Digest of IEEE International Solid-State Circuits Conference (1996) 362-363 8. Irita; Ogura; Fujishima; Hoh: Microprocessor Architecture Utilizing Redundant-Binary Operation. IEICE Transactions D-I, Vol. J81-D-I 9. Lynch, W.L.; Lauterbach, G.; Chamdani, J.I.: Low Latency through Sum-Addressed Memory. IEEE 1998 10. Lutz, D.R.; Jayasimha, D.N.: Early Zero Detection. 1996 IEEE International Conference on Computer Design: VLSI in Computers and Processors, ICCD '96 Proceedings 545-550 11. Cortadella, J.; Llaberia, J.M.: Evaluation of A+B=K Conditions Without Carry Propagation. IEEE Transactions on Computers, Vol. 41, No. 11 Nov 1992
The Decoupled-Style Prefetch Architecture Kevin D. Rich and Matthew K. Farrens University of California at Davis Abstract. Decoupled processing seeks to dynamically schedule memory accesses in order to tolerate memory latency by prefetching operands. Since decoupled processors can not speculatively issue memory operations, control flow operations can significantly impact their ability to prefetch data. The prefetching architecture proposed here seeks to leverage the dynamic scheduling benefits of decoupled processing while allowing memory operations to be speculatively invoked. The prefetching mechanism is evaluated using the SPEC95 suite of benchmarks and significant reductions in cache miss rate are achieved, resulting in speed-ups of over 40% of peak for most of the inputs.
1
Introduction
Decoupled access/execute (DAE) architectures [6, 8, 7] rely on an intelligent compiler to separate the memory access and execution components of a program into independent instruction streams so they can be executed on separate processors. These processors are loosely coupled through architectural queues, which provide an asynchronous message passing channel. Using the queues to maintain the logical order of instructions, the two instruction streams are able to “slip” with respect to one another, allowing the access processor to lead the execute processor. This effectively migrates data fetching to a earlier time and produces what amounts to dynamic loop unrolling and pipelining. Unfortunately, the strict FIFO nature of the queues limits the performance of decoupled architectures, since they cannot easily exploit branch prediction and speculative execution. In this paper an approach is proposed which makes use of the access instruction stream identification technology developed for the decoupled compiler to create an access stream which will prefetch data into the cache in order to reduce the occurrence of first-reference cache misses (a category of misses not generally addressed by current prefetching techniques.) In the proposed architecture (D-SPA), the main processor (MP) is augmented with a prefetch processor (PFP), whose job is essentially that of decoupling’s access processor - issue memory requests prior to the data being needed by the MP. However, the requirement that the data accesses be correct is removed since the prefetches only bring data into the data cache and do not effect MP state.
2
Background
Memory latency is not a new problem and many prefetching schemes have been explored to address it. Most of these proposals augment a uniprocessor with a A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 989–993, 2000. c Springer-Verlag Berlin Heidelberg 2000
990
Kevin D. Rich and Matthew K. Farrens
simple, parameterizable prefetch-engine. The compiler is responsible for analyzing the source code and identifying regular access patterns that can be described with parameters of the form <start address>, <stride>, and <stop address>. Examples of this include work by Chiueh [3], Chen [2], VanderWiel and Lilja [9]. The prefetch processor of D-SPA differs in that the hardware is considerably more powerful. Instead of having the prefetch engine execute a small set of parameterizable routines, the prefetcher is a fully functional integer-only processor — essentially the access processor from the DAE architecture. Like the access processor, it is loaded with its own executable. The prefetcher relies on the main processor only for flow control information and, when absolutely necessary, some data (function parameters and return values). Unlike the access processor, the prefetch processor only issues non-binding prefetches. The prefetched data is loaded into a cache or prefetch buffer, not into the processor’s register file or a queue. Whether or not the data is ever used is solely a performance issue — it has no impact on program correctness.
3
The Decoupled-Style Prefetch Architecture
D-SPA was designed to take advantage of the compilation techniques developed for decoupled processing, while eliminating the restrictions on the execution model which prevent speculative execution. This paper presents only an overview of the architecture - for a more detailed description of the proposal and it’s effectiveness please see [4]. Conceptually, D-SPA can be seen as a merging of a standard uniprocessor and the DAE architecture. It consists of a uniprocessor (MP) augmented with a prefetching processor (PFP). The PFP employs a subset of the same base RISC-ISA as the MP. The MP performs loads and stores in the usual manner, oblivious to the existence of the PFP. Branches are implemented using the <external branch>/
The Decoupled-Style Prefetch Architecture
991
stack; if this can not be guaranteed at compile time, the store is not included in the PFP executable. (To reduce the limitations on slip, each processor maintains a private run-time stack.) This may result in the PFP producing incorrect memory requests, but since all of the PFP’s prefetches are non-binding these errors will not affect program correctness. The ability to run uniprocessor binaries on D-SPA is achieved via a few minor modifications to the hardware necessary to address the external branch implementation and the addition of the copy queue. In order to run D-SPA binaries on an un-augmented MP the existence of copy operations in the D-SPA binary must be addressed. For a detailed description see [4]. By specifying the architectural and execution models to closely model those of decoupled processing, it is possible to leverage the compilation techniques developed for DAE processing. The most significant task is the identification of those instructions necessary to compute addresses — this must be done in order to generate the PFP instruction stream. Function calls present perhaps the biggest challenge for the compiler. In order to accurately prefetch within a function which accepts parameters, it may be necessary for the PFP to have the parameter values which are calculated by the MP and must be communicated to the PFP via the copy queue. The PFP may also require a copy of the return value of a function in order to generate prefetch addresses (this is identified during def-use analysis). Therefore, the compiler must insert the copy operations necessary to communicate function parameters and return values from the MP to the PFP.
4
Results
The experiments performed use a combination of trace-driven and executiondriven simulation techniques, because it provides a reasonable first-order approximation without requiring extensive compiler modifications or the creation of a special multiprocessor simulator. The cycle-level sim-outorder SimpleScalar simulator was used to simulate the execution of the benchmarks [1]. Sixteen of the eighteen SPEC95 benchmarks were used in order perform the preliminary analyses of D-SPA (two of the eight integer benchmarks were omitted because of incompatibilities with the simulation environment.) Due to disk space limitations, full runs of these benchmarks was not feasible; instead, a “window” of several million instructions was selected on which to perform the simulation, these windows were selected based on work performed by Skadron [5]. Two different cache configurations were employed. The first consisted of an 8K (256 blocks by 8 words/block) direct-mapped, single cycle, L1 cache backed by a 256K (1024 blocks by 16 words/blocks) 4-way set-associative, six cycle, L2 cache. The second consisted of a 16K (512 blocks by 8 words/block) direct-mapped, single cycle, L1 cache backed by a 256K (1024 blocks by 16 words/blocks) 4-way set-associative, six cycle, L2 cache. The small cache sizes were used because the SPEC95 benchmarks generally exhibit very low data cache miss rates. Since the goal of this research is to
992
Kevin D. Rich and Matthew K. Farrens
determine if number of data cache misses can be reduced, there needs to be some data cache misses. Future studies are planned with different applications and more representative cache configurations. The memory latency used was 18 cycles for the first cache-to-main memory bus transfer unit and 2 cycles for each other unit in a single cache-block request. An additional consideration is the number of memory ports available. The default SimpleScalar configuration provides two memory ports; however, preliminary studies in which the two processors shared two ports indicated that performance suffered. The results presented in this chapter use a configuration with three memory ports, with two dedicated to the MP and one dedicated to the PFP. Four different prefetching schemes were investigated. All four issue prefetches for loads of global data. The four possible combinations are derived by including or excluding prefetches for references to stack-based data, and including or excluding prefetches for store operations. The simulations were done using both the 8K and 16K cache configurations described previously. Full details of these simulations are available in [4] - in this paper there is only room for an overview of the results. Ideally the PFP will be running (will “slip”) far enough ahead of the MP to effectively prefetch, but not so far ahead that it adversely impacts cache performance. With unbounded slip, the PFP reduces cache misses (and as a result, achieves a large percentage of the peak speed-up) for many of the benchmarks. However, prefetching results in an increase in the cache miss rate for several others. In order to determine if the behavior of D-SPA on the poor performing benchmarks is in fact attributable to excessive slip, a set of experiments with metered slip was performed. The slip metering approach employed here sets a threshold on the number of outstanding prefetches permitted. When this quantity is hit, the PFP stalls until it drops below the threshold. For these experiments, the slip was metered at 10, 25, 50, and 100 outstanding prefetches. With slip metered at 50, D-SPA achieves over 40% of peak speed-up on ten of the sixteen benchmarks, and between 25% and 40% of peak speed-up on three others. The overall impact of metering slip was substantial. In the case of tomcatv for instance, the negative impact of prefetching can be directly attributed to excessive slip. With unmetered slip, prefetching results in increased execution time. When slip is metered at 50 outstanding prefetches, 40% of peak speed-up is achieved. For applu and hyrdro2d the impact of metering is more dramatic. Metering slip results in over 60% of peak speed-up being achieved for applu, while for hydro2d over 75% of peak speed-up is achieved. Metering has a lesser impact on apsi, fpppp, mgrid, and swim, but in all cases it significantly improves the performance over the unmetered case.
5
Conclusion
In order to exploit the capabilities of decoupled processing, while removing some of the performance constraining restrictions of its execution model, the
The Decoupled-Style Prefetch Architecture
993
decoupled-style prefetch architecture (D-SPA) is proposed. D-SPA shares several characteristics with the decoupled processing model, including the use of branch and copy queues for communication between the processors. Unlike the decoupled model, it does not employ queues in its interface with memory. The elimination of the memory queues, and the use of non-binding prefetches, allows for the use of speculative execution in the PFP, reducing the slip limiting effect of conditional branch operations. By specifying D-SPA’s architectural and execution model to closely resemble that of decoupled processing, the compilation techniques developed for decoupled processing can be employed in a D-SPA compiler. Despite the existence of the branch and copy queues, binary compatibility with an existing RISC instruction set architecture is accomplished with a modicum of effort. D-SPA reduces the restrictions on slip by employing speculative execution and issuing non-binding prefetches. As a result, it may be possible for the PFP to slip too far ahead of the MP, adversely impacting performance. In order to counter this, techniques to constrain, or meter, slip were developed. The experiments performed here have shown that D-SPA has the potential to significantly reduce the frequency of data cache misses. While these experiments are only a first order approximation of the performance of D-SPA, they indicate that further evaluation is warranted.
References [1] Doug Burger and Todd M. Austin. The SimpleScalar tool set, version 2.0. Technical Report 1342, University of Wisconsin-Madison, 1997. [2] Tien-Fu Chen. An effective programmable prefetch engine for on-chip caches. In Proceedings of the 28th Annual International Symposium on Microarchitecture, 1995. [3] T.-C. Chiueh. Sunder: A programmable hardware prefetch architecture for numerical loops. In IEEE, editor, Proceedings, Supercomputing ’94: Washington, DC, November 14–18, 1994, Supercomputing, pages 488–497, 1109 Spring Street, Suite 300, Silver Spring, MD 20910, USA, 1994. IEEE Computer Society Press. [4] Kevin D. Rich. Compiler Techniques for Evaluating and Extending Decoupled Architectures. PhD thesis, University of California at Davis, 2000. [5] Kevin Skadron. Characterizing and Removing Branch Mispredictions. PhD thesis, Princeton University, June 1999. [6] James E. Smith. Dynamic instruction scheduling and the Astronautics ZS-1. IEEE Computer, 22(7):21–35, July 1989. [7] N.P. Topham and K. McDougall. Performance of the decoupled ACRI-1 architecture: the Perfect Club. In Proceedings of High Performance Computing - Europe, 1995. [8] Gary S. Tyson. Evaluation of a Scalable Decoupled Microprocessor Design. PhD thesis, University of California at Davis, 1997. [9] Steven P. VanderWiel and David J. Lilja. A compiler-assisted data prefetch controller. Technical Report ARCTiC 99-05, University of Minnesota, May 1999.
Exploiting Java Bytecode Parallelism by Enhanced POC Folding Model Lee-Ren Ton1, Lung-Chung Chang2, and Chung-Ping Chung1 1
Department of Computer Science and Information Engineering National Chiao Tung University No. 1001, Dashiue Rd., Hsinchu, Taiwan 30056, ROC {lrton, cpchung}@csie.nctu.edu.tw 2 Computer & Communications Research Laboratories Industrial Technology Research Institute Building 51, No. 195-11, Sec. 4, Jungshing Rd., Judung Jen, Hsinchu, Taiwan 31041, ROC [email protected]
Abstract. Instruction-level parallelism of stack codes like Java is severely limited by accessing the operand stack sequentially. To resolve this problem in Java processor design, our earlier works have presented stack operations folding to reduce the number of push/pop operations in between the operand stack and the local variable. In those studies, Java bytecodes are classified into three major POC types. Statistical data indicates that the 4-foldable strategy of the POC folding model can eliminate 86% of push/pop operations. In this research note, we propose an Enhanced POC (EPOC) folding model to eliminate more than 99% of push/pop operations with an instruction buffer size of 8 bytes and the same 4-foldable strategy. The average issued instructions per cycle for a single pipelined architecture is further enhanced from 1.70 to 1.87.
1
Introduction
Internet has become the most feasible means of accessing information and performing electronic transactions. Java [1] is the most popular language used over the Internet owing to its portability, compact code size, object-oriented, multi-threaded nature, and write-once-run-anywhere characteristics. The performance of the stack-based Java Virtual Machine (JVM) [2, 3] is limited by true data dependency. A means of avoiding such a limitation, i.e. stack operations folding, was proposed by Sun Microelectronics [4, 5, 6, 7] with folding capabilities of up to 2 and 4 bytecode. While executing, pre-defined and pre-stored folding patterns are compared with bytecodes in instruction stream sequentially. Consequently, we call this kind of folding as fixed-folding-pattern matching. Other researchers [8, 9, 10] have also proposed the folding method of this fixed-matching type. In our earlier study, we proposed a dynamic-folding-pattern matching named POC folding model [11]. All bytecode instructions are classified into three major POC (Producer, Operator, and Consumer) types. Table 1 lists the POC types and the ‘O’ type is further divided into four sub-types according to their execution behavior. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 994-997, 2000. Springer-Verlag Berlin Heidelberg 2000
Exploiting Java Bytecode Parallelism by Enhanced POC Folding Model
995
Table 1. POC Instruction Types POC P OE OB OC OT C
Description An operation that pushes constant or loads variable from LV to OS An operation that will be executed in execution units An operation that conditionally branches or jumps to target address An operation that will be executed in micro-ROM or trap An operation that will force the folding check to be terminated An operation that pops the value from OS and stores it into LV
Occurrence 47.14% 10.87% 11.54% 22.19% 3.96% 4.29%
In the POC folding model, foldability check is performed by examining each pair of consecutive instructions. By applying the POC folding rules, the two consecutive bytecode instructions may be combined into a new POC type, which is used in further foldability check with the following bytecode instructions. Consequently, the POC folding model is quite different from the fixed-matching one because there is no fixed-instruction-pattern. The POC folding rules can be represented as a state diagram shown in Fig. 1. If the Ps are not consumed immediately by O or C type instructions, they will be issued sequentially.
S ta rt F o ld in g R u le C heck O
P
B
O O
O
C
C
B
O
E
C
B
T
S ta te _ P O
S ta te _ O
O
P
E
S ta te _ O
E
T
O
C C
C
S ta te _ O
P ,O
C
S ta te _ C
P ,O P ,O ,C
P ,O ,C E n d F o ld in g R u le C heck
Fig. 1. Folding Rules for POC Model
2
Enhanced POC Folding Model
The main improvement of the EPOC model over the POC model is the capability of folding the discontinuous Ps. As shown in Fig. 2, the P Counting state will record how much Ps are there before the O or C type instructions. If there is no O or C type instruction in the instruction buffer, the Ps will be issued sequentially to the execution unit like the POC does. The C Counting state will check whether the preceding state is OE state or P Counting state. If the preceding state is OE, C Counting state will fold
996
Lee-Ren Ton, Lung-Chung Chang, and Chung-Ping Chung
the Cs into OE. If it is P Counting state, then Ps are folded into Cs according to the number of Cs. Otherwise, if the C type instruction is the first instruction in instruction buffer, the EPOC issues the C sequentially. Start EPOC Folding Rules Check OE
OC
C OE OT
OB
P
C
OE
P Counting
OC
P, O
OC
P, O, C C
OT
C Counting
OB OB
P, O
P, O, C End EPOC Folding Rules Check, Issue FBI
Fig. 2. Folding Rules for EPOC Folding Model
3
Performance Comparison of Various Folding Models
In Fig. 3, the average percentages of eliminated P + C type instructions for different foldability are shown. Note that the picoJava-II has the foldability of four according to the released specification. We duplicate the picoJava-II’s performance results to each column for comparison only. The issued instructions per cycle (IIPC) for a single pipelined picoJava-II architecture for each model is shown in Fig. 4. Average Percentage of Eliminated P + C Type Instructions
100% 80% 60% 40% 20% 0%
67.85%79.04% 82.71% 61.39%
2-foldable
96.18%
POC Folding Model picoJava-II 85.52% 99.01%
EPOC Folding Model 85.67%99.16%
61.39%
61.39%
3-foldable
4-foldable
61.39%
Max. foldability
Fig. 3. Average Percentages of Eliminated P + C Type Instructions POC-2 POC-Max EPOC-4
IIPC
POC-3 EPOC-2 EPOC-Max
POC-4 EPOC-3 picoJava-II
2.0 1.8 1.6 1.4 1.2 1.0 Assign
BitOps
IDEA
NumSort
StringSort
LU
Linpack
Fig. 4. Issued Instructions per Cycle for Each Folding Model
Average
Exploiting Java Bytecode Parallelism by Enhanced POC Folding Model
4
997
Conclusion
In this research note, we have proposed the EPOC folding model based on the previously proposed POC folding model. The dynamic-folding-pattern matching of POC and EPOC overrides the fixed-folding-pattern matching used in picoJava-II with the folding ratio of 39% and 61%, respectively. The performance enhancement from POC to EPOC folding model benefits mainly from the foldability of discontinuous P type instructions, which results 85.5% and 99.0%.folding ratio using the 4-foldable strategy, respectively. That is, 44% and 50.9% program codes are folded, respectively. The hardware implementation of the recursive EPOC folding model is integrated into decoding stage using parallel priority encoders to generate the source and destination fields of the FBI in constant delay time (non-recursive). Extra circuits must be added to maintain the instruction buffer after folding. For a superscalar Java processor of our current research, the EPOC folding model is integrated with the stack reorder buffer. Furthermore, the source-ready FBIs can be issued in parallel to exploit higher ILP.
References 1. James Gosling, Bill Joy and Guy Steele: The Java™ Language Specification, AddisonWesley, Reading MA (1996) 2. Tim Lindholm and Frank Yellin: The Java™ Virtual Machine Specification, AddisonWesley, Reading MA (1996) 3. Venners, B.: Inside the Java Virtual Machine,McGraw Hill, New York (1998) 4. M. O’Connor and M. Tremblay: picoJava-I: The Java Virtual Machine in Hardware, IEEE Micro, Vol. 17, No. 2, (1997) 45-53 5. H. McGhan and M. O’Connor,: picoJava: A Direct Execution Engine for Java Bytecode, IEEE Computer, (1998) 22-30 6. Sun Microsystems Inc.: picoJava-II Microarchitecture Guide, Sun Microsystems, CA USA (1999) 7. Sun Microelectronics: microJava™-701 Processor,” http://www.sun.com/microelectronics/ microJava-701/ 8. Han-Min Tseng, et. al.: Performance Enhancement by Folding Strategies of a Java Processor, Proceedings of International Conference on Computer Systems Technology for Industrial Applications – Internet and Multimedia, (1997) 9. Lee-Ren Ton, et. al.: Instruction Folding in Java Processor, Proceedings of the International Conference on Parallel and Distributed Systems, (1997) 10.N. Vijaykrishnan, N. Ranganathan and R. Gadekarla: Object-Oriented Architectural Support for a Java Processor, Proceedings of the ECOOP’98, Lecture Notes in Computer Science, Springer Verlag, (1998) 11.L. C. Chang, L. R. Ton, M. F. Kao and C. P. Chung,: Stack Operations Folding in Java Processors, IEE Proceedings on Computer and Digital Techniques, Vol. 145, No. 5, (1998)
Cache Remapping to Improve the Performance of Tiled Algorithms Kristof E. Beyls and Erik H. D’Hollander University of Ghent Department of Electronics and Information Systems St.-Pietersnieuwstraat 41 B-9000 Gent, Belgium
Abstract. With the increasing processing power, the latency of the memory hierarchy becomes the stumbling block of many modern computer architectures. In order to speed-up the calculations, different forms of tiling are used to keep data at the fastest cache level. However, conflict misses cannot easily be avoided using the current techniques. In this paper cache remapping is presented as a new way to eliminate conflict as well as capacity and cold misses in regular array computations. The method uses advanced cache hints which can be exploited at compile time. The results on a set of typical examples are very favorable and it is shown that cache remapping is amenable to an efficient compiler implementation.
1
Introduction
With Moore’s Law still doubling the performance in 18 months, there is almost no limit to the processing power for the foreseeable future. Many performance programmers know that this is not the case, due to the speed gap between the processor and the memory. In fact, where the processor speed gains about 67% per year, the memory lags behind with only a gain of about 5-10%[9]. Using the same reasoning as Moore, one quickly finds out that a similar law says the memory speed with respect to the processor halves each 22 months... . From this observation, the growing importance of L1, L2 and L3 caches becomes evident and the objective is to keep the used data in the cache all the time. Tiling[1, 15] is a well known method to improve the reuse of cached data in numerical applications by shortening the distance between the use and reuse of array elements. Tiling algorithms successfully eliminate capacity misses and therefore increase the cache hit ratio. However, the low associativity of caches may lead to a high number of conflict misses and slow down execution so that only a fraction of the attainable performance is obtained. Additional fine tuning of the tiling transformation is needed to reduce the conflict misses[2, 6, 8, 11, 13].
Research financed by the Flemish government under contracts IWT-SB/991147 and GOA-12.0508.95)
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 998–1007, 2000. c Springer-Verlag Berlin Heidelberg 2000
Cache Remapping to Improve the Performance of Tiled Algorithms
999
In this paper, cache remapping is offered as a new technique to eliminate conflict misses in tiled algorithms. In addition, cache remapping produces no capacity misses and also cold misses are avoided for all but the first iteration. Cache remapping is based on a dynamic rearrangement of the data at run time. During the execution of a loop, a parallel remap thread running concurrently with the original program thread relocates the tiled data needed by future iterations. The cache is split in two regions. One region contains all the data in the tile currently being processed, enabling the calculations to continue without memory stalls. At the same time the remap thread copies the data of the next tile into the other cache region, using proper address relocation and cache bypass. When the calculations have completed processing a tile, the original processing thread can immediately continue processing the next tile as it is already brought in the cache by the remap thread. Section 2 explains cache remapping in detail. In section 3, experimental results are presented. Section 4 compares the techniques and the results with related work.
2 2.1
Cache Remapping Technique Tiled Loop Nests
Fig. 1 shows a loop nest and the corresponding tiled loop. Definition 1. Tiling[15] transforms an n-deep loop nest into a 2n-deep loop nest. The tiled loops in the resulting tiled loop nest are the n inner loops. The tiling loops are the n outermost loops. A loop nest will be notated as L. The tiled and the tiling loops for L are Td(L) and Ti(L) respectively. An iteration tile is the iteration space traversed by Td(L). The part of an array that is referenced during the execution of an iteration tile is a data tile. A tile set is the union of the data tiles of all the arrays accessed during an iteration tile execution.
do i=1,N,1 do j=1,N,1 do k=1,N,1 H(i,j,k) (a) Original loop nest
Ti(L) L
do II = 1,N,B1 do JJ = 1,N,B2 do KK = 1,N,B3 do i = II,min(II+B1-1,N),1 do j = JJ,min(JJ+B2-1,N),1 Td(L) do k = KK,min(KK+B3-1,N),1 H(i,j,k) (b) Tiled loop nest
Fig. 1. A tiled loop nest
1000
2.2
Kristof E. Beyls and Erik H. D’Hollander
Cache Memory
For the development of the cache remapping technique, a cache is represented by a tuple (Cs , Ns , k, Ls )[3] Definition 2. The cache size (Cs ) defines the total number of bytes in the cache. The line size (Ls ) determines how many contiguous bytes are fetched from memory on a cache miss. A memory line refers to a cache-line-sized block in the memory which is aligned such that the data in it map into the same cache line. A cache set is the collection of cache lines in which a particular memory line can reside. Ns denotes the number of cache sets in a cache. Associativity (k) refers to the number of cache lines in a cache set. These parameters are related by the equation Cs = Ns × k × Ls . The start address A of a memory line determines the cache set N it maps to: A (1) mod Ns N= Ls A replacement algorithm decides the cache line in set N a memory line is copied to. In the rest of this paper the least recently used (LRU) replacement policy is assumed. Definition 3. Consider the memory lines accessed during the execution of L. Then Ml(L, N ) represents the set of memory lines which map to cache set N . 2.3
Conflict Misses in Tiled Algorithms
Consider a tiled loop Td(L) and a cache set N . When #Ml(Td(L), N ) > k, more than k memory lines must be placed in the same cache set N . Only part of the memory lines can reside in the cache at the same time, and conflict misses arise. Cache remapping copies tile sets into a contiguous Cs -sized buffer. Because of (1), k memory lines in the buffer map to each cache set. So, ∀N ∈ Ns : #Ml(Td(L), N ) = k and no conflict misses arise. 2.4
High-Level View of Cache Remapping
Cache remapping adds a remap thread to the program, which executes concurrently with the original processing thread executing the tiled loop nest L (see fig. 2). These two threads can execute in parallel on processors with multiple functional units. (for further detail, see sect. 2.5). Consider an iteration point i of Ti(L). The two threads work in a pipelined way(see fig. 3): – The processing thread executes tile i. – The remap thread copies data tile set i + 1 into the cache. If there is written data of tile set i − 1 in the cache, it is first copied back to main memory to make place for tile set i + 1. At most two consecutive tile sets are in the cache at the same time. Between iterations of Ti(L), the two threads synchronize.
Cache Remapping to Improve the Performance of Tiled Algorithms
Remap Thread
000 111 current tile set000 111 111 000
processor
Processing Thread
1001
next tile set scalar area
1111 0000 0000 1111 000 111 000 111 000 111
cache
00000000 11111111 00000000P1 11111111 11111111 00000000 00000000 11111111 P2 11111111 00000000 11111111 00000000
main memory
111111111111111 000000000000000 111111111111111 000000000000000
P3
1111 0000 0000 0011 11111100 1100
Fig. 2. The remap thread puts the next tile set in the cache while the original thread processes the current tile set. In the next phase, the processing and the remap thread will access P3 and P2 respectively.
Fitting the Current and Next Tile Set in the Cache The process thread accesses two kinds of variables in the memory: scalar variables which do not fit into the registers and arrays. To ensure that all data referenced by the process thread is in the cache, it is logically divided in three partitions: P1 , P2 and P3 . P1 is used to cache the scalars. P2 and P3 will each contain one tile set. During the odd iterations of Ti(L), the process thread uses P2 , during the even iterations, it accesses P3 . The remap thread uses P3 during the odd iterations and P2 during the even iterations. It is clear that P2 and P3 must have the same size as they are used symmetrically. Respecting Data Dependencies Problems arise when there are data dependencies between the tile sets of two consecutive iteration tiles. If the process thread currently processes tile set i and writes into elements of tile set i + 1, the remap thread prefetches these elements into the cache with the old values. When the process thread executes tile i + 1, it will use the old values instead of the new. A solution is to copy the new value of the shared elements to the cache partition the process thread will use. This must be done during the thread synchronization, which occurs between iterations of Ti(L).
1002
Kristof E. Beyls and Erik H. D’Hollander
111 0 0110 000 0 1 00 111 10 0 11 0 1 0 0 1 00 11 111 000 11 1010 01 1 01010 1010 10 1010 10 10 10
0 1 11 00 111 000 1111 000 00 11 0 00 11 0 1 1111111111111111 0000000000000000
remap thread: copy tile process thread: calculations remap thread: put back modified data
time
execution time of 1 iteration of Ti(L) Fig. 3. The pipelined nature of cache remapping
2.5
Low-Level Details
Controlling Cache Behavior Cache Shadow At the start of the program, a consecutive block of memory with size Cs is allocated and aligned on a memory line. We call this memory block the cache shadow. There’s a one-to-one relation between the addresses in the cache shadow and the storage area in the cache. The area’s P1 , P2 and P3 are allocated in this cache shadow. To assure that the contents of the cache shadow always resides in the cache completely, cache hints are used. They make it possible to only cache the addresses in the cache shadow by bypassing the cache on memory references outside the cache shadow. Cache Hints In modern instruction sets, cache hints[4, 5] are attached to load and store instructions. They specify if the referred data should be cached or not. When data is loaded/stored from/to P1 , P2 or P3 , a cache hint tells the processor to cache the data. If an address outside the cache shadow is referenced, the cache hint tells the processor not to cache the data. Thread Scheduling on a Single Threaded Superscalar Processor The process and remap threads need to run concurrently. Current microprocessors offer parallelism at the instruction level (ILP). This means that only nearby instructions in one thread can be executed simultaneously. To execute the remap thread and the process thread concurrently, these two threads need to be interwoven into a single thread at compile time. The instructions of the two threads must be interleaved so that the processor can execute instructions of the two threads during the same cycle. On current processors, about a dozen functional units are present. Typically, data dependencies cause an average IPC (instructions per clock-cycle) no more than 2 to 3, so about ten functional units are left unused every clock-cycle. There are no dependencies between the remap and the process thread during the execution of Td(L). As a result, the remap thread can use the functional
Cache Remapping to Improve the Performance of Tiled Algorithms
1003
units that are not used by the process thread. A good optimizing compiler can schedule the instructions of both threads so that they execute simultaneously. Non-Stalling Memory Access The remap thread accesses main memory. Because the two threads are interwoven into one, it is important that the memory access doesn’t stall the processor. When an instruction from the remap thread accesses main memory, there are enough independent instructions from the process thread ahead in the instruction stream to perform useful in-cache computations to overlap the latency. Selection of Tile Size The tile size (B1 ,. . .,Bn ; n = 3 in the example) is chosen so that every tile set fits in P2 and P3 . A large number of tile sizes satisfy this constraint. Let iterp = B1 ∗ · · · ∗ Bn , the number of iterations executed by the process thread during a tile execution. Let iterr be the number of array elements that need to be remapped or put back during a tile execution. We iter choose to optimize the tile sizes so that the ratio r = iterpr is maximal. This choice assures that the processing power needed by the remap thread is as small as possible relative to the processing power needed by the process thread. Loop Transformations and Thread Scheduling To lower the scheduling overhead, a number of loop transformations are performed to the loop nests in both threads. The remap thread originally consists of Q loop nests. Every loop nest remaps or puts back a data tile. Qri is the number of elements that are remapped by loop nest i. Each of these loop nests are coalesced[10], and the body of the remaining loop is placed in an inlined remapping function (e.g. remapA in fig. 4(a)). The innermost loop in the tiled loop nest Td(L) is unrolled r times, then a remap call is inserted (see fig. 4(c)). It is known at compile time how many times each remap function must be executed per iteration of Td(L). The outermost loop of Td(L) is split into Q parts. In each part, another remap function is called. The number of iterations in each part is chosen sothat every remapping function is called at least Qri B3 1 ≥ Qri . times. So QB i ∗ B2 ∗ r
3 3.1
Implementation and Results Processor Requirements
Three conditions must be met to enable cache remapping: 1. the processor provides the possibility to load data from main memory without bringing it into the cache, e.g. using cache hints, 2. multiple instructions execute concurrently, e.g. a superscalar processor, 3. the processor does not stall on a cache miss, as long as independent instructions are available in the instruction stream. This can be achieved using speculative loads[4]. The IA-64 architecture satisfies these requirements as well as all Explicitly Parallel Instruction Computing (EPIC)-style architectures.
1004
Kristof E. Beyls and Erik H. D’Hollander
remapA(int iter,A,p) { i1 = de_coalesce(iter); i2 = de_coalesce(iter); remap(p+i1*B2+i2, A[i1+II,i2+JJ]); } (a) One of the Q functions that remap one element remap(double* x, double* y) { fld.nta r1,y fst.t1 x,r1 } (b) The remap function. The nta cache hint means “don’t cache”, the t1 cache hint means “cache into L1”.
swap(p2,p3) iter=0 1 do i = II,II+QB 1 -1 do j = JJ,JJ+B2-1 do k = KK,KK+B3-1,r H(i,j,k,p2) /* body r */ ... /* times unrolled */ H(i,j,k+r-1,p2) /* remap code */ remapA(iter++,A,p3) iter=0 B1 B1 1 do i = II+QB 1 ,II+Q1 + Q2 -1 do j = JJ,JJ+B2-1 do k = KK,KK+B3-1,r ... remapB(iter++,B,p3) ... (c) The transformed tiled loop nest.
Fig. 4. The program transformations to efficiently interweave and schedule both threads into one. p2 and p3 are the start addresses of P2 and P3 respectively. It is assumed that — after inlining — the instruction scheduler will move enough independent instructions between both instructions in remap to allow useful computations during the main memory access.
3.2
Simulation
Since EPIC processors are not yet available, the Trimaran simulator[14] was used to simulate the behavior of the processor. The cache behavior was modeled by the well known Dinero cache simulator. The experiment is a tiled matrix multiply executed on a processor with a 2-level cache. The L1 cache is 16Kb direct mapped with 32 byte lines. The L2 cache is a 256Kb 4-way set associative with 64 byte lines. We assume that the access latency of the L2 cache is 20 clock cycles and the access latency of the main memory is 65 clock cycles. The cache remapping technique was compared with the original algorithm, a naively tiled algorithm not considering limited cache associativity and three optimized tiling algorithms, namely padding[8], copying[13] and LRW[6]. Each algorithm was coded, compiled and simulated for matrix dimensions between 20 and 400. For the cache remapping algorithm, the tiles on the border of the
Cache Remapping to Improve the Performance of Tiled Algorithms
1005
Performance (Flop/Clock Cycle)
0.3 0.25 0.2
detail: see fig.6
0.15 0.1 0.05 0
0
50
100
150
200
250
300
350
400
Matrix Dimension cache remapping original
padding copying
LRW naively tiled
Fig. 5. Smoothed plot of the performance of several tiled matrix multiplications for dimensions 20 to 400. In this smoothed plot it is clear that the cache remapped algorithm outperforms the others for matrix sizes bigger than 150. A zoom of the actual performance plot can be found in figure 6.
Performance (Flop/Clock Cycle)
0.3
0.28
0.26
0.24
0.22
0.2 200
250 cache remapping padding
300 Matrix Dimension
350
400
copying LRW
Fig. 6. The performance of cache remapping, padding[8], copying[13] and LRW[6] on matrix dimensions 200 to 400. The cache remapped algorithm has the same performance as the next best algorithm at worst. At best, a speedup of 10% over the next best algorithm is obtained.
1006
Kristof E. Beyls and Erik H. D’Hollander
iteration space were processed using the copying technique, because the pipelined nature of cache remapping suffers from processing tiles not completely filled with data. The performance of the algorithms, expressed in number of floating point operations per clock cycle, is plotted in figure 5. Because the performance of some algorithms fluctuate, the data was smoothed using bezier curves to clearly visualize the trends. In figure 6 an exact plot is given for the four best algorithms for matrix dimensions 200 to 400. This plot shows that cache remapping always performs at least as good as the next best algorithm. At best, it yields a 10% speedup over the next best algorithm. For matrix dimensions bigger than 150, cache remapping outperforms the alternative tiled algorithms. For matrix dimensions between 200 and 400, the average speedup compared to the second best algorithm (copying) is 5%. Compared with the original non-tiled algorithm, an average speedup of 4.5 is obtained.
4
Comparison with Related Work
Methods that select tile sizes to eliminate conflict misses[2, 6] sometimes result in small tiles, which reduce the performance. Padding[8] on the other hand uses large tile sizes and changes the data layout of the arrays by enlarging the dimensions with unused elements, in order to avoid cache conflicts. Unfortunately, this static adjustment cannot be optimized for every loop in a program simultaneously. Copying[13, 6, 11] eliminates conflict misses by copying the array tiles with the worst self interference to a contiguous buffer. Copying naturally involves overhead and the tradeoffs between copying and cache conflicts are discussed in [13]. In contrast to Padding and Tile Size Selection, cache remapping is independent of the array dimensions and doesn’t require a change of the data layout. With respect to copying, cache remapping is able to cache tiles in a parallel thread, which runs concurrently with the processing thread. As a consequence, cache remapping has no conflict misses and incurs a minimal overhead. The cache bypass and relocation technique was exploited by Lee[7] to use the cache as a set of vector register on i860 processors mimicking Cray’s strided get/put[12]. Yamada[16] proposed prefetching and relocation by extending the hardware with a special data fetch unit which enables prefetching strided data without cache pollution. Our technique also combines cache bypass and relocation, but isn’t limited to strided data patterns which allows it to prefetch and relocate data structures with non-constant strides such as data tiles.
5
Conclusion
The Von Neumann bottleneck nowadays hinders even a single processor. Cache remapping represents a promising technique to bridge the steadily growing gap between processor and memory speeds. It favorably compares with existing tiling
Cache Remapping to Improve the Performance of Tiled Algorithms
1007
techniques and it uses the concepts of a new generation of processors. In future work the presented technique will be embedded in a EPIC compiler.
References [1] S. Carr and K. Kennedy. Compiler blockability of numerical algorithms. In Proceedings, Supercomputing ’92, pages 114–124. IEEE Computer Society Press, November 1992. [2] S. Coleman and K. McKinley. Tile size selection using cache organization and data layout. In SIGPLAN’95: conference on programming language design and implementation, pages 279–290, June 1995. [3] S. Ghosh. Cache Miss Equations: Compiler Analysis Framework for Tuning Memory Behaviour. PhD thesis, Princeton University, November 1999. [4] Intel. IA-64 Application Developer’s Architecture Guide, May 1999. [5] G. Kane. PA-RISC 2.0 architecture. Prentice Hall, 1996. [6] M. S. Lam, E. E. Rothberg, and M. E. Wolf. The cache performance and optimizations of blocked algorithms. In Proceedings of the 4th International Conference on Architectural Support for Programming Languages and Operating Systems, Palo Alto, California, pages 63–74, April 1991. [7] K. Lee. The NAS860 library user’s manual. Technical report, NASA Ames Research Center, Moffett Field, CA, March 1993., 1993. [8] P. Panda, H. Nakamura, N. Dutt, and A. Nicolau. Augmenting loop tiling with data alignment for improved cache performance. IEEE transactions on computers, 48(2):142–149, Feb 1999. [9] D. Patterson. A case for intelligent RAM. IEEE Micro, 17(2):34–44, March-April 1997. [10] C. D. Polychronopoulos. Loop coalesing: A compiler transformation for parallel machines. In International Conference on Parallel Processing, pages 235–242, Pennsylvania, Pa, USA, Aug. 1987. Pennsylvania State Univ. Press. [11] G. Rivera and C.-W. Tseng. A comparison of compiler tiling algorithms. In 8th International Conference on Compiler Construction (CC’99), March 1999. [12] S. L. Scott. Synchronization and communication in the T3E multiprocessor. In Proc. ASPLOS VII, Cambridge, MA, Octobe 1996. [13] O. Temam, E. D. Granston, and W. Jalby. To copy or not to copy: A compiletime technique for assessing when data copying should be used to eliminate cache conflicts. In IEEE, editor, Proceedings, Supercomputing ’93, pages 410–419, March 1993. [14] Trimaran. The Trimaran Compiler Research Infrastructure for Instruction Level Parallelism. The Trimaran Consortium, 1998. http://www.trimaran.org. [15] M. E. Wolf and M. S. Lam. A data locality optimizing algorithm. In Proceedings of the ACM SIGPLAN ’91 Conference on Programming Language Design and Implementation, pages 30–44, 1991. [16] Y. Yamada, J. Gyllenhaal, G. Haab, and W. mei Hwu. Data relocation and prefetching for programs with large data sets. In Proceedings of the 27th Annual International Symposium on Microarchitecture, pages 118–127, San Jose, California, Nov. 30–Dec. 2, 1994. ACM SIGMICRO and IEEE Computer Society TC-MICRO.
Code Partitioning in Decoupled Compilers Kevin D. Rich and Matthew K. Farrens University of California at Davis
Abstract. Decoupled access/execute architectures seek to maximize performance by dividing a given program into two separate instruction streams and executing the streams on independent cooperating processors. The instruction streams consist of those instructions involved in generating memory accesses (the Access stream) and those that consume the data (the Execute stream). If the processor running the access stream is able to get ahead of the execute stream, then dynamic pre-loading of operands will occur and the penalty due to long latency operations (such as memory accesses) will be reduced or eliminated. Although these architectures have been around for many years, the performance analyses performed have been incomplete for want of a compiler. Very little has been published on how to construct a compiler for such an architecture. In this paper we describe the partitioning method employed in Daecomp, a compiler for decoupled access/execute processors.
1
Introduction
Program execution can be viewed as a two-part process — the moving of data to and from memory and the performing of some operation on that data. Conceptually, these two steps can be represented by two cooperating processes, the memory access process and the computation (or execute) process. A decoupled architecture seeks to achieve high performance by running these two processes on separate cooperating processors, allowing out-of-order execution between the two instruction streams. The processors both traverse the same dynamic flow graph, though not necessarily at the same pace. To allow the processors to execute different portions of the graph, architecturally visible queues are employed to buffer information produced by one process for consumption by the other. If the access process can run sufficiently ahead of the execute process on this flow graph then it will dynamically preload the operands consumed by the execute process and hide the latency of memory access operations. If this occurs it is said that access process has slipped with respect to the execute process, or that slip has been achieved. The execution of the two processes on separate processors provides a simple mechanism for supporting limited out-of-order execution, and the ability to issue more than one instruction per cycle. Because of its simplicity and potential ability to tolerate long memory latencies, there is a continuing interest in decoupled architectures [8, 9, 10, 3, 12]. In particular, decoupled architectures A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1008–1017, 2000. c Springer-Verlag Berlin Heidelberg 2000
Code Partitioning in Decoupled Compilers
1009
may be of great interest in the growing field of power-conscious processor design because the simplified method of exploiting ILP does not require many of the large, power-hungry circuits needed by superscalar designs. While decoupled processing has existed in various incarnations for years [7, 10, 1, 9] there has been very little published on the compilation techniques necessary. The most fundamental compilation issue is how the instructions/operations will be allocated or assigned to the access and execute processors. This process is referred to as the partitioning of the code. In the rest of this paper we describe the partitioning scheme employed by Daecomp, an ANSI-C compiler we developed to allow for a more complete analysis of decoupled processing. An example of actual compiler output will be shown, as will some simulation results obtained using Daecomp-produced code.
2
Background
Despite the appearance of decoupled architectures in the literature for many years, very little information is available regarding the techniques necessary for a decoupled compiler. At least two functioning decoupled compilers exist that we are aware of, but both were for commercial products (the ZS-1 [7] and the ACRI-1 [9]) and therefore details of the compiler construction have not been published. The research most closely related to the work we are presenting here was performed by Topham et al. [8] in which they investigate source-code level transformations that can be made for the ACRI-1 with the goal of reducing the frequency of AP-EP synchronization. Our work focuses on the lower-level partitioning task performed by the compiler.
3
Processor Model
The two cooperating instruction streams produced by a decoupled compiler will execute on two separate processing elements, which communicate in a message passing manner via architecturally visible queues. A block diagram of the target architecture is shown in Figure 1. Memory addresses are sent to memory via the Load and Store Address Queues (LAQ,SAQ), and memory operands are received or sent via the Load and Store Data Queues (LDQ,SDQ). In addition to the LAQ-LDQ and SAQ-SDQ queue pairs, each processor in this model has a complete set of alternate memory queues. These queues allow the processor to perform loads and stores on its own behalf (self-loads and self-stores). There are also Copy Queues (to allow data transfer between the processors) and Branch Queues (to provide control flow synchronization). Use of these queues allows instructions from the two instruction streams to slip with respect to one other, providing dynamic scheduling and out-of-order issue capabilities without requiring the architectural complexity of a superscalar processors instruction window. Decoupling seeks to tolerate memory latency via this dynamic scheduling, in particular via the early issue of memory operations.
1010
Kevin D. Rich and Matthew K. Farrens
ASAQ
ASDQ
I-Cache
ALAQ
Branch Queues
I-Fetch Unit
LDQ ALDQ
I-Cache
SDQ
SAQ
LAQ
ALDQ
ASDQ ALAQ
ASAQ
Main Memory or Cache Interface
I-Fetch Unit
Copy Queues
Processing Unit
Access Processor
Processing Unit
Execute Processor
Fig. 1. Decoupled Access/Execute Processor - Conceptual Diagram
4
The Compiler
Daecomp is based on cbend, a multi-threaded compiler for the Concurro architecture produced by Bernard K. Gunther. The multi-threaded aspects of cbend were stripped out and the code specific to decoupled compiling was added. In addition, cbend’s register allocator and code emission routines underwent modifications, primarily to deal with the use of queues. Aside from partitioning the code, perhaps the biggest challenge encountered during compiler construction was handling function calls efficiently and correctly. Since the compiler is responsible for parameter set-up on the stack and/or in the parameter registers (each processor has its own register file), one major design decision is whether each processor will have a private copy of the run-time stack or if a single stack will be shared. The decision impacts how function calls are managed, requiring different protocols for parameter passing and different considerations for local (i.e., stackbased) variable management. Passing parameters to a function under a single shared stack model is simpler than under a private stack model, but since the stack based variables are a shared resource there are a variety of potential performance ramifications. For performance reasons a dual-stack model was initially explored — unfortunately it proved to have a fatal flaw related to passing pointers to pointers so the partitioning technique detailed here assumes a single, shared run-time stack.
5
Code Partitioning
The task of the compiler is to partition a directed acyclic graph into two separate, cooperating, decoupled instruction streams based on def-use information. The graph consists of nodes, which represent the instructions/operations to be performed, and edges, which convey the dependencies between nodes. The goal is to produce a partitioning that reasonably balances the processor workload and
Code Partitioning in Decoupled Compilers
1011
makes it possible for the two instruction streams to slip with respect to each other. In the partitioning technique employed by Daecomp, each operation is assigned to a processor based on the class of operation (e.g., address generation, function parameter set-up, etc.) to which it belongs. To start the marking process operations like loads and stores are designated as anchors. Anchor nodes are either sinks or sources of expressions. Sink nodes represent operations (e.g., stores) upon which no other operation directly depends, while source nodes represent operations which do not depend directly on any other operation (e.g., dequeuing a data value). Once the anchors are identified, the graph is again traversed, this time assigning (or marking1 ) instructions which depend on an anchor (or on which an anchor depends) to the same processor as the anchor. Interprocessor data dependencies are handled by inserting copy instructions from one processor to the other where necessary. Once the partitioning (detailed below) is completed the compiler has created two graphs, one for each processor. Register allocation and code emission are performed on each graph resulting in the decoupled object code file. The fivestep partitioning process will now be detailed. Step 1: Anchor Return Value Usage To minimize inter-processor copies it is desirable to anchor a function’s return value calculation on the processor which is going to use the value. Unfortunately, a function may be called from several different locations within the program (or programs, in the case of a library routine) and the return value may be needed on different processors depending upon the calling location. Additionally, it is unlikely the compiler will have any information regarding the caller(s) of the function that it is compiling. Therefore, it must be determined a priori which is the return value processor. The node which puts the return value into the return value register is marked for the return value processor and serves as an anchor. The compiler currently requires all return values to reside on the EP2 . Step 2: Split Loads and Stores The splitting of memory accesses into address and data portions lies at the heart of decoupled processing. Load operations are split into two parts: the operation that enqueues the load address onto the load queue, and the operation that dequeues the loaded data from the load data queue. Stores are similarly split into the operation which enqueues the store address and the operation which enqueues the store data. 1 2
A node which is marked has been assigned to a processor. A node which has not yet been assigned to a processor is unmarked. In the future, the compiler will be modified to handle return values more intelligently. For a function which does not return a pointer it will assume that the value computed is needed by the EP and thus the EP will be the return value processor. Pointer valued functions are most likely computing a value related to an address calculation, so the AP will be the return value processor if the return value is a pointer.
1012
Kevin D. Rich and Matthew K. Farrens
UNIPROCESSOR load rA, rB, 8 store rX, rY, 8
⇒
ACCESS PROCESSOR add LAQ, rB, 8 add SAQ, rX, 8
EXECUTE PROCESSOR move rA, LDQ move SDQ, rY
As indicated in the above example, the node which enqueues the load or store address is assigned to the AP and the node which enqueues or dequeues the data is marked for the EP; these nodes serve as anchors. For the load operation, the node for the instruction which dequeues from the LDQ is a source node and the node which enqueues the load address on the LAQ is a sink. For the store operation, both the node for the instruction which enqueues the address on the SDQ and the node which enqueues the data on the SDQ are sinks. It is important to note that while the processors may be identical, the EP cannot be allowed to enqueue addresses for data which the AP can access (e.g., global data or data on the AP’s stack), because there is the potential for the violation of RAW, WAR, or WAW dependencies. So functionally speaking such reverse decoupled loads and stores are not permitted. In the shared stack model these restrictions result in the EP not be being permitted to use its alternate memory queues. Step 3: Propagate Load/Store Processor Markings Propagation of markings entails starting at each of the anchors operations and traversing the (sub)graph rooted at that anchor, assigning all unmarked data-dependent nodes in the graph to the same processor as the anchor. The propagation occurs in two directions, down from a sink and up from a source. When propagating the markings down from a sink, the marking propagates down through its children to all of its descendants. When propagating the marking up from a source, the marking propagates up through its parents to all of its ancestors. If during the traversal a node is encountered that is marked for the other processor, a copy is necessary in order to communicate the value from one processor to the other. In the below example of an AP-to-EP copy, AP line 2 enqueues the value on the copy queue, and EP line 1 dequeues the value from the copy queue.
1: 2:
ACCESS PROCESSOR add rA, rB, 8 move CPQ, rA
EXECUTE PROCESSOR sub rY, CPQ, rX
The compiler attempts to avoid EP-to-AP copy operations as they introduce slip-reducing dependencies. The fact that a copy is required means that the value is needed by both processors and thus the computing expression is a common sub-expression (CSE). If the CSE is inexpensive to compute, common sub-expression replication may be employed (in which both processors perform the calculation of the sub-expression). If the CSE is too expensive to replicate, but cheap enough that moving it to the AP would not lead to grossly unbalanced code, it is moved to the AP (it is stolen) and an AP-to-EP copy is inserted. Allowing the AP to steal the sub-expression eliminates the need for the slip-reducing
Code Partitioning in Decoupled Compilers
1013
EP-to-AP copy. If this common sub-expression stealing would result in code balancing problems, or is otherwise infeasible, an EP-to-AP copy is inserted. The compiler makes these decisions by estimating the computation cost of the subexpression and comparing it to threshold values3 . Copies related to load and store operations may also be eliminated by converting standard decoupled loads or stores into self-loads or self-stores on the access processor. If an EP-to-AP copy is unavoidable, the compiler performs code scheduling in order to minimize the impact of the copy. The EP’s enqueuing operation is placed as early as possible in its instruction stream, and the AP’s dequeuing operation is placed as late as possible in its instruction stream. This technique is used extensively with function parameters. Step 4: Branch Splitting After marking propagation, conditional branch operations are split into their two cooperating counterparts, the external branch operation and the branch from queue operation. An external branch is a conventional branch which also writes the branch decision (taken/not taken) to the processor’s outgoing branch queue, while a branch from queue (bfq) is a conditional branch that is taken or not based on the value at the head of the processor’s incoming branch queue. This mechanism allows the processors to traverse the flow graph in identical fashion by easily synchronizing control flow decisions. In the below example the x in the brxnz indicates an external branch. UNIPROCESSOR cmpne rC, rA, rB brnz rC, @L1
⇒
ACCESS PROCESSOR cmpne rC, rA, rB brxnz rC, @L1
EXECUTE PROCESSOR bfq
@L1
This splitting is only performed on conditional branches. Unconditional flow control operations (e.g., jumps, function calls) are simply duplicated on each of the processors and do not need to be split. The compiler must determine which processor is to receive each of the two branch operation components. There are several issues to be considered when making this decision — for example, if the bfq is put on the AP then a slip reducing AP-on-EP dependency is introduced. In order to avoid such dependencies, the compiler makes every attempt to put the comparison and the external branch on the AP and the bfq on the EP. Forcing the comparison to be on the AP may, however, significantly impact code balance between the AP and EP. For example, functions which approximate solutions to equations often iterate until two successive approximations are within a pre-specified limit. In this case the EP is the natural choice to perform the value computations, and moving these computations to the AP would leave the EP with little or nothing to do, negatively impacting performance. Therefore, the compiler takes into account both existing processor markings and the cost of the sub-expression which performs the comparison when determining the branch assignments. 3
If an AP to EP copy is required then the only consideration is code expansion due to the copy operations. In this case replication of the CSE is employed only if the CSE consists of a small number of low-latency instructions.
1014
Kevin D. Rich and Matthew K. Farrens
Step 5: Propagate All Markings At this point in the process the entire graph has been traversed and node markings have been propagated to the ancestors and descendants of the anchor nodes. The final step is to traverse the graph once more with node markings being propagated from all nodes. As described previously, copies are inserted where necessary and common sub-expressions may be replicated. The propagation continues as long as it results in changes to the graph (by copy insertion, processor markings, or common sub-expression replication). Once this process terminates there are no unmarked nodes and the two graphs have been extracted.
6
Example Compiler Output
This section presents the compiler output for a simple source file which computes the sum of a sequence of integers (shown in left-most column of Figure 2). This program was chosen because it includes examples of decoupled loads and stores, self loads and stores, copies, external branches, and branch from queue operations. Figure 2 also shows the decoupled access/execute code generated by the compiler. The labels of the form @FSxx are stack frame sizes and are simply used to make adjustments to the stack pointer (r30). r1 is the return value register. A single parameter register was used in order to illustrate parameter passing both in a register and on the stack. Relevant assembly language explanations are given in Table 1. Opcode
Description
mvfq dest, queue brnz src1, label brxnz src1, label
Move from queue: dest = value at head of queue Branch not zero: if (src1){P C = label} Branch external not zero: if (src1){P C = label}, send results of branch to branch queue Branch from queue: if (value on branch queue){P C = label}
bfq
label
Table 1. Selected Assembly Language Opcodes
Looking at Figure 2 one can see an example of a standard decoupled load on line 14 on the AP (AP.14) and line 21 on the EP (EP.21). AP.2 and EP.3 are the two halves of a standard decoupled store. An example of a slip-reducing copy is on lines EP.1 and AP.11; this particular copy is used to send the parameter passed in register 2 on the EP to the AP. Note that the slip reducing effect is mitigated by placing the producer operation before the SDQ accesses (EP.3 EP.5) which save temporary registers for the EP, and the consumer operation after the corresponding SAQ, ASAQ, and ASDQ accesses (AP.2-AP.10) which assist the EP in saving its temporaries, and save the AP temporaries. EP.21 and EP.22 set up the first parameter to sum() in r2. AP.32 and EP.20 store the second parameter to sum() on the stack. AP.18 is an external branch and EP.10 is the corresponding branch from queue operation.
Code Partitioning in Decoupled Compilers 1 2 int sol; 3 4 int sum(int a, int b) 5 { 6 int i, sum = 0; 7 for (i = a; i <= b; i++) 8 sum += i; 9 return sum; 10 } 11 12 void main(void) 13 { 14 int a = 1, b = 100; 15 sol = sum(a,b); 16 return; 17 } 18 19 20 21 22 23 24 25 26 27 28 29 30
sum: addc addc addc addc addc mvut addc mvut addc mvut mvut addc mvfq mvut br @L3: addc @L5: cmpw brxnz br @L4: addc addc addc addc addc addc mvfq mvfq mvfq subc ret
31 main: addc 32 addc 33 call 34 addc 35 subc 36 halt
Source Code
r30, r30, @FSaf SAQ, r30, -56 SAQ, r30, -52 SAQ, r30, -48 ASAQ, r30, -44 ASDQ, r5 ASAQ, r30, -40 ASDQ, r4 ASAQ, r30, -36 ASDQ, r3 r2, CPQ ALAQ, r30, b-@FSaf r3, ALDQ r4, r2 @L5 r4, r4, 1 r23, r3, r4, LT r23, @L4 @L3 LAQ, r30, -56 LAQ, r30, -52 LAQ, r30, -48 ALAQ, r30, -44 ALAQ, r30, -40 ALAQ, r30, -36 r5, ALDQ r4, ALDQ r3, ALDQ r30, r30, @FSaf r30, SAQ, sum SAQ, r30,
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
sum: mvut addc mvut mvut mvut addi mvut br @L3: addc @L5: bfq addw br @L4: mvut mvfq mvfq mvfq subc ret
1015
CPQ, r2 r30, r30, @FSaf SDQ, r5 SDQ, r4 SDQ, r3 r5, 0, 0 r4, r2 @L5 r4, r4, 1 @L4 r5, r5, r4 @L3 r1, r5 r5, LDQ r4, LDQ r3, LDQ r30, r30, @FSaf
19 main: addc r30, r30, @FScm 20 addi SDQ, 0, 100 21 addi r3, 0, 1 22 mvut r2, r3 23 call sum 24 mvut SDQ, r1 25 subc r30, r30, @FScm 26 halt
r30, @FScm r30, 0-16 0, sol r30, @FScm
Access Processor
Execute Processor
Fig. 2. Source and Assembly Code for Compilation Example
7
Results
The book Numerical Recipes in C contains a wide variety of algorithms, eight of which were selected as a representative sample [5]. These benchmarks have between 82 and 163 lines of source code which result in actual instructions in the executable (i.e., no comments or variable declarations are counted). The decoupled simulator (Decsim) is written in C and accepts a simple object code format containing tuples of
1016
Kevin D. Rich and Matthew K. Farrens
Speed-Up vs. Uniprocessor
lying on standard load and store operations to communicate with memory. In the uniprocessor mode a data cache is employed; no data cache is used in the decoupled mode. Figure 3 shows the speed-up achieved by running the benchmarks on the twoprocessor decoupled architecture vs. a uniprocessor architecture with an 8K-byte cache4 at memory latency of 20 cycles. With the 20 cycle memory the average speed-up is only 1.06, with four of the benchmarks actually running slower on decoupled processor. In all four of poor performing benchmarks the AP spends over 50% of its cycles stalled on either an empty branch or copy queue. As a basis for comparison the original fourteen Livermore Loops were compiled and simulated showing considerable speed-up (2.05 on average) corroborating the results from the previous studies of decoupling using these benchmarks [11, 2]. Intuitively, a speed-up of less than 1.0 seems unlikely since the decoupled processor enjoys a 2:1 advantage in raw processor resources. However, poor decoupled performance can occur if the decoupled processor is unable to achieve slip and the AP experiences the full memory latency on each memory access, while good uniprocessor performance can result if the cache hit rate is high. If these two events occur for the same benchmark the decoupled processor would run significantly slower than the uniprocessor. Investigation into the run-time behavior indicates that the poor performance is attributable to control dependencies and copy operations significantly limiting the slip — research is ongoing on techniques to address these issues. Daecomp was used to perform many other experiments, the results of which are available in [6].
1.5 1.0
20 cycles
0.5 0.0
xei
xfour1
xsort2
xtutest
xgasdev
xbcuint
xpccheb
xtred2
Fig. 3. Speed-Up Comparison with 20 Cycle Memory Latency — NRC
4
If a typical cache size were used with the benchmarks in question, the cache would like suffer only compulsory/first reference misses, and conflict and capacity misses would likely not be an issue. Therefore, a small cache was used in order to keep it on scale with the working sets of the benchmarks. The cache selected resulted in overall hit rate of 97%.
Code Partitioning in Decoupled Compilers
8
1017
Conclusion
To fully evaluate any architecture a compiler is needed, for this reason the decoupled access/execute compiler Daecomp was constructed. The primary role of a a decoupled compiler is to partition the instructions into two cooperating instruction streams. Daecomp implements a five-step partitioning process to identify the access and execute instruction streams. The results of some of these studies performed with Daecomp confirmed published result obtained using small, hand-compiled benchmarks [11, 1, 4, 2]. However, using larger, more varied benchmarks revealed that many of these earlier conclusions were erroneous, underscoring the importance of constructing a compiler. Further work is planned to determine if the architectural model can be modified to permit speculative execution, or to employ data caches.
References [1] Ali Berrached, Lee D. Coraor, and Paul T. Hulina. A decoupled access/execute architecture for efficient access of structured data. In Proceedings of the 26th Annual Hawaii International Conference on System Sciences, pages 438–447, 1993. [2] Jian-tu Hsieh, Andrew R. Pleszkun, and James R. Goodman. Performance evaluation of the PIPE computer architecture. Technical Report 566, University of Wisconsin - Madison, November 1984. [3] G.P. Jones and N.P. Topham. A comparison of data prefetching on an access decoupled and superscalar machine. In Proceedings of the 30th Annual International Symposium on Microarchitecture, 1997. [4] William Mangione-Smith, Santosh G. Abraham, and Edward S. Davidson. The effect of memory latency and fine-grain parallelism on Astronautics ZS-1 performance. In Proceedings of the 23rd Annual Hawaii International Conference on System Sciences, pages 288–296, 1990. [5] William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brain P. Flannery. Numerical Recipes in C. Cambridge University Press, 2 edition, 1996. [6] Kevin D. Rich. Compiler Techniques for Evaluating and Extending Decoupled Architectures. PhD thesis, University of California at Davis, 2000. [7] James E. Smith. Dynamic instruction scheduling and the Astronautics ZS-1. IEEE Computer, 22(7):21–35, July 1989. [8] Nigel Topham, Alasdair Rawsthorne, Callum McLean, Muriel Mewissen, and Peter Bird. Compiling and optimizing for decoupled architectures. In Proceedings of the 1995 ACM/IEEE Supercomputing Conference, 1995. [9] N.P. Topham and K. McDougall. Performance of the decoupled ACRI-1 architecture: the Perfect Club. In Proceedings of High Performance Computing - Europe, 1995. [10] Gary S. Tyson. Evaluation of a Scalable Decoupled Microprocessor Design. PhD thesis, University of California at Davis, 1997. [11] Honesty Cheng Young. Evaluation of a decoupled computer architecture and the design of a vector extension. Technical Report 603, University of WisconsinMadison, 1985. [12] Yinong Zhang and George B. Adams III. Performance modeling and code partitioning for the DS architecture. In Proceedings of the 25th Annual International Symposium on Computer Architecture, 1998.
Limits and Graph Structure of Available Instruction-Level Parallelism Darko Stefanovi´c and Margaret Martonosi Princeton University, Princeton NJ 08544, USA
Abstract. We reexamine the limits of parallelism available in programs, using run-time reconstruction of program data-flow graphs. While limits of parallelism have been examined in the context of superscalar and VLIW machines, we also wish to study the causes of observed parallelism by examining the structure of the reconstructed data-flow graph. One aspect of structure analysis that we focus on is the isolation of instructions involved only in address calculations. We examine how address calculations present in RISC instruction streams generated by optimizing compilers affect the shape of the data-flow graph and often significantly reduce available parallelism.
1 Background and Related Work Most studies of the limits of available instruction-level parallelism have focused on the timing of an optimal schedule of the instruction sequence for an idealized processor model. We propose to examine directly the data flow graph of the instruction sequence. Thus we will be able to gain insight into the structural properties of the available parallelism, so that we may understand which elements of the instruction sequence, or which compiler idioms, affect available parallelism. In particular, here we show that the presence of address calculations for memory operations greatly affects parallelism; in some programs, it is precisely the address calculations that limit the asymptotically achievable parallelism. As in earlier studies, we assume no hardware limitations: the degree of parallelism available is the degree exploitable. Examining very long dynamic code sequences means that control flow is entirely revealed and does not constrain parallelism. Imperfect alias analysis in compilers [1, 2] sequentializes code by enforcing the order of the store-load pair, together with all potentially aliased memory operations, for any value that cannot be held in registers and is temporarily stored in memory (register spill, call save, or otherwise); by precise memory disambiguation at run-time we remove all such constraints as well. A number of studies over the past three decades have looked at the limits of parallelism [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], using instruction-scheduling simulators. The simulator reports the number of cycles needed to execute the program, and the number of instructions executed. The ratio of the two gives the IPC as the standard measure of instruction-level parallelism [11]. The simulator effectively constructs the moving “front line” of the data-flow graph [3]; thus, constructing an entire data-flow graph is not necessary to obtain a single number, the cycle count. However, having an explicitly constructed graph permits us to study its structure: we can inspect the computation nodes A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1018–1022, 2000. c Springer-Verlag Berlin Heidelberg 2000
Limits and Graph Structure of Available Instruction-Level Parallelism
1019
repeatedly, and evaluate the graph using multi-pass and backward-flow algorithms. We will illustrate this new possibility on one example: we will recognize instructions involved in address calculations using a backward-flow algorithm. While in the past reconstructing large graphs was dismissed as impractical [3], that is no longer the case. Currently available memory space permits building graphs sufficiently large to capture interesting application behavior—parallelism analysis using a conceptual dependence graph of a moving window of program execution was demonstrated by Austin and Sohi [1]. Recently, Ebcio˘glu et al. described a system for dynamic code translation and optimization [12], aimed at transparent porting of applications to a VLIW execution engine. Among other results, they evaluate achieved parallelism without resource constraints, and with store-load bypassing. We obtain comparable parallelism numbers, except for their results with the “combining” optimization, which in some cases show much higher parallelism. This optimization breaks dependence chains of immediate-operand instructions with the dependence on a common register, by adjusting the immediate values (a form of constant folding at the machine level); code modifications are outside the scope of our study.
2 Run-Time Analysis of Programs Our analysis uses the core of the SimpleScalar architectural simulation toolset [13] for the Alpha instruction set, and dynamically constructs a program’s data-flow graph. Conceptually, graph nodes correspond to executed instructions, while graph edges correspond to computed operand values. The values are tracked through memory, including multi-byte values through partial and unaligned accesses. This allows us to recognize when entire stored values are reloaded. Nodes are not created for instructions identifiable as data transport: register moves and memory loads; instead, the values are appropriately bypassed from the producing node to the using node. Thus the data flow of the computation is reconstructed independent of the storage layout. We simulated a number of SPEC95 and Mediabench programs, with up to 1800 million instructions executed. Benchmarks were compiled on a Digital Alpha 21164 EV56 using native C and Fortran compilers, and highly optimized as specified by SPEC. For each benchmark, we varied the size of the instruction window as powers of 2, between 16 and 1M (limited by the memory capacity of the simulator host). We first look at the parallelism reported for the graphs consisting of all instructions in the examined window; the results are presented in plots (a) and (b) in Figures 1 and 2. The solid lines, labelled all in the graph height plots (a), show the growth of average graph height (length of critical path) with increasing instruction window size. The axes in graphs (a) are both logarithmic; the slopes of the curves (below 1) show that the dependence is sublinear. The solid lines, labelled all in the graph parallelism plots (b), show the ratio of graph size (number of instruction nodes) to height. This ratio is a measure of average available parallelism, because it reflects the potential speedup of a machine with unbounded hardware resources (limited only by data dependences) over a sequential machine that executes exactly one instruction per cycle in program order. As the instruction window size increases, so does the parallelism. However, we note some distinct behaviors. In 145.fpppp, the parallelism saturates quickly: with an instruction
1020
Darko Stefanovi´c and Margaret Martonosi
window size of 128K, it is 314, with 1M, it is 357. Not so in 110.applu: parallelism grows smoothly (but sublinearly) even as very large window sizes are reached. The absolute values of parallelism are vastly different: whereas 145.fpppp achieves over 300, and 110.applu over 1000, we have only 45 for 129.compress (not shown). This agrees with observations [3] that some numerical programs have very high intrinsic parallelism, proportional to problem size and exposed by unrolling loops (which we in effect do).
10000
1000
all excluding address
all excluding address
Graph height
Graph height
1000
100
100
10
10
1
10
100
1
1000 10000 100000 1e+06 1e+07
1
10
Instructions in window
(a) Graph height 1800
1200 1000 800 600 400
10
100
400
0
0.8 0.6 0.4 0.2 graph size ratio graph height ratio 10
100
1000 10000 100000 1e+06 1e+07 Instructions in window
1
10
100 1000 100001000001e+06 1e+07 Instructions in window
(b) Graph parallelism measure Ratios (excluding address)/(all)
(b) Graph parallelism measure Ratios (excluding address)/(all)
600
1000 10000 100000 1e+06 1e+07 Instructions in window
1
0
800
200
200 0
all excluding address
1000 Graph size/height
Graph size/height
1400
1000 100001000001e+06 1e+07
(a) Graph height 1200
all excluding address
1600
100
Instructions in window
1 0.8 0.6 0.4 0.2 0
graph size ratio graph height ratio 1
10
100 1000 100001000001e+06 1e+07 Instructions in window
(c) Ratios excluding address calculation
(c) Ratios excluding address calculation
Fig. 1. Benchmark 145.fpppp
Fig. 2. Benchmark 110.applu
Excluding Address Calculations. Will there be differences with respect to available parallelism between the data-flow graph as built, and its subgraph that excludes purely address calculations? This is an interesting question, because the latter graph
Limits and Graph Structure of Available Instruction-Level Parallelism
1021
seems closer to the algorithmic intent of the program, address calculations being partly an artifact of the particular compiler/RISC architecture realization of the program. Recall that while we are reconstructing the data-flow graph at run-time, we are able to recognize when a load instruction L retrieves a value written to memory by a previous store instruction S and produced by a previous computational instruction C. We bypass such a load—an instruction that uses the loaded value sees it instead as coming from C, similar to the load-store telescoping optimization [12]. Note that L is no longer needed to represent the computation, and in some cases S also is no longer needed (if L is the only load of the value). Loads and stores are preceded by instructions to calculate an address. (These instructions may in turn include other loads.) If certain loads and stores are no longer needed to represent the computation, then the corresponding address calculations are not needed either. However, while we are building the graph we cannot know which computations will end up being used only to calculate addresses. This we determine in a separate, backward-propagating pass over the data-flow graph. (Address calculation recognition subsumes the stack pointer register analysis of [10].) The dashed lines, labelled excluding address in plots (a) and (b), give the graph height and graph parallelism measure for the data-flow subgraph without address calculations. Plots (c) show the relative size and height of the subgraph with respect to the full graph. We show both in the same plot area to make it easier to compare with the graph parallelism measure plot. (Consider the intersections of (c) curves and the intersections of (b) curves: their abscissæ coincide.) Let us first look at the relative subgraph size, labelled “graph size ratio” in plots (c). This ratio changes very little with instruction window size, and the small observed change is in the direction of somewhat smaller ratios as the window size is increased. Indeed, in the backward-propagating algorithm we must conservatively assume that values present at the end of the instruction window may be used as non-addresses in the continuation of the program after the window; as the window grows, the inaccuracy of that assumption diminishes and with it the number of instructions inaccurately assumed to be involved in non-address computation. The ratio varies greatly across benchmarks: 0.9 for 145.fpppp, 0.8 for 110.applu and 124.m88ksim, but just 0.2 for 129.compress. Relative subgraph height, labelled “graph height ratio” in plots (c), shows significant variation with window size. In 145.fpppp it remains close to 1 up to a window size of 16K, but drops sharply thereafter, so that by 1M it is just 0.2. In other words, for smaller windows, the subgraph height is about the same as the full graph height, but for larger windows, the subgraph height collapses. The critical path is determined by a dependence chain of address calculations carried in a loop. If address calculations are eliminated, a much larger amount of parallelism is exposed. We observed the same pattern in 141.apsi, 099.go, 134.perl, 126.gcc, 130.li, and mpeg2decode. On the other hand, in 110.applu the ratio of graph heights is close to 1: the critical path is for the most part determined by the “data” calculations, i.e., instructions other than address calculations. We observed a similar pattern in 146.wave5, 124.m88ksim, and adpcm. We may summarize the findings as follows: When address calculations form long dependence chains, they can dominate “data” computations, and their removal is beneficial for parallelism. When address calculations are localized, their removal does not affect graph height, yet it reduces graph size; therefore, parallelism is reduced.
1022
Darko Stefanovi´c and Margaret Martonosi
3 Future Directions With data-flow graphs explicitly constructed we are not restricted to critical paths through the entire graph, but can zoom in on particular nodes. For instance, we can examine the critical path of the computation that produces the address for a load (with a view to prefetching), or the critical path that produces a conditional value (with a view to scheduling beyond the corresponding branch). We should consider what can be done in language implementation to reform the way memory data are accessed: a compiler optimization such as array index “strength reduction” can introduce a chain of address calculations where none is apparent at the source level. On the other hand, to appreciate the practical repercussions of available parallelism, we should consider code mappings to realistic processors, where memory bandwidth and control flow uncertainty are taken into account. We intend to combine the analysis of instruction-level parallelism with analysis of bit usage [14], which will lead to a finer-granularity description of parallelism as the basis for code mapping decisions for hybrid fixed-configurable processors.
References [1] T. M. Austin and G. S. Sohi. Dynamic dependency analysis of ordinary programs. In 19th ISCA, pages 342–351, May 1992. [2] J. W. Davidson and S. Jinturkar. Improving instruction-level parallelism by loop unrolling and dynamic memory disambiguation. In MICRO-28, Dec. 1995. [3] A. Nicolau and J. A. Fisher. Measuring the parallelism available for very long instruction word architectures. IEEE Trans. Comput., C-33(11):968–976, Nov. 1984. [4] D. W. Wall. Limits of instruction-level parallelism. WRL Research Report 93/6, Digital Equipment Corporation, Western Research Laboratory, Palo Alto, CA, Nov. 1993. [5] C. C. Foster and E. M. Riseman. Percolation of code to enhance parallel dispatching and execution. IEEE Trans. Comput., C-21(12):1411–1415, Dec. 1972. [6] N. P. Jouppi. The nonuniform distribution of instruction-level and machine parallelism and its effect on performance. IEEE Trans. Comput., 38(12):1645–1658, Dec. 1989. [7] M. D. Smith, M. Johnson, and M. A. Horowitz. Limits on multiple instruction issue. In ASPLOS III, pages 290–302, Boston, Massachusetts, 1989. [8] M. Butler, T.-Y. Yeh, Y. Patt, M. Alsup, H. Scales, and M. Shebanow. Single instruction stream parallelism is greater than two. In 18th ISCA, pages 276–286, May 1991. [9] M. S. Lam and R. P. Wilson. Limits of control flow on parallelism. In 19th ISCA, pages 46–57, May 1992. [10] M. A. Postiff, D. A. Greene, G. S. Tyson, and T. N. Mudge. The limits of instructions level parallelism in SPEC95 applications. In 3rd Workshop on Interaction Between Compilers and Computer Architecture, Oct. 1998. [11] J. L. Hennessy and D. A. Patterson. Computer Architecture: A Quantitative Approach. Morgan Kaufman Publishers, Inc., San Mateo, California, 1996. Second Edition. [12] K. Ebcio˘glu, E. R. Altman, S. Sathaye, and M. Gschwind. Optimizations and oracle parallelism with dynamic translation. In MICRO-32, Nov. 1999. [13] D. Burger and T. M. Austin. The SimpleScalar tool set, version 2.0. Computer Architecture News, pages 13–25, June 1997. [14] D. Stefanovi´c and M. Martonosi. On availability of bit-narrow operations in generalpurpose applications. In 10th FPL, Villach, Austria, 2000.
Pseudo-vectorizing Compiler for the SR8000 Hiroyasu Nishiyama, Keiko Motokawa, Ichiro Kyushima, and Sumio Kikuchi Systems Development Laboratory, HITACHI, Co.Ltd. {nisiyama,motokawa,kyushima,kikuchi}@sdl.hitachi.co.jp
Abstract. The pseudo-vector processing (PVP) is a framework that enables fast processing similar to vector processing. In this paper, we describe the compiler optimizations that effectively utilize PVP on the SR8000. These include access method analysis, preloading, and prefetching optimizations. Evaluations on the SR8000 indicate that PVP can effectively hides memory latency.
1
Introduction
The SR8000[1, 6] super technical server consists of multiple SMP (Symmetric Multi Processor) nodes. The SMP nodes are connected by high-speed interconnects to provide fast inter-node communications. Each node contains nine RISC microprocessors and main memory. Eight of the microprocessors are used for computation (IP), and the remaining one is used for systems operation (SP). The microprocessor used for IP and SP is based on the PowerPC1 architecture. It incorporates a pseudo-vector processing (PVP) mechanism. PVP is a framework that enables fast computation similar to vector processing. It is implemented using the following mechanisms: (1) large floating-point registers and cache memory, (2) a continuous data supply from main memory to the floatingpoint registers and cache memory using preloading and prefetching, and (3) instruction level parallel execution using pipelining and out-of-order execution. The following is a list of the special features of the SR8000 related to PVP. Data prefetch instruction The SR8000 has a 128k byte 4-way set associative data cache with lines of 128 bytes. Each IP can handle up to 16 data prefetch[2] requests simultaneously. Data preload instruction The data preload instruction loads floating-point data from main memory to a floating-point register directly bypassing cache memory. It can transfer up to 128 floating-point items simultaneously. It does not require useless data transfer even for non-continuous data references. Extended registers In addition to the 32 floating-point registers defined in the PowerPC specification, 128 extended registers are defined. The section from FR0 to FR31 is called the global part, and the section from FR32 to FR128 is called the slide part. Floating-point operations are also extended to allow use of the slide part. The slide part can be renamed by software control using 1
PowerPC is a trademark of IBM Corp.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1023–1027, 2000. c Springer-Verlag Berlin Heidelberg 2000
1024
Hiroyasu Nishiyama et al.
slide instructions[5, 3]. After executing a slide instruction, the contents of register FRn become accessible by FRm , where m = (n−32+P ) mod 128+32. P means the distance of register number before and after execution of the slide instruction, and is called slide pitch.
2
Pseudo-vector Optimization
2.1
Access Method Analysis
We call a method to tolerate memory latency an access method. The SR8000 has three access methods: PREFETCH, PRELOAD, and LOAD2 . The access method for a reference is selected using access method analysis. To select an access method, references in a given loop that have a constant difference of address expression are grouped. An access method is then calculated for each reference group. The access method for a reference is determined by considering the cache line reuse ratio, δ, in a reference group. If δ is larger than or equals to the threshold value, the references of the group can be considerd continuous. If this is the case, PREFETCH is used as the access method of the group; otherwise PRELOAD is used. Since the SR8000 does not support preloading of integer data, PREFETCH is used for integer data even for small δ. For a group that uses PREFETCH, redundant prefetch instructions can be eliminated using spatial locality. To effectively use spatial reuse, we set the access method for the front reference in the group to PREFETCH as a representative, and set the remaining references to LOAD. The reference length of the group is calculated to insert prefetch instructions for the group, which is described later. Although the access method for a reference is selected by analyzing its access pattern, static analysis is not always correct. Thus, we also provide user directives and compiler options to explicitly specify access methods. 2.2
Preloading Optimization
Preloading optimization hides memory latency using software pipelining. Kernel creation and slide register allocation are performed assuming memory latency (L) and initiation interval (II). The kernel creation is based on Iterative Modulo Scheduling[4], which iterates kernel creation increasing values of II from MinII, defined by resource and recurrence constraints. After obtaining a kernel schedule, register allocation for the slide part is performed. Slide register allocation uses an extension of graph coloring algorithm[3]. (1) The slide register allocator obtains live ranges of each floating-point variables. Live ranges that cover whole loop iterations are assigned to the global part. (2) Each candidate live range on the slide part is divided into segments that belong different stage of kernel. We call a group of divided live ranges a slide group. (3) An interference graph is created for the divided live ranges. We use graph coloring to assign registers, considering the following restriction: the register number 2
The case when cache hit is expected.
Pseudo-vectorizing Compiler for the SR8000
1025
of node Gi is defined by (n − 32 + P ×i) mod 128 + 32, where the i-th node in a group is Gi , slide pitch is P , and the assigned register number of G0 is n. The difference between the preloading optimization from previous studies[4, 3] is that the our compiler retries scheduling after decreasing the estimated value for memory latency, L, when the slide register allocation fails. When the value of L becomes lower than threshold value, latency can not be hidden effectively by preloading. Thus, the access method of the front reference in the group is changed to PREFETCH and the access methods of other references are changed to LOAD making the references candidates for the prefetching. 2.3
Prefetching Optimization
Prefetching optimization hides memory latency by inserting prefetch instructions for references with PREFETCH access method. For each reference group, the number of iterations, N , required to hide latency is defined as Memory Latency/Estimated Cycle of Loop. Thus the front address of prefetching for a group G is defined as F (G)+N ×S(G), where F (G) is the front reference address of G, and S(G) is the distance in the addresses referenced by G between iterations i-th and i+1-th. To prefetch every item in the group, Len[G]/128 prefetch instructions are issued from the front address in every cache line, where Len[G] is reference length of G. Since the front reference may not be on the cache line border, a prefetch instruction should be issued for the last reference when the reference length is not a multiple of cache line size. In addition, when difference between the next to the last prefetching address, F (Gi ) + Len[Gi ]/128 ×128, for the i-th iteration of G and the first reference address, F (Gi+1 ), for the i + 1th iteration of G is smaller than the cache line size, we can assume that the reference of G is consecutive between iterations. Thus the prefetching of the last reference in the group is not required in this case. Since the cache line of the SR8000 is relatively large (128 bytes), prefetching for small stride reference is a waste of instruction bandwidth. The SR8000 compiler eliminates redundant instructions by unrolling the loop without eliminating exit branches to reduce the overhead of short loops.
3
Evaluation
We describe the results of evaluations on the SR8000. All evaluations were performed on a single node of the SR8000 without automatic SMP parallelization using a hardware performance counter so as to precisely evaluate the effect of PVP. In the following discussion, we denote the case that uses neither prefetching nor preloading by LD, the case that uses only preloading by PLD, the case that uses only prefetching by PF, and the case that selectively uses prefetching and preloading via automatic analysis by PLD+PF. Basic loop performance Figure 1(a) shows the performance of a DAXPY (Y[i]=A*X[i]+Y[i]) loop for a range of stride values. The performance of LD was low even for sequential access. The performance of PLD was stable and
8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0 1
LD PLD
5
10 stride (a) stride loop
PF PLD+PF
15
19
relative performance
Hiroyasu Nishiyama et al.
relative performance
1026
3.0
LD PLD
PF PLD+PF
cons
rand
2.0 1.0 0.0
(b) indirect access loop
relative performance
Fig. 1. Basic loop performance
4.0 3.0 2.0
LD PLD PF PLD+PF
1.0 0.0
tomcatv swim su2cor hydro2d mgrid applu turb3d apsi fpppp wave5 BT
CG
FT
SP Avg.
Fig. 2. Performance of SPEC&NPB
4∼5 times better than sequential LD case. PF achieved still higher performance for sequential access. However the performance dropped rapidly to the level of LD as stride value increased. Automatic analysis, assuming that the threshold value of δ is 0.5, showed the most stable performance. This method selects an appropriate method for each stride value, except for a stride value of 2. Figure 1(b) shows the results of an indirect access loop. The indirect access loop (Z[L[i]]=A*X[L[i]]+ Y[L[i]]) was tested for having consequtive (cons) and random (rand) elements. The performance is shown relative to that of LD with consequtive elements. For the case where the values of L were consecutive, PF obtained slightly better performance than LD because of cache reuse. However, the performance gain is small since there is instruction overhead for each loop iteration for prefetching. PLD showed much larger performance improvements by hiding the memory latency with low instruction overhead even without hiding the memory latency of L. Hiding memory latency with a combination of preloading and prefetching showed better improvements than other methods because memory latency is effectively hidden. The performance of PF and LD drops remarkably for the random access case. However PLD and PLD+PF also hold better performance than LD. This shows good toleration of memory latency by pseudo-vector processing using access method analysis. SPEC&NPB benchmarks Figure 2 shows the results of performance evaluation on SPEC95fp benchmarks and four benchmarks (BT,CG,FT,SP) from
Pseudo-vectorizing Compiler for the SR8000
1027
NAS Parallel Benchmarks. The performances of PLD, PF, and PLD+PF relative to the performance of LD are shown in this figure. We used ‘ref’ data for SPEC and class B for NAS. The maximum performance gain of PLD is 237% on CG(avg. 34%), the maximum performance gain of PF is 114% on swim(avg. 45%), and the maximum performance gain of PLD+PF is 323% on CG(avg. 71%). Automatic selection shows equivalent or better performance than use of only a single latency hiding method for all benchmarks except turb3d. For a preloading loop, when the loop iteration count is small, it is difficult to issue preload instructions in advance using software pipelining. Further, when the dependencies between array references are uncertain at compile time, preload instructions are also difficult to issue in advance. These are the reasons for the low performance of PLD on benchmarks such as mgrid. On the other hand, the preloading, for benchmarks such as CG, outperforms prefetching. The reason is better tolerance of non-continuous data references and prefetching’s performance degradation from cache slashing. Since data preloading bypasses cache access, a performance drop from cache slashing does not occur.
4
Conclusion
The SR8000 can hide memory latency effectively with software managed controlling of data movement using pseudo-vector processing. In this paper, we have described optimizations of the pseudo-vectorizing compiler for the SR8000. According to the results of this evaluation, the SR8000 shows good tolerance of memory latency with pseudo-vectorization.
References [1] T. Kurihara, K. Shimada, E. Kamada, and T. Shimizu. A RISC Processor for SR8000: Accelerating Large Scale Scientific Computing with SMP. In Hot Chips 11, 1999. [2] T.C. Mowry, M.S. Lam, and A. Gupta. Design and Evaluation of a Compiler Algorithm for Prefetching. In Proceedings of the 5th Conference on Architectural Support for Programming Languages and Operating Systems, pages 62–75, 1992. [3] K. Nakazawa, H. Nakamura, H. Imori, and S. Kawabe. Pseudo Vector Processor Based on Register-Windowed Superscalar Pipeline. In Proceedings of Supercomputing’92, pages 642–651, 1992. [4] B.R. Rau. Iterative Modulo Scheduling: An Algorithm For Software Pipelining Loops. In Proceedings of the 27th Symposium on Microarchitecture, pages 63–74, 1994. [5] B.R. Rau, D.W.L. Yen, W. Yen, and R.A. Towle. The Cydra 5 Departmental Supercomputer: Design philosophies, decisions and trade-offs. Computer, 22(1):12– 35, 1989. [6] Y. Tamaki, N. Sukegawa, M. Ito, Y. Tanaka, M. Fukagawa, T. Sumimoto, and N. Ioki. Node Architecture and Performance Evaluation of the Hitachi Super Technical Server SR8000. In Int’l Conference on Parallel and Distributed Computing and Systems, pages pp.487–493, 1999.
Topic 15 Object Oriented Architectures, Tools, and Applications Gul A. Agha Global Chair
For some time, object-oriented programming has become standard practice in sequential programming. Objects separate the interface from the representation and promote reuse of code. Although concurrency is a natural consequence of objects, the standard model of objects uses sequential procedure calls. Early research in actors unified concurrency with objects and provided a basis for use of objects in parallel and distributed systems. The Euro-Par conference, as its predecessor PARLE, has a long tradition of covering cutting edge research in concurrent object-oriented programming. Since the late 1980s, the field has matured, and increasingly, research is being conducted on architectures, tools and applications. Our call for papers for this workshop chose to emphasize this aspect. Part of this shift has been the widespread acceptance of Java which incorporates some support for concurrency and distribution. Five regular papers and one short paper were chosen for this workshop from twice as many submissions. The three papers in the first session are closely tied to Java. The first paper in the workshop by Ngo and Barton discusses how reflection may be employed across distributed platforms on a Java Virtual Machine written in Java. In Java, the term reflection refers to the ability of objects to describe themselves (also called reification). This paper proposes to provide reflection remotely so that code can be inspected and debugged during execution. One of the unfortunate features of Java is the fact that it does not provide a uniform address space. Objects are referenced with respect to their current location and their reflection code must be executed within the same address space as that in which the objects reside. Ngo and Barton show how to address this difficulty. The paper by Antoniu et al. focuses on compilation of Java bytecode into native code in a distributed memory environment. The idea is to provide transparent distribution by executing a single Java virtual machine over a shared memory abstraction. The idea that code does not have to be rewritten for a different architecture is always an attractive one. Experience will show the eventual applicability of such an approach. For example, location awareness rather than transparency may facilitate load balancing. However, the performance results reported in this paper show the approach is promising. In the final paper in the Java related session, Chiao, Wu and Yuan describe an alternative to Java’s concurrency constructs called EMonitor. One of the difficulties in using Java for concurrent programming is that a programmer is forced A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1029–1030, 2000. c Springer-Verlag Berlin Heidelberg 2000
1030
Gul A. Agha
to do low-level synchronization. Objects are data encapsulation boundaries but not concurrent execution boundaries; instead they serve as surrogates for thread synchronization. Java provides simple, efficient mechanisms for this, allowing methods in objects to be explicitly locked to protect against undesirable concurrent access. This solution suffers from a number of problems such as single condition queues, deadlocks of inter-monitor nested calls, etc. The paper discusses these problems and offers EMonitor as an alternate which provides more flexible multithreaded programming under high-contention without significant performance overhead in the low-contention case. The second session starts with a paper by Tran and G´erodolle on an objectoriented framework for building “large-scale real-time networked virtual environment applications.” Their system has been used for multi-player game prototypes. The paper illustrates an increasingly accepted trend in software – using middleware to effectively separate policy from mechanism. Incidentally, the system described has also been implementated in Java. The paper by Nolte, Sato and Ishikawa describes a template library for data parallel programming on distributed objects. The idea is to exploit the polymorphism in function templates mechanism of C++ and provide reusable topology classes (such as grips, lists, and trees). The topologies can be used for globally synchronized operations. While performance comparisons to collective operations of MPI are in a competitive range, it would be interesting to see how the performance compares with asynchronous versions of algorithms that make use of the overlap of computation and communication to improve execution efficiency. The final paper of the session by Grundman, Ritt and Rosenstiel concisely describes a message-passing library implemented using C++. It provides typesafety and easy transmission of objects. The library is an effective way of systematically extending the messaging facilities using object-oriented techniques.
Debugging by Remote Reflection Ton Ngo and John Barton IBM T. J. Watson Research Center P.O. Box 218 Yorktown Heights, NY 10598 [email protected], john [email protected]
Abstract. Reflection in an object-oriented system allows the structure of objects and classes to be queried at run-time, thus enabling “metaobject” programming such as program debugging. Remote Reflection allows objects in one address space to reflect upon objects in a different address space. Used with a debugger, remote reflection makes available the full power of object-oriented reflection even when the object examined is within a malfunctioning or terminated system. We implemented remote reflection as an extension to an interpreter to create a very effective debugger for Jalape˜ no, a Java Virtual Machine written in Java.
1
Introduction
Reflection in an object-oriented language supports programs that manipulate the fields (data values) of an object using symbolic names specified at run-time. For instance, an object obj may provide a method getClass() to describe its own type, the type (class) in turn may provide a method getFields() to describe its fields (data values), and the field object may provide a get() method for accessing the corresponding data value from an object. Reflection enjoys extensive support in several modern object-oriented languages such as Java. A standard Java object provides numerous reflective methods for querying its internal values. The package java.lang.reflect provides a complete set of utilities to manage Java objects reflectively. With the description of the object encoded in the reflection methods, a program can inspect and manage an arbitrary object without any special knowledge about the object. This meta-object programming[5, 9] is especially useful for system components or utility programs such as debuggers. In an object-oriented system, reflective methods are encapsulated within the object; therefore to access the internal values of an object, the reflection code must be executed in the same address space where the data resides. Although this is the desired behavior in most cases, debugging is one case where this encapsulation of code and data may present a problem. The reason is that a program being debugged generally needs to be halted, i.e. its execution frozen at an arbitrary point, so that its values and states can be inspected reliably.
Current address: Hewlett Packard Laboratories, MS 1U-17, 1501 Page Mill Road, Palo Alto, CA 94304
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1031–1038, 2000. c Springer-Verlag Berlin Heidelberg 2000
1032
Ton Ngo and John Barton
In the case of debugging user applications, the debugger can still take full advantage of reflection since the user application is running on a stable system. The system can halt the application thread but continue to execute the debugger thread. Such a debugger is typically called in-process because it runs in the same process as the program being debugged. For the case of debugging system code, reflection is not possible for several reasons. First, halting the system itself would prevent the system from responding to any reflective queries. Second, allowing the system to execute the request will unintentionally change the states of the system while it is being inspected1 . Third, if the system crashes, the core image can be saved and inspected post-mortem, but no code can be executed. To solve these problems, the debugger must execute in a different process and control the system being developed through some debugging interface provided by the operating system. Consequently, such a debugger is called out-of-process. This situation arose in the development of Jalape˜ no[1, 2], a virtual machine for Java servers under development at the IBM T. J. Watson Research Center. Jalape˜ no is a compile-only system: instead of being interpreted, a method is compiled and optimized directly into machine instructions. Because the entire system is written in Java including the runtime, the compiler and garbage collector, reflection is used extensively to integrate the various components; consequently, there is a strong motivation for the Jalape˜ no debugger to use the same reflection facilities to inspect the system. In this paper, we present Remote Reflection as a technique that allows a program to execute a reflection method on an object that resides in a different process. In our case, this technique allows the debugger to make reflective queries to the Jalape˜ no system that has been completely halted in a different process. Remote reflection thus extends the power of reflection across different address spaces, improving the reusability of object-oriented codes. Although this technique was developed for Jalape˜ no and Java, we believe it is applicable to other Java implementations and other object-oriented languages. In the remainder of the paper, we will discuss remote reflection within the context of Java. Section 2 describes the general programming model for using remote reflection. Section 3 describes an implementation of remote reflection for Jalape˜ no. Section 4 illustrates the implementation with a detailed example, and possible further developments are discussed in Section 5. 1.1
Related Works
The Sun JDK debugger [8] and the more recent Java Platform Debugger Architecture [7] are also out-of-process and are based on reflection; however, there are several important differences. First, the Sun approach requires a debugging thread running internally in the virtual machine, dedicated to responding to external queries. For Jalape˜ no this is not possible for the reasons described earlier. 1
Consider debugging thread scheduling code: dispatching the debugger thread itself would change the thread states
Debugging by Remote Reflection
1033
Second, the reflection interface for the debugger is different from the internal reflection interface. In contrast, remote reflection requires no effort on the target system and the same reflection interface is used internally or externally.
2
Remote Reflection
Consider (1) a Java Virtual Machine (JVM) in a remote process that has been halted at an arbitrary point; (2) a program written based on the reflection interface of this remote JVM; and (3) another JVM in a local process executing this program. Remote reflection allows the program in the local JVM to execute a reflection method that operates directly on an object residing in a remote JVM. The key to remote reflection is a proxy object in the local JVM called the Remote Object which represents the real object in the remote JVM. As illustrated in Figure 1, the programming model for remote reflection is simple yet effective. The user specifies that certain methods of the reflection interface will return remote objects from a different JVM. These methods in the local JVM are said to be mapped to the remote JVM since they serve as the link between the two JVM’s. Once a remote object is obtained from a mapped method, all values or objects derived from it will also originate from the remote JVM. Aside from the list of mapped methods, a remote object is indistinguishable from a normal object in the local JVM from the program perspective. Consider a simple example in Figure 2. To compute the line number, the method Debugger.lineNumberOf() obtains a table of VM Method’s, selects the desired element and invokes its virtual getLineNumberAt() method. This reflection method then consults the object’s internal array to return a line number. Supposed that on a local JVM with remote reflection, the static method VM Dictionary.getMethods() has been mapped to an array of VM Method’s in the remote space. When we execute lineNumberOf(), the variable methodTable receives the initial remote object from VM Dictionary.getMethods(). The variable candidate then gets another remote object from accessing the remote array, and finally the method getLineNumberAt() is invoked on the remote object. The uniform treatment of local and remote objects provides the main advantage of remote reflection. Because a remote object is logically identical to a local object, a program uses the same reflection interface whether it executes in-process or out-of-process. As a result, the maintenance of both the reflection interface and programs using it is greatly simplified. A second advantage is that no effort is required in the remote JVM, since remote reflection relies on the underlying operating system to access the JVM address space. Finally, mapping per method instead of per class allows flexibility in selecting the object to be mapped. A class may have some instances in the local process and other instances in the remote process without conflict. While a mapped method is not necessarily tied to a single object in the remote JVM, in practice, it is more convenient to map access method which return a specific object.
1034
Ton Ngo and John Barton
remote JVM
local JVM classA.getObj()
real object classA xx
remote object
real object classA xx xx xx
xx xx
Fig. 1. Programming model for remote reflection: certain methods, e.g. classA.getObj(), are specially designated to return remote objects that are proxies for the real objects in the remote JVM. In this Figure, the boxes in each JVM represent objects with fields. There are two real instances of classA: one in the local JVM and one in the remote JVM.
3
Implementation
In this section, we describe an implementation in the Jalape˜ no system. In Java, remote reflection is supported at the level of the virtual machine by either the interpreter or the runtime compiler. Our debugging environment involves three components: the Jalape˜ no system being debugged, the debugger, and a Java interpreter that has been extended to support remote reflection. The extension includes managing the remote object and extending the bytecodes to operate on the remote object. Remote reflection also requires operating system support for access across processes. This functionality is typically provided by the system debugging interface, which in the Jalape˜ no implementation is the Unix ptrace facility. Our implementation is simplified by the fact that the debugger only makes queries and does not modify the remote JVM (except explicitly by a user command); therefore, we do not have to address the issue of creating new objects in the remote space. 3.1
Remote Object
The remote object is simply a wrapper that holds sufficient information to find the real object in the remote process. For Jalape˜ no, this includes the type of the object and its real address. Remote objects originate from a mapped method or another remote object. In first case, the address is provided to the interpreter from the process of building the boot image[1]. For the latter case, the address is computed based on the field offset from the address of the remote object. For native methods, a complete implementation will involve extending the JNI implementation to handle remote objects. However in our implementation, it was sufficient to clone the remote object and remote 1-D array of primitives because this satisfies the need of the debugger.
Debugging by Remote Reflection
3.2
1035
Bytecode Extensions
Since the initial remote object is obtained via a mapped method, the bytecode invokestatic or invokevirtual to invoke a method are extended as follows. The target class and method are checked against the mapping list. Those to be mapped are intercepted so that the actual invocation is not made. Instead, if the return type is an object, a remote object is created containing the type and the address of the corresponding object in the remote JVM. If the return type is a primitive, the actual value is fetched from the remote JVM. In addition, all bytecodes that operate on a reference need to be extended to handle remote objects appropriately - for Java, this includes 23 bytecodes. If the result of the bytecode is a primitive value, the interpreter computes the actual address, makes the system call to obtain the value from the remote address space, and pushes the value onto the local Java stack. If the result is an object, the interpreter computes the address of the field holding the reference, makes the system call to obtain the field value and pushes onto the Java stack a new remote object with the appropriate type.
4
Example
In this section, we return to the example in Figure 2 to analyze the actions that occur when the call lineNumberOf(5,4) is executed. For reference, Figure 2 also shows the bytecodes for the methods with dashed lines correlating the Java source lines with its bytecodes. In Figure 3, the box at the right represents the remote JVM, showing a number of objects that have been created in its space. Recall that the static method VM Dictionary.getMethods() has been mapped to the array of VM Method objects in the remote space. The states of the Java stack at successive points are shown in the top and bottom rows, labeled with highlighted numbers from 1 to 11. The state numbers are cross-referenced between Figure 2 and Figure 3. Also shown in Figure 3 are the remote objects (center) in the local JVM that serve as proxies to the corresponding real objects in the remote JVM. Due to limited space, we will only examine in details states 1-3 at the beginning and states 9-11 at the end; the remaining states exhibit similar behavior. First, the interpreter recognizes VM Dictionary.getMethods() as a mapped method and intercepts the bytecode invokestatic to create the initial remote object. The remote object contains the return type VM Method array and the address of the real array in the remote process. In Figure 3, the Java stack in state (1) shows the local variable methods on top of the stack and holding a reference to the newly created remote object. The following bytecodes astore 3, aload 3 and iload 1 make the preparation for accessing the remote array, resulting in Java stack state (2). When the interpreter executes the bytecode aaload to access an array element, it detects that the array reference is a remote object. Since it is an array of objects, the interpreter determines the element type, computes the address and pushes onto the stack the new remote object, resulting in Java stack state (3).
1036
Ton Ngo and John Barton
java source class Debugger { public int lineNumberOf(int methodNumber, int offset) { VM_Method[] methodTable = VM_Dictionary.getMethods(); VM_Method candidate = methodTable[methodNumber]; int lineNumber = candidate.getLineNumberAt(offset); return lineNumber; } } class VM_Method { private int[] lineTable; public int getLineNumberAt(int offset) { if (offset > lineTable.length) return 0; return lineTable[offset]; } }
compiled bytecode Method int lineNumberOf(int, int) 1 invokestatic #18 <Method VM_Method getMethods()[]> astore_3 aload_3 2 iload_1 3 aaload astore 4 aload 4 iload_2 4 invokevirtual #24 <Method int getLineNumberAt(int)> istore 5 iload 5 ireturn
compiled bytecode Method int getLineNumberAt(int) iload_1 5 aload_0 getfield #14
Fig. 2. Example: Java programs making reflective queries and the corresponding bytecodes. The dashed lines correlates the Java source lines with its bytecodes. The highlighted numbers refer to successive states of the Java stack during the program execution; they are cross-referenced with Figure 3.
The execution continues likewise with more remote objects created. To arrive at state (9), the bytecode getfield accesses the field lineTable and the interpreter creates another remote object on top of the stack. In state (10), the array index is pushed onto the stack and the interpreter executes iaload to access the array element. It detects that the array is remote and that the element type is an integer. The interpreter computes the address of the array element and makes the system call to read the value from the remote JVM space. In state (11), the value from the remote array is placed on the stack as a return value. It is worth noting that remote objects are always temporary; they only exist on the Java stack because they contain real addresses in the remote process that are only valid until the remote JVM resumes execution.
5
Status and Future Works
The interpreter with the remote reflection extension was completed and together with the Jalape˜ no debugger has been indispensable in the development of the Jalape˜ no system. Future extensions include several possibilities. For a production JVM system, debugging requires additional care since the system cannot be taken down and restarted for each debugging session. In this situation, remote reflection must be able to connect to the running system with-
Debugging by Remote Reflection
1037
Stack state in local process while executing bytecodes method intercepted by interpreter: generate first remote object methods
array access: get another remote object
...
5 methods
1
candidate
2
invoke virtual method on a remote object
...
4 candidate
Remote Java VM methodTable
4
3
0
remote object VM_Method[ ] address
VM_Method header
remote object VM_Method address
lineTable
remote object lineTable address
this 3
5
lineTable 3
6 get a field of a remote object
10 3
...
7 array length: get real value
lineTable 0
3
17
9
this
lineTable
3 lineTable
17
8
9
10
11 array access: get real int value for array element
Fig. 3. Example: the successive states of the Java stack (top and bottom rows) during the execution of the reflective queries, showing the remote objects being computed and resolved. The states are labeled with highlighted numbers to crossreference with the bytecodes in Figure 2. The Remote Java VM box (right) shows three real objects existing in the remote space, while the boxes in the center are the remote objects serving as proxies for the real objects.
out any significant side effect. A system facility other than ptrace will be necessary so that the running system would retain most of its own process control. Remote reflection can also be useful in the Java Platform Debugger Architecture of Java 2. We would only need to base the Java Debug Interface implementation directly on the internal reflection interface of the target JVM. This JDI implementation and a JDI-based debugger would run on another JVM that has been extended for remote reflection. This configuration would then bring to JPDA the same capabilities in debugging Jalape˜ no: low level debugging and the ability to halt the JVM to avoid perturbing its state. For distributed Java applications runing on several remote JVM’s, remote reflection can provide the convenience of a shared memory programming model, allowing them to readily access remote objects. However, since the applications would not be halted, synchronization as well as other issues will need to be studied carefully.
1038
6
Ton Ngo and John Barton
Conclusions
Reflection is an important addition to object-oriented systems. In this paper, we describe remote reflection, a transparent mapping technique that preserves the benefits of reflection in situations where it is necessary to decouple the code and the data involved in reflection because they reside in different address spaces. While the concept of reflection programming has been used across processes, our technique offers several advantages not present in previous efforts. First, it is not necessary to define a different reflection interface to use across processes; the same interface is used whether the program is in-process or out-of-process. Second, no effort is required in the target process; therefore it does not need to be functional and its state is not perturbed. We describe a simple programming model using remote reflection and an implementation in the Jalape˜ no system, a Java Virtual Machine developed at the IBM T. J. Watson Research Center. In this context, remote reflection is used to support an out-of-process debugger for the Jalape˜ no system code. Remote reflection allows the debugger to exploit the benefits of both in-process and out-of-process debugging, resulting in a very effective tool for developing an object-oriented system.
References [1] Bowen Alpern, Dick Attanasio, John J. Barton, Michael G. Burke, Perry Cheng, Jong-Deok Choi, Anthony Cocchi, Stephen Fink, David Grove, Michael Hind, Susan Flynn Hummel, Derek Lieber, Vassily Litvinov, Ton Ngo, Mark Mergen, Vivek Sarkar, Mauricio J. Serrano, Janice Shepherd, Stephen Smith, V. C. Sreedhar, Harini Srinivasan, and John Whaley. The Jalape˜ no Virtual Machine. IBM Systems Journal, 2000, Vol 39, No 1, pp 211-238. [2] Bowen Alpern, Dick Attanasio, John J. Barton, Anthony Cocchi, Susan Flynn Hummel, Derek Lieber, Ton Ngo, Mark Mergen, Janice Shepherd, and Stephen Smith. Implementing Jalape˜ no in Java. ACM SIGPLAN Conference on ObjectOriented Programming Systems, Languages and Applications (OOPSLA), November 1999, pp 314-324. [3] James Gosling, Bill Joy, and Guy Steele. The Java Language Specification. The Java Series. Addison-Wesley, 1996. [4] Dan Ingalls, Ted Kaehler, John Maloney, Scott Wallace, and Alan Kay. Back to the Future, The Story of Squeak. ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA), October 1997, pp 318-326. [5] Gregor Kiczales, Jim des Rivieres, and Daniel G. Bobrow. The Art of the Metaobject Protocol. The MIT Press, 1992. [6] Tim Lindholm and Frank Yellin. The Java Virtual Machine Specification. The Java Series. Addison-Wesley, 1996. [7] Sun Microsystems. Java 2 SDK Standard Edition. [8] Sun Microsystems. Java Development Kit 1.1. [9] Andreas Paepcke. Object-Oriented Programming: The CLOS Perspective. MIT Press, 1993.
Compiling Multithreaded Java Bytecode for Distributed Execution Gabriel Antoniu1 , Luc Boug´e1, Philip Hatcher2 , Mark MacBeth2 , Keith McGuigan2 , and Raymond Namyst1 1
2
LIP, ENS Lyon, 46 All´ee d’Italie, 69364 Lyon Cedex 07, France. [email protected] Dept. Computer Science, Univ. New Hampshire, Durham, NH 03824, USA. [email protected]
Abstract. Our work combines Java compilation to native code with a run-time library that executes Java threads in a distributed-memory environment. This allows a Java programmer to view a cluster of processors as executing a single Java virtual machine. The separate processors are simply resources for executing Java threads with true concurrency and the run-time system provides the illusion of a shared memory on top of the private memories of the processors. The environment we present is available on top of several UNIX systems and can use a large variety of network protocols thanks to the high portability of its run-time system. To evaluate our approach, we compare serial C, serial Java, and multithreaded Java implementations of a branch-and-bound solution to the minimal-cost map-coloring problem. All measurements have been carried out on two platforms using two different network protocols: SISCI/SCI and MPI-BIP/Myrinet.
1
Introduction
The Java programming language is an attractive vehicle for constructing parallel programs to execute on clusters of computers. The Java language design reflects two emerging trends in parallel computing: the widespread acceptance of both a threads programming model and the use of a distributed-shared memory (DSM). While many researchers have endeavored to build Java-based tools for parallel programming, we think most people have failed to appreciate the possibilities inherent in Java’s use of threads and a “relaxed” memory model. There are a large number of parallel Java efforts that connect multiple Java virtual machines by utilizing Java’s remote-method-invocation facility (e.g., [4, 5, 10, 13]) or by grafting an existing message-passing library (e.g., [7, 8]) onto Java. In our work we view a cluster as executing a single Java virtual machine. The separate nodes of the cluster are hidden from the programmer and are simply resources for executing Java threads with true concurrency. The separate
Mark MacBeth is currently affiliated with Sanders, A Lockheed Martin Company, PTP02-D001, P.O. Box 868, Nashua, NH, USA.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1039–1052, 2000. c Springer-Verlag Berlin Heidelberg 2000
1040
Gabriel Antoniu et al.
memories of the nodes are also hidden from the programmer and our implementation must support the illusion of a shared memory within the context of the Java memory model, which is “relaxed” in that it does not require sequential consistency. Our approach is most closely related to efforts to implement Java interpreters on top of a distributed shared memory [2, 6, 17]. However, we are interested in computationally intensive programs that can exploit parallel hardware. We expect the cost of compiling to native code will be recovered many times over in the course of executing such programs. Therefore we focus on combining Java compilation with support for executing Java threads in a distributed-memory environment. Our work is done in the context of the Hyperion environment for the highperformance execution of Java programs. Hyperion was developed at the University of New Hampshire and comprises a Java-bytecode-to-C translator and a run-time library for the distributed execution of Java threads. Hyperion has been ´ built using the PM2 distributed, multithreaded run-time system from the Ecole Normale Sup´erieure de Lyon [12]. As well as providing lightweight threads and efficient inter-node communication, PM2 provides a generic distributed-sharedmemory layer, DSM-PM2 [1]. Another important advantage of PM2 is its high portability on several UNIX platforms and on a large variety of network protocols (BIP, SCI, VIA, MPI, PVM, TCP). Thanks to this feature, Java programs compiled by Hyperion can be executed with true parallelism in all these environments. In this paper we describe the overall design of the Hyperion system, the strategy followed for the implementation of Hyperion using PM2, and a preliminary evaluation of Hyperion/PM2 by comparing serial C, serial Java, and multithreaded Java implementations of a branch-and-bound solution to the minimalcost map-coloring problem. The evaluation is performed on two different platforms using two different network protocols: SISCI/SCI and MPI-BIP/Myrinet.
2 2.1
The Hyperion System Compiling Java
Our vision is that programmers will develop Java programs using the workstations on their desks and then submit the programs for production runs to a “high-performance Java execution server” that appears as a resource on the network. Instead of the conventional Java paradigm of pulling bytecode back to their workstation for execution, programmers will push bytecode to the highperformance server for remote execution. Upon arrival at the server the bytecode is translated for native execution on the processors of the server. We utilize our own Java-bytecode-to-C compiler (java2c) for this task and then leverage the native C compiler for the translation to machine code. As an aside, note that the security issues surrounding “pushing” or “pulling” bytecodes can be handled differently. When pulling bytecodes, users want to
Compiling Multithreaded Java Bytecode for Distributed Execution
1041
bring applications from potentially untrusted locations on the network. The Java features for bytecode validation can be very useful in this context. In contrast, when “pushing” bytecodes to a high-performance Java server, conventional security methods might be employed, such as only accepting programs from trusted users. However, the Java security features could still be useful if one wanted to support an “open” Java server, accepting programs from untrusted users.
Prog.java
Sun’s javac compiler
Prog.class (bytecode)
java2c compiler
Prog.[ch]
gcc
Prog
libs
Fig. 1. Compiling Java programs with Hyperion Code generation in java2c is straightforward (see Figure 1). Each virtual machine instruction is translated directly into a separate C statement, similar to the approaches taken in the Harissa [11] or Toba [14] compilers. As a result of this method, we rely on the C compiler to remove all the extraneous temporary variables created along the way. Currently, java2c supports all non-wide format instructions as well as exception handling. The java2c compiler also includes an optimizer for improving the performance of object references with respect to the distributed-shared memory. For example, if an object is referenced on each iteration of a loop, the optimizer will lift out of the loop the code for obtaining a locally cached copy of the object. Inside the loop, therefore, the object can be directly accessed with low overhead via a simple pointer. This optimization needs to be supported by both compiler analysis and run-time support to ensure that the local cache will not be flushed for the duration of the loop. 2.2
The Hyperion Run-Time System Design
To build a user program, user class files are compiled (first by Hyperion’s java2c and then the generated C code by a C compiler) and linked with the Hyperion run-time library and with the necessary external libraries. The Hyperion run-time system is structured as a collection of modules that interact with one another (see Figure 2). We now present the main ones. Java API Support. Hyperion currently uses the Sun Microsystems JDK 1.1 as the basis for its Java API support. Classes in the Java API that do not include native methods can simply be compiled by java2c. However, classes with native methods need to have those native methods written by hand to fit the Hyperion
1042
Gabriel Antoniu et al.
design. Unfortunately, the Sun JDK 1.1 has a large number of native methods scattered throughout the API classes. To date, we have only implemented a small number of these native methods and therefore our support for the full API is limited. We hope that further releases of Java 2 (e.g., Sun JDK 1.2) will be more amenable to being compiled by java2c. Threads Subsystem. The threads module provides support for lightweight threads, on top of which Java threads can be implemented. This support obviously includes thread creation and thread synchronization. For portability reasons, we model the interface to this subsystem on the core functions provided by POSIX threads. Thread migration is also available, thanks to PM2’s support. We plan to use this feature in future investigations of dynamic and transparent application load balancing. Communication Subsystem. The communication module supports the transmission of messages between the nodes of a cluster. The interface is based upon message handlers being asynchronously invoked on the receiving end. This type of interface is mandatory since most communications, either one-way or roundtrip, must occur without any explicit contribution of the remote node: incoming requests are handled by a special daemon thread which runs concurrently with the application threads. For example, in our implementation of the Java memory model, one node of a cluster can asynchronously request data from another node. Memory Subsystem. The Java memory model [9] allows threads to keep locally cached copies of objects. Consistency is provided by requiring that a thread’s object cache be flushed upon entry to a monitor and that local modifications made to cached objects be transmitted to the central memory when a thread exits a monitor. Table 1 provides the key primitives of the Hyperion memory subsystem that are used to provide Java consistency. The DSM environment on top of which they are built is required to provide direct support for their implementation. This condition is fulfilled by the API of the DSM layer of PM2 (see Section 3.2 for additional details). loadIntoCache Load an object into the cache invalidateCache Invalidate all entries in the cache updateMainMemory Update memory with modifications made to objects in the cache get Retrieve a field from an object previously loaded into the cache put Modify a field in an object previously loaded into the cache
Table 1. Key DSM primitives
Compiling Multithreaded Java Bytecode for Distributed Execution
1043
Hyperion’s memory module also includes mechanisms for object allocation, garbage collection and distributed synchronization. Java monitors and the associated wait/notify methods are supported by attaching mutexes and condition variables from the Hyperion threads module to the Java objects managed by the Hyperion memory layer. Load Balancer. The load balancer is responsible for choosing the most appropriate node on which to place a newly created thread. The current strategy is rather simple: threads are assigned to nodes in a round-robin fashion. We use a distributed algorithm, with each node using round-robin placement of its locally created threads, independently of the other nodes. More complex load balancing strategies based on dynamic thread migration and on the interaction between thread migration and the memory consistency mechanisms are currently under development.
3
Hyperion/PM2 Implementation Details
The current implementation of Hyperion is based on the PM2 distributed multithreaded environment (Figure 2). PM2’s programming interface allows threads to be created locally and remotely and to communicate through RPCs (Remote Procedure Calls). PM2 also provides a thread migration mechanism that allows threads to be transparently and preemptively moved from one node to another during their execution. Such functionality is typically useful to implement dynamic load balancing policies. The interactions between thread migration and data sharing are handled through a distributed shared memory facility: the DSMPM2 [1] layer. Most Hyperion run-time primitives in the threads, communication and shared memory subsystems are implemented by directly mapping onto the corresponding PM2 functions. 3.1
Threads and Communication
Threads Subsystem. The threads component of Hyperion is a very thin layer that interfaces to Marcel (PM2’s thread library). Marcel is an efficient, user-level, POSIX-like thread package featuring thread migration. Most of the functions in Marcel’s API provide the same syntax and semantics as the corresponding POSIX Threads functions. However, it is important to note that the Hyperion thread component uses the PM2 thread component through the PM2 API and does not access the thread component directly, as would be typical when using a classical Pthreads-compliant package. PM2 implements a careful integration of multithreading and communication that actually required several modifications of the thread management functions (e.g., thread creation). Thus, it would be inefficient (and even dangerous) to bypass the PM2 API by using the underlying thread package directly.
1044
Gabriel Antoniu et al. Hyperion runtime Load balancer Thread subsystem
Native Java API
Memory subsystem
Communication subsystem
PM2 API: pm2_rpc, pm2_thread_create, etc. PM2 DSM subsystem Thread subsystem
Comm. subsystem
Fig. 2. Overview of the Hyperion software architecture Communication Subsystem. The communication component of Hyperion is implemented using PM2 remote procedure calls (RPCs), which allow PM2 threads to invoke the remote execution of user-defined services (i.e., functions). On the remote node, PM2 RPC invocations can either be handled by a preexisting thread or they can involve the creation of a new thread. This latter functionality allows us to easily implement Hyperion’s communication subsystem. PM2 utilizes a generic communication package [3] that provides an efficient interface to a wide-variety of high-performance communication libraries, including low-level ones. The following network protocols are currently supported: BIP (Myrinet), SISCI (SCI), VIA, MPI, PVM and TCP. 3.2
Memory Management
The memory management primitives described in Table 1 are implemented on top of PM2’s distributed-shared-memory layer, DSM-PM2 [1]. DSM-PM2 has been designed to be generic enough to support multiple consistency models. Sequential consistency and Java consistency are currently available. Moreover, for a given consistency model, alternative protocols (based on page migration and/or on thread migration) are provided. Also, new consistency models can be easily implemented using the existing generic DSM-PM2 library routines. DSM-PM2 is structured in layers. At the high level, a DSM protocol policy layer is responsible for implementing consistency models out of a subset of the available library routines and for associating each application data with its own consistency model. The library routines (used to bring a copy of a page to a thread, to migrate a thread to a page, to invalidate all copies of a page, etc.) are grouped in the lower-level DSM protocol library layer. Finally, these library routines are built on top of two base components: the DSM page manager and the DSM communication module. The DSM page manager is essentially dedicated
Compiling Multithreaded Java Bytecode for Distributed Execution
1045
to the low-level management of memory pages. It implements a distributed table containing page ownership information and maintains the appropriate access rights on each node. The DSM communication module is responsible for providing elementary communication mechanisms, such as delivering requests for page copies, sending pages, and invalidating pages. The DSM-PM2 user has three alternatives that may be utilized according to the user’s specific needs: (1) use a built-in protocol, (2) build a new protocol out of a subset of library routines, or (3) write new protocols using the API of the DSM page manager and DSM communication module (for more elaborate features not implemented by the library routines). The Hyperion DSM primitives (loadIntoCache, updateMainMemory, invalidateCache, get and put) have been implemented using this latter approach. Object replication: main memory and caches. To implement the concept of main memory specified by the Java model, the run-time system associates a home node to each object. The home node is in charge of managing the reference copy. Initially, the objects are stored on their home nodes. They can be replicated if accessed on other nodes. Note that at most one copy of an object may exist on a node and this copy is shared by all the threads running on that node. Thus, we avoid wasting memory by associating caches to nodes rather than to threads. Access detection and modification recording. Hyperion uses specific access primitives to shared data (get and put), which allows us to use explicit checks to detect if an object is present (i.e., has a copy) on the local node. If the object is present, it is directly accessed, else the page(s) containing the object is locally cached. Thanks to the access primitives, the modifications can be recorded at the moment when they are carried out. For this purpose, a bitmap is created on a node when a copy of the page is received. The put primitive uses it to record all writes to the object, using object-field granularity. All local modifications are sent to the home node of the page by the updateMainMemory primitive. Implementing objects on top of pages. Java objects are implemented on top of DSM-PM2 pages. If an object spans several pages, all the pages are cached locally when the object is loaded. Consequently, loading an object into the local cache may generate prefetching, since all objects on the corresponding page(s) are actually brought to the current node. Similarly, when updating the master copy of an object, other objects located on the same page will get their master copy updated. This implementation has been carefully designed to be fully compliant with Java consistency [9]. Object ownership. Our implementation allocates all Java objects within the section of memory controlled by DSM-PM2. We align Java objects on 2k byte boundaries. An object reference is basically the address of the object, but now we can use the bottom k bits to store the node number of the owner of the object. (k can be adjusted to accommodate larger number of nodes, at the expense of increasing internal fragmentation.) This allows us to do an efficient ownership test in the loadIntoCache primitive with a bitwise
1046
Gabriel Antoniu et al.
AND, a subtract, and a test against zero. If the ownership test fails, then the DSM-PM2 page table is consulted to see if the page containing the object is locally cached. In addition, each object is given a standard header and one of the bits of the header is used to indicate if the object is a cached copy or not. This bit is used by the put primitive to quickly determine whether the page is locally owned or not. If not, the modification needs to be recorded in the bitmap associated with the DSM-PM2 page holding the cached copy of the object.
4 4.1
Performance Evaluation: Minimal-Cost Map-Coloring Experimental Conditions and Benchmark Programs
We have implemented branch-and-bound solutions to the minimal-cost mapcoloring problem, using serial C, serial Java, and multithreaded Java. These programs have first been run on a eight-node cluster of 200 MHz Pentium Pro processors, running Linux 2.2, interconnected by a Myrinet network and using MPI implemented on top of the BIP protocol [15]. We have also executed the programs without any modification on a four-node cluster of 450 MHz Pentium II processors running Linux 2.2, interconnected by a SCI network using the SISCI protocol. The serial C program is compiled using the GNU C compiler, version 2.7.2.3 with -O6 optimization, and runs “natively” as a normal Linux executable. The Java programs are translated to C by Hyperion’s java2c compiler, the generated C code is also compiled by GNU C with -O6 optimization, and the resulting object files are linked with the Hyperion/PM2 run-time system. The two serial programs use identical algorithms, based upon storing the search states in a priority queue. The queue first gives priority to states in the bottom half of the search tree and then secondly sorts by bound value. (Giving priority to the states in the bottom half of the search tree drives the search to find solutions more quickly, which in turn allows the search space to be more efficiently pruned.) The parallel program does an initial, single-threaded, breadth-first expansion of the search tree to generate sixty-four search states. Sixty-four threads are then started and each one is given a single state to expand. Each thread keeps its own priority queue, using the same search strategy as employed by the serial programs. The best current solution is stored in a single location, protected by a Java monitor. All threads poll this location at regular intervals in order to effectively prune their search space. Maintaining a constant number of threads across executions on different size clusters helps to keep fairly constant the aggregate amount of work performed across the benchmarking runs. However, the pattern of interaction of the threads (via the detection of solutions) does vary and thus the work performed also varies slightly across different runs. All programs use a pre-allocated pool of search-state data objects. This avoids making a large number of calls to either the C storage allocation primitives
Compiling Multithreaded Java Bytecode for Distributed Execution
1047
(malloc/free) or utilizing the Java garbage collector. (Our distributed Java garbage collector is still under development.) If the pool is exhausted, the search mechanism switches to a depth-first strategy until the pool is replenished. For benchmarking, we have solved the problem of coloring the twenty-nine eastern-most states in the USA using four colors with different costs. Assigning sixty-four threads to this problem in the manner described above, and using Hyperion’s round-robin assignment of threads to nodes, is reasonably effective at evenly spreading the number of state expansions performed around a cluster, if the number of nodes divides evenly into the number of threads. (In the future we plan to investigate dynamic and transparent load balancing approaches that utilized the thread migration features of PM2.) 4.2
Overhead of Hyperion/PM2 vs. Hand-Written C Code
First, we compare the performance of the serial programs on a single 450 MHz Pentium II processor running Linux 2.2. Both the hand-written C program and the C code generated by java2c were compiled to native Pentium II instructions using gcc2.7.2.3 with option -O6. Execution times are given in seconds. Hand-written C 63 Java via Hyperion/PM2 324 Java via Hyperion/PM2, in-line DSM checks disabled 168 Java via Hyperion/PM2, array bound checks also disabled 98 We consider this a “hard” comparison because we are comparing against hand-written, optimized C code and because the amount of straight-line computation is minimal. This application features a large amount of object manipulation (inserting to and retrieving from the priority queue; allocating new states from the pool and returning “dead-end” states to the pool) with a relatively small amount of computation involved in expanding and evaluating a single state. In fact, the top two lines in the table demonstrate that the original Hyperion/PM2 execution is roughly five times slower than the execution of the hand-written C program. The bottom two lines in the table help explain the current overheads in the Hyperion/PM2 implementation of the Java code and represent cumulative improvements to the performance of the program. In the third line of the table, the Java code is executed on a single node with the in-line checks disabled: inline checks are used by Hyperion to test for the presence or the absence of a given Java object at the local node in the distributed implementation; as there is only one node at work in the case at stake, they are always satisfied. In the fourth line of the table, the array bound checks are additionally disabled. This last version can be considered as the closest to the hand-written C code. It is only 55% slower. A comparison with hand-written C++ code would probably be more fair to Hyperion, and would probably result in an even lower gap. We can draw two conclusions from these figures. First, the in-line checks used to implement the Hyperion DSM primitives (e.g., loadIntoCache, get and put)
1048
Gabriel Antoniu et al.
are very expensive for this application. By disabling these checks in the C code generated by java2c, we save nearly 50% of the execution time. (This emphasizes the map-coloring application’s heavy use of object manipulation and light use of integer or floating-point calculations.) For this application it may be better to utilize a DSM-implementation technique that relies on page-fault detection rather than in-line tests for locality. This can be easily done within the context of DSM-PM2’s generic support and we are currently evaluating this alternative, i.e., in-line vs. page-fault checks, with an expanded set of applications. Second, the cost of the array-bounds check in the Java array-index operation, at least in the Hyperion implementation, is also quite significant. We implement the bounds check by an explicit test in the generated C code. In the mapcoloring application, arrays are used to implement the priority queues and in the representation of search states. In both cases the actual index calculations are straightforward and would be amenable to optimization by a compiler that supported the guaranteed safe removal of such checks. Such an optimization could be implemented in java2c. Alternatively this optimization might be done by analysis of the bytecode prior to the execution of java2c. Or, the optimization could be performed on the generated C code. We plan to further investigate these alternatives in the future. 4.3
Performance of the Multithreaded Version
Next, we provide the performance of the multithreaded version of the Java program on the two clusters described in Section 4.1. Parallelizability results are presented in Figure 3: the multi-node execution times are compared to the single-node execution time of the same multithreaded Java program run with 64 threads. On the 200 MHz Pentium Pro cluster using MPI-BIP/Myrinet, the execution time decreases from 638 s for a single-node execution to 178 s for an 4-node execution (90% efficiency), and further to 89 s for an 8-node execution (still 90% efficiency). On the 450 MHz Pentium II cluster using SISCI/SCI, the efficiency is slightly lower (78% on 4 nodes), but the execution time is significantly better: the program runs in 273 s on 1 node, and 89 s on 4 nodes. Observe that the multithreaded program on one 450 MHz Pentium II node follows a more efficient search path for the particular problem being solved than its serial, single-threaded version reported in Section 4.1. The efficiency decreases slightly as the number of nodes increases on a cluster. This is due to an increasing number of communications that are performed to the single node holding the current best answer. With a smaller number of nodes, there are more threads per node and a greater chance that, on a given node, requests by multiple threads to fetch the page holding the best answer can be satisfied by a single message exchange on the network. (That is, roughly concurrent page requests at one node may be satisfied by one message exchange.) We believe these results indicate the strong promise of our approach. However, further study is warranted. We plan to investigate the performance under
Compiling Multithreaded Java Bytecode for Distributed Execution
1049
8 7
Parallelizability
6 5 4 3 200 MHz, MPI-BIP/Myrinet 450 MHz, SISCI/SCI Ideal parallelizability
2 1
1
2
3
4
5
6
7
8
Number of nodes
Fig. 3. Parallelizability results for the multithreaded version of our Java program solving the problem of coloring the twenty-nine eastern-most states in the USA using four colors with different costs. Tests have been done on two cluster platforms: 200 MHz Pentium Pro using MPI-BIP/Myrinet and 450 MHz Pentium II using SISCI/SCI. The program is run in all cases with 64 threads.
Hyperion/PM2 of additional Java multithreaded programs, including applications converted from the SPLASH-2 benchmark suite.
5
Related Work
The use of Java for distributed parallel programming has been the object of a large number of research efforts during the past several years. Most of the recently published results highlight the importance of transparency with respect to the possibly distributed underlying architecture: multithreaded Java applications written using the shared-memory paradigm should run unmodified in distributed environments. Though this goal is put forward by almost all distributed Java projects, many of them fail to fully achieve it. The JavaParty [13] platform provides a shared address space and hides the inter-node communication and network exceptions internally. Object and thread location is transparent and no explicit communication protocol needs to be designed nor implemented by the user. JavaParty extends Java with a preprocessor and a run-time system handling distributed parallel programming. The source code is transformed into regular Java code plus RMI hooks and the
1050
Gabriel Antoniu et al.
latter are fed into Sun’s RMI compiler. Multithreaded Java programs are turned into distributed JavaParty programs by identifying the classes and objects that need to be spread across the distributed environment. Unfortunately, this operation is not transparent for the programmer, who has to explicitly use the keyword remote as a class modifier. A very similar approach is taken by the Do! project [10], which obtains distribution by changing the framework classes used by the program and by transforming classes to transparently use the Java RMI, while keeping an unchanged API. Again, potentially remote objects are explicitly indicated using the remote annotation. Another approach consists in implementing Java interpreters on top of a distributed shared memory [6, 17] system. Java/DSM [17] is such an example, relying on the Treadmarks distributed-shared-memory system. Nevertheless, using an off-the-shelf DSM may not lead to the best performance, for a number of reasons. First, to our knowledge, no available DSM provides specific support for Java consistency. Second, using a general-purpose release-consistent DSM sets up a limit to the potential specific optimizations that could be implemented to guarantee Java consistency. Locality and caching are handled by the DSM support, which is not flexible enough to allow the higher-level layer of the system to configure its behavior. cJVM [2] is another interpreter-based JVM providing a single image of a traditional JVM while running on a cluster. Each cluster node has a cJVM process that implements the Java interpreter loop while executing part of the application’s Java threads and containing part of the application objects. In contrast to Hyperion’s object caching approach, cJVM executes methods on the node holding the master copy of the object, but includes optimizations for data caching and replication in some cases. Our interest in computationally intensive programs that can exploit parallel hardware justifies three main original design decisions for Hyperion. First, we rely on a Java-to-C compiler to transform bytecode to native code and we expect the compilation cost will be recovered many times over in the course of executing such programs. We believe this approach will lead to much better execution times compared to the interpreter-based approaches mentioned above. Second, Hyperion uses the generic, multi-protocol DSM-PM2 run-time system, which is configured to specifically support Java consistency. Finally, we are able to take advantage of fast cluster networks, such as SCI and Myrinet, thanks to our portable and efficient communication library provided by the PM2 run-time system.
6
Conclusion
We propose utilizing a cluster to execute a single Java Virtual Machine. This allows us to run Java threads completely transparently in a distributed environment. Java threads are mapped to native threads available on the nodes and run with true concurrency. An original feature of our system is its use of a Java-to-C compiler (and hence of machine code). Hyperion’s implementation supports a
Compiling Multithreaded Java Bytecode for Distributed Execution
1051
globally shared address space via the DSM-PM2 run-time system that we configured to guarantee Java consistency. The generic support provided by DSM-PM2 allowed us to implement Java-specific optimizations that are not available in standard DSM systems, such as Treadmarks. Thanks to the portability of the PM2 run-time support, the full system we present is available on top of several UNIX systems and can use a large variety of network protocols. To evaluate our approach, we compare serial C, serial Java, and multithreaded Java implementations of a branch-and-bound solution to the minimal-cost map-coloring problem. We report good parallelizability on two platforms using two different network protocols: SISCI/SCI and MPI-BIP/Myrinet. References [1] Gabriel Antoniu, Luc Boug´e, and Raymond Namyst. Generic distributed shared memory: the DSM-PM2 approach. Research Report RR2000-19, LIP, ENS Lyon, Lyon, France, May 2000. [2] Y. Aridor, M. Factor, and A. Teperman. cJVM: A single system image of a JVM on a cluster. In Proceedings of the International Conference on Parallel Processing, Fukushima, Japan, September 1999. [3] Luc Boug´e, Jean-Fran¸cois M´ehaut, and Raymond Namyst. Efficient communications in multithreaded runtime systems. In Parallel and Distributed Processing. Proc. 3rd Workshop on Runtime Systems for Parallel Programming (RTSPP ’99), volume 1586 of Lect. Notes in Comp. Science, pages 468–182, San Juan, Puerto Rico, April 1999. Held in conjunction with IPPS/SPDP 1999., Springer-Verlag. [4] F. Breg, S. Diwan, J. Villacis, J. Balasubramanian, E. Akman, and D. Gannon. Java RMI performance and object model interoperability: Experiments with Java/HPC++. In Proceedings of the ACM 1998 Workshop on Java for HighPerformance Network Computing, pages 91–100, Palo Alto, California, February 1998. [5] D. Caromel, W. Klauser, and J. Vayssiere. Towards seamless computing and metacomputing in Java. Concurrency: Practice and Experience, 10:1125–1242, 1998. [6] X. Chen and V. Allan. MultiJav: A distributed shared memory system based on multiple Java virtual machines. In Proceedings of the Conference on Parallel and Distributed Processing Techniques and Applications, Las Vegas, Nevada, June 1998. [7] A. Ferrari. JPVM: Network parallel computing in Java. In Proceedings of the ACM 1998 Workshop on Java for High-Performance Network Computing, pages 245–249, Palo Alto, California, 1998. [8] V. Getov, S. Flynn-Hummell, and S. Mintchev. High-performance parallel programming in Java: Exploiting native libraries. In Proceedings of the ACM 1998 Workshop on Java for High-Performance Network Computing, pages 45–54, Palo Alto, California, February 1998. [9] J. Gosling, W. Joy, and G. Steele Jr. The Java Language Specification. AddisionWesley, Reading, Massachusetts, 1996. [10] P. Launay and J.-L. Pazat. A framework for parallel programming in Java. In High-Performance Computing and Networking (HPCN ’98), volume 1401 of Lect. Notes in Comp. Science, pages 628–637. Springer-Verlag, 1998.
1052
Gabriel Antoniu et al.
[11] G. Muller, B. Moura, F. Bellard, and C. Consel. Harissa: A flexible and efficient Java environment mixing bytecode and compiled code. In Third Usenix Conference on Object-Oriented Technologies and Systems, Portland, Oregon, June 1997. [12] R. Namyst and J.F. Mehaut. PM2 : Parallel Multithreaded Machine: A computing environment for distributed architectures. In ParCo’95 (Parallel Computing), pages 279–285. Elsevier Science Publishers, September 1995. [13] M. Philippsen and M. Zenger. JavaParty — transparent remote objects in Java. Concurrency: Practice and Experience, 9(11):1125–1242, November 1997. [14] T. Proebsting, G. Townsend, P. Bridges, J. Hartman, T. Newsham, and S. Watterson. Toba: Java for applications - a way ahead of time (WAT) compiler. In Third Usenix Conference on Object-Oriented Technologies and Systems, Portland, Oregon, June 1997. [15] L. Prylli and B. Tourancheau. BIP: A new protocol designed for high performance networking on Myrinet. In Proceedings of First Workshop on Personal Computer Based Networks Of Workstations, volume 1388 of Lect. Notes in Comp. Science. Springer-Verlag., April 1998. [16] T. von Eicken, D. Culler, S. Goldstein, and K. Schauser. Active messages: A mechanism for integrated communication and computation. In International Symposium on Computer Architectures, pages 256–266, Gold Coast, Australia, May 1992. [17] W. Yu and A. Cox. Java/DSM: A platform for heterogeneous computing. In Proceedings of the Workshop on Java for High-Performance Scientific and Engineering Computing, Las Vegas, Nevada, June 1997.
A More Expressive Monitor for Concurrent Java Programming1 Hsin-Ta Chiao, Chi-Houng Wu, and Shyan-Ming Yuan Department of Computer and Information Science National Chiao Tung University 1001 Ta Hsueh Rd., Hsinchu 300, Taiwan {gis84532, gis86501, smyuan}@cis.nctu.edu.tw Abstract. The thread synchronization mechanism employed by Java is derived from the Hoare’s monitor concept. In order to minimize its implementation complexity, the monitor provided by Java is quite primitive. This design decision prefers simple concurrent objects and single-thread programs. However, we think the Java monitor is over simplified for developing more elaborated concurrent objects. Besides, several features of the Java monitor will bring extra overhead when thread contention gets higher. Currently, we have identified five drawbacks of the Java monitor. In this paper, we will first analyze these drawbacks in depth, and then propose a new monitor-based synchronization mechanism called EMonior. It has better expressive power, and introduces fewer overheads than the Java monitor when contention occurs. EMonior uses a preprocessor to translate the Java programs containing EMonitor syntax to regular Java programs that invoke EMonitor class libraries. It is very suitable to replace the Java monitor with the EMonitor when developing elaborate concurrent objects or high-contention concurrent systems.
1 The Introduction to Java Monitor The thread synchronization mechanism offered by Java is a monitor [5], which is a simplification of the original Hoare’s monitor concept [8]. For implementing monitors, each Java object contains a monitor lock and a condition queue. The keyword synchronize can be inserted into a method’s definition for specifying it as a synchronized method. In each Java object, there is always only one synchronized method can be running at any moment. In addition to synchronized methods, Java also offers synchronized blocks for reducing the size of critical sections. Besides, for condition synchronization, Java provides the following three methods in each object: wait( ), notify( ), and notifyAll( ). They can be invoked only inside a synchronized method or inside a synchronized block. The design philosophy of Java monitor is to keep it as simple as possible. Consequently, its simplicity also leads to efficient implementation. All concurrent objects that can be implemented by the Java monitor easily will benefit from this design principle. Besides the multi-thread, concurrent programs, this design principle 1
This work was supported both by the National Science Council grant NSC88-2213-E-009-087 and the industry reaearch program 89-EC-2-A-17-0285-006 of the ROC Economic Bureau. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1053-1060, 2000. Springer-Verlag Berlin Heidelberg 2000
1054
Hsin-Ta Chiao, Chi-Houng Wu, and Shyan-Ming Yuan
will also reduce the run-time overhead of single-thread programs. This is because Java offers only one suite of class libraries to both single-thread and multi-thread programs. Hence, all classes in the class libraries have to employ the Java monitor if necessary to guarantee the thread-safe property. Once the Java monitor’s overhead is shrunk, both the single-thread and multi-thread programs will have a more efficient class libraries. In addition, by employing the thin-lock approach that has first been implemented in JDK 1.1.2 for AIX [3], the Java monitor lock can be further optimized for the situation where no contention occur. The similar techniques are currently employed by many Java virtual machines. Hence, if the required concurrent objects are simple and the contention level of the execution environment is low, the current design of Java monitor seems very successful. However, for designing elaborate concurrent Java objects, we think that the Java monitor is over simplified and lacks of expressive power. Besides, several features of the Java monitor will bring extra overhead when the contention between the cooperative threads gets higher. Consequently, using Java monitor in the above scenarios may both complicate program design and produce less efficient programs. In this paper, we will first analyze the five known drawbacks of Java monitor in section 2. Then, in section 3, we will propose a new monitor-based thread synchronization mechanism called EMonitor for replacing the Java monitor. The performance comparison between the Java monitor and EMonitor are shown in section 4. Last, section 5 concludes this paper.
2 The Drawbacks of Java Monitor 2.1 The Problems Introduced by Single Condition Queue As introduced in section 1, there is only one condition queue in a Java object. If all pending threads in a condition queue wait for the same event, no further problem will occur. However, if this preferred property is spoiled, the event triggering a notify( ) method may be different from the event required by the thread waked by the notify( ). For solving this problem, the awaked thread should check whether the required event has happened. If not, it will first use notify( ) to resume another pending thread in the condition queue, and then invoke wait( ) to block itself again. The above steps are repeated until a correct pending thread is picked. Another solution is to replace notify( ) with notifyAll( ) to resume all the pending threads at once. Each awaked thread has to determine whether the required event is happened. However, these two approach both wake up many irrelevant pending threads, and extra thread context switches are introduced. This overhead is even worse if a Java virtual machine directly maps a Java thread to an operating system’s kernel thread. [6]. Several programming techniques [12] or design patterns [10] have been proposed for solving the Java monitor’s single-condition-queue problem. All these solution use extra supporting objects to simulate the presence of multiple condition queues inside a Java object. However, we find that designing a concurrent Java object by cooperating with these supporting objects is complex, error-prone and deadlock-prone. Hence, we
A More Expressive Monitor for Concurrent Java Programming
1055
think it is still necessary to provide multiple condition queues inside a monitor at the programming language level directly. 2.2 No Additional Support for Scheduling Scheduling means to choose a preferred request among a group of incoming requests. This also implies the scheduler must possess some global information about these requests for scheduling decision. For example, many existent monitors [1][11][13] [14] offer a prioritized condition queue to simplify the task of writing static scheduling programs. However, Java monitor barely provides a random condition queue, and does not support any other facilities that can help a program to maintain the global information for scheduling. Consequently, a Java program has to keep the required scheduling information in other more expensive ways. 2.3 The Troubles Caused by No-Priority Monitor The behavior of different monitors can be classified by a generalized queue model [2]. In this model, a monitor lock contains three queues: an entry queue (holding the incoming threads), a waiting queue (holding the signalled or the notified threads), and a signaller queue (holding the signaller or the notifying thread). For preventing starvation, the entry queue’s priority Ep should always be the lowest. If a monitor’s Ep is equal to either the Sp (the signaller queue’s priority) or the Wp (the waiting queue’s priority), we refer to it as a no-priority monitor. Otherwise, it is a priority monitor. Because the Java monitor use non-blocking signal semantic, its monitor lock contains no signaller queue. Furthermore, since the property of the lock waiting queue is left unspecified in the Java specification [5], it is reasonable to assume Java monitor’s Ep and Wp are equal. Hence, we can classify the Java monitor as a nopriority one. However, no-priority monitors have two problems. The first problem is wait( )’s postcondition may be different from the corresponding notify( )‘s pre-condition [1][2] [12]. In fact, this condition-breakup problem is caused by other imcoming threads that preempt the notified thread, and enter the monitor to destroy the notify( )’s precondition. Once the wait( ) returns, if its post-condition is identical to the notify( )’s pre-condition, the statements behind it are allowed to proceed. Otherwise, the notified thread has to invoke wait( ) again to block itself. This will cause extra context switches and sink the program performance. In addition to the condition-breakup problem, no-priority monitors may also complicate the task of writing scheduling programs [1][2][7]. In a scheduler implemented by a monitor, a calling thread often judges whether itself can be scheduled immediately at the beginning of a monitor method. If it cannot, it will block itself on the condition queue. Then, when a pending thread in the condition queue can be scheduled to run, it will be notified by another thread, and stay in waiting queue for entering the monitor again. However, under some boundary conditions (for instance, the condition queue becomes empty after it is notified), due to the property of no-priority monitors, the notified thread may be preempted by the threads in the entry queue. This disturbance will cause the execution order of threads
1056
Hsin-Ta Chiao, Chi-Houng Wu, and Shyan-Ming Yuan
to violate the desired scheduling rule. Consequently, if no-priority monitors are adopted to write scheduling programs, programmers have to consider the above interference. 2.4 Insufficient Signal Semantics In a Java monitor, both notify( ) and notifyAll( ) do not release the monitor lock, and they merely resume either one or all pending threads in the condition queue, respectively. Hence, the semantics of both notify( ) and notifyAll( ) belong to the nonblocking signal [2][12]. However, besides the non-blocking signal, there are other practical signal semantics. For instance, if blocking signal [2] is employed, wait( )’s post-condition will always match the corresponding notify( )’s pre-condition. Therefore, the blocking-signal monitor is easier to use. However, the performance of programs written in the non-blocking signal is better than the ones written in the blocking signal. Hence, good performance is the chief advantage of the non-blocking signal. In addition, the immediate-return signal semantic is also worthy to be considered. We thought the signal and exit action of this semantic is useful. It can reduce the overhead introduced by a thread that calls notify( ) and then instantly leaves the monitor. Another kind of action offered by the immediate-return signal semantic is the signal and wait action. Since implementing this action will cause different kinds of condition queues (if more than one kind of condition queue is offered) depend on each other, we suggest to abandon it. 2.5 Deadlock of Inter-monitor Nested Calls Java uses nested mutually exclusive locks to implement synchronized methods. Consequently, deadlock will not occur during an intra-monitor nested call. However, deadlock may arise in inter-monitor nested calls. This can be further divided into two categories. The first kind of deadlock is the mutually-dependent deadlock, which is well known and has been figured out in [4][12][15]. The second kind of deadlock is the condition-wait deadlock [9]. For preventing the deadlock arise in inter-monitor nested calls, the open call semantic [1][9] can be employed. Before a thread performs an open inter-monitor nested call, it will release the caller monitor lock. After the inter-monitor nested call returns, this thread will acquire the monitor lock again for reentering the caller monitor. Since each thread always holds at most one monitor lock, the mutually-dependent deadlock and the condition wait deadlock are eliminated. However, the open calls enforce extra restrictions on the programming style. Before invoking an open call, the calling thread has to transfer the caller monitor’s state to a consistent state, in which all the monitor invariants are true.
3 Our Solution In this section, we will propose a new monitor-based synchronization mechanism – the EMonitor. It is designed to replace the Java monitor for constructing elaborate concurrent Java objects, or for building the concurrent systems with higher contention level.
A More Expressive Monitor for Concurrent Java Programming
1057
3.1 The Characteristics of the EMonitor The EMonitor is a priority monitor. It supports multiple signal semantics within a monitor. These semantics are blocking signal, non-blocking signal (include nonblocking signal all), and the signal and exit action of the immediate-return signal semantic. In addition, EMonitor provides two different kinds of condition queues – the prioritized condition queues and the FIFO condition queues. The FIFO condition queues are very generalized and have enough expressive power to deal with more complex dynamic scheduling problems. Of course, there can be more than one condition queue within an EMonitor object. Since preventing deadlock and keeping the programming style simple are conflicting requirements, we determine to provide both the open calls and the closed calls (the original semantic of Java’s inter-monitor nested call.) 3.2 The Syntax and Implementation of EMonitor Our EMonitor can be used in three ways, and each of them has a different granularity of mutual exclusion. The first way is through the EMonitored classes. We add a new class modifier, EMonitored, to Java. If it is inserted into the declaration of a Java class, this class will become an EMonitored class. In an instance of an EMonitored class, only one method invocation can be running at the same time. The other two ways of using EMonitor are through the EMonitored methods and the EMonitored blocks. The syntax and functionality of these two constructs are very similar to the synchronized methods and synchronized blocks of Java (except the keyword becomes EMonitored.) We treat the EMonitor as a language extension to Java. A preprocessor is responsible for translating a Java program using the EMonitor to a pure Java program that invokes the EMonitor class libraries. Both the preprocessor and the EMonitor class libraries are implemented only by Java itself. Consequently, the Java virtual machine requires no modification. The EMonitor class libraries contains five primary classes, and they are the EMonitor class, the EMonitoredThread class, the Condition class, the PrioritizedCondition class, and the FIFOCondition class. The EMonitor class implements a nested lock – the EMonitor lock. Unlike the Java monitor lock, it contains an extra queue - the return queue. It holds the threads return from the issued open calls. The EMonitoredThread class itself is a subclass of the Java’s Thread class. It contains an extra stack to store the EMonitor locks’ nested count for each unreturned open call. Hence, each thread using the EMonitor has to be an instance of the EMonitoredThread class or its subclass. As mentioned in the previous subsection, our EMonitor provides two kinds of condition queues. The Condition class is an abstract class that defines the common properties for both the prioritized and FIFO condition queues. The PrioritizedCondition class is for the prioritized condition queues The FIFOCondition class is for the FIFO condition queues. In a FIFO condition queue, each pending thread can be associated with a customized object that is specified when the thread invokes the Wait( ) methods. Therefore, the scheduling information about a pending thread can be stored in the object attached with it. Besides, a specialized
1058
Hsin-Ta Chiao, Chi-Houng Wu, and Shyan-Ming Yuan
Wait(Object attribute, int n) method is also offered. It can insert the calling thread into a designated location (indexed by n) of a FIFO condition queue. For retrieving the scheduling information, the Dump( ) and the Peek( ) method are provided. The Swap( ) function is offered to swap the position of two pending threads. Besides, for all signal semantics except the non-blocking signal all, the corresponding indexed signal methods are provided in this class. Consequently, we can signal a specific pending thread through one of these methods by assigning its index in a condition queue.
4 Experimental Result In this section, we will compare the performance of our EMonitor with the Java monitor. We first construct two multiple-reader/single-writer locks. One is implemented by the EMonitor, and another is implemented by the Java monitor. Then, we measure the lock overhead introduced by each kind of implementation. Release lock
Thread is terminated
Acquire lock
Reader/writer lock
Lock request issued
Pool of lock requests
Fig. 1. The experiment for measuring the average lock overhead Please refer to Figure 1. All the necessary threads are created in advance, and then put into the lock request pool. When the experiment starts, the lock requests (threads) will be issued from the lock request pool in a one-by-one fashion. Each issued thread intends to acquire an assigned lock from the reader/writer lock, and we always fix the ratio of read locks to write locks as 6:4. Once the desired lock is granted, the issued thread will perform an operation, which takes about 3.9 ms. After this operation is finished, the lock held by this thread will be released. In an experiment, we measure the elapsed time of each thread since it starts to acquire a lock, until it releases the lock. Then, we calculate the average elapsed time per each thread. In this section, we conduct two different kinds of experiments to reveal the properties of both the EMonitor and the Java monitor. In the first experiment, we fix the number of issued lock requests as 500. Then, we observe the change in the lock overheads by altering the time interval for issuing a new lock request. The curve of the measured lock overheads is shown in Figure 2(a). In the second experiment, we fix the interval between lock requests as 1 ms, and vary the total number of lock requests. Furthermore, this short interval will cause all issued threads (except the first one) to wait for the desired locks. Figure 2(b) shows the measured lock overheads of this experiment. In this section, all the experimental results are gathered in an unloaded, 300MHz Sun UltraSPARC-II machine. The operation system is Sun Solaris 2.6, and the version of JDK is 1.1.3.
A More Expressive Monitor for Concurrent Java Programming 0.5 Average Lock Overhead (ms)
Average Lock Overhead (ms)
0.5
1059
0.4 0.3 0.2 0.1 0 0
0.5
1
EMonitor Java Monitor
1.5
2
2.5
3
3.5
Interval between Requests (ms)
(a)
4
4.5
5
5.5
0.4
0.3 0.2
0.1 0 100
200
300
EMonitor Java Monitor
400
500
600
700
800
900
1000
Total Number of Requests
(b)
Fig. 2 (a) shows the experimental results for fixing the total number of lock requests. (b) shows the experimental results for fixing the time interval between lock requests.
In Figure 2(a), we can observe that the lock overheads increase drastically once the time interval is reduced to 4 ms. This overhead boost also implies the synchronization contention occurs. In this figure, when contention is present, the EMonitor’s average lock overhead is about 15% less than the Java monitor’s lock overhead. In contrast, if no contention happens, the Java monitor’s lock overhead will become smaller than the EMonitor’s. This result is reasonable since the EMonitor itself is implemented by the Java monitor. In Figure 2(b), the short time interval causes contention always to occur. Consequently, the EMonitor’s average lock overhead is about 19.4% less than the Java monitor’s. From both Figure 2(a) and Figure 2(b), we can find that once contention is present, in most circumstances, the EMonitor’s lock overhead is smaller than the Java monitor’s. The reason is stated as follows: When a thread releases the lock it holds, the Java monitor’s reader/writer lock will invoke notifyAll( ) to wake up all pending reader and pending writer threads. Then, each awaked thread will enter the monitor, and determine whether it can get the desired lock or not. Consequently, extra thread context switches are introduced, and the Java monitor’s average lock overhead is increased. On the other hand, the EMonitor can avoid this problem by providing more precise event notification.
5 Conclusion In this paper, we first analyze the five known drawbacks of the Java’s monitor in detail. Then, we proposed our solution – the EMonitor. It is still monitor-based, and can solve all Java monitor’s problems we have identified. The EMonior offers better expressive power. Besides, when contention is present, the concurrent programs implemented by the EMonitor are more efficient than the ones implemented by the Java monitor. The experimental result in section 4 confirms our idea. Hence, we suggest replacing the Java monitor with the EMonitor when developing elaborate concurrent objects or high-contention concurrent systems.
1060
Hsin-Ta Chiao, Chi-Houng Wu, and Shyan-Ming Yuan
References 1. G. Andrews, Concurrent Programming - Principles and Practice, The Benjamin/Cummings Publishing Company, Inc., 1991. 2. P. Buhr, M. Fortier, and M. Coffin, “Monitor Classification,” ACM Computing Surveys, vol. 27, no. 1, pp. 63-107, March 1995. 3. D. Bacon, R. Konura, and C. Murthy, et al, “Thin Locks: Featherweight Synchronization for Java,” Proc. ACM SIGPLAN ’98 Conf. on Programming Language Design and Implementation, pp. 258-268, 1998. 4. B. Brosgol, “A Comparison of the Concurrency Features of Ada 95 and Java,” Proc. of ACM SIGAda Annual Int’l Conf. on Ada Technology, pp. 175-192, 1998. 5. J. Gosling, B. Joy, and G. Steele, The Java Language Specification, Addison-Wesley, 1996. 6. Y. Gu, B. Lee, and W. Cai, “Evaluation of Java Thread Performance on Two Differenct Multithreaded Kernels,” Operating Systems Review, vol. 33, no. 1, pp. 34-46, Jan. 1999. 7. N. Gehani, “Capsules: A Shared Memory Access Mechanism for Concurrent C/C++,” IEEE Transactions on Parallel and Distributed Systems, vol. 4, no. 7, pp. 795-811, July 1993. 8. C. Hoare, “Monitor: An Operating System Constructing Concept,” CACM, vol. 17, no. 10, pp. 549-557, 1974. 9. L. Kotulski, “About the Semantic Nested Monitor Calls,” SIGPLAN Notices, vol. 22, no. 4, pp. 80-82, 1987. 10. D. Lea, Concurrent Programming in Java – Design Principles and Patterns, AddisonWesley, 1997. 11. 11. R. Olsson and C. McNamee, “Experience Using the C Preprocessor to Implement CCR, Monitor, and CSP Preprocessors for SR,” Software - Practice and Experience, vol. 26, no. 2, pp.125-134. Feb. 1996. 12. S. Oaks and H. Wong, Java Threads, O‘Reilly & Associates, Inc., 1997. 13. S. Stubbs, D. Carver, and A. Hoppe, “IPCC++: A Concurrent C++ Based on a SharedMemory Model,” Journal of Object-Oriented Programming, vol. 8, no.2, pp. 45-50, 66, May 1995. 14. S. Yuan and Y. Hsu, “Design and Implementation of a Distributed Monitor Facility,” Computer Systems Science and Engineering, vol. 12, no. 1, pp. 43-51, Jan. 1997. 15. C. Varela and G. Agha, “What after Java? From Objects to Actors,” Computer Networks and ISDN Systems, vol. 30, no. 1-7, pp.573-577, 1998.
An Object-Oriented Software Framework for Large-Scale Networked Virtual Environments Fr´ed´eric Dang Tran and Anne G´erodolle France Telecom R&D DTL/ASR, 38-40 rue du G´en´eral Leclerc, 92794 Issy Moulineaux Cedex 9, FRANCE {frederic.dangtran, anne.gerodolle}@francetelecom.fr
Abstract. Continuum is an object-oriented software framework that aims to offer an open and extensible foundation for building large-scale real-time networked virtual environment applications. This platform relies on a partial replication model in which the whole simulation space is spread on a federation of processes based on application-specific criteria. A configurable event communication framework allows arbitrary consistency and synchronization protocols to be implemented in close cooperation with the application semantics. Continuum has been experimented with success for multi-player game prototypes involving both human-driven and autonomous simulated entities.
1
Introduction
With the tremendous success of the Internet, multi-participant real-time distributed simulations based on a shared ”virtual world” paradigm are bound to develop at a fast pace: multi-user online games, virtual shopping malls, concurrent design applications etc. These applications attempt to provide to end-users the illusion of being immersed in a conceptually unique shared virtual environment where they can (usually by way of their so-called avatar) interact in real-time with other users or computer-controlled objects. Large-scale shared virtual world applications raise many challenges. Foremost among them is the necessity to provide a globally consistent virtual environment to end-users. Problems typical of distributed systems such as synchronization, consistency and availability are made all the more difficult by the real-time and scalability requirements of large-scale human-in-the-loop simulations. A user may initiate an event, such as modifying an object in the 3D scene, and expects the effect of that action to be visible within a bounded time. The Continuum platform presented in this paper aims to offer an open and extensible object-oriented architecture which meets the following requirements: – support for shared spaces involving several hundreds or thousands of simultaneous participants spread over a vast geographic area; – support for dynamic and persistent virtual environments which are sustained and ”alive” even when all human participants have left; A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1061–1070, 2000. c Springer-Verlag Berlin Heidelberg 2000
1062
Fr´ed´eric Dang Tran and Anne G´erodolle
– support for highly interactive shared spaces populated by both user-controlled and autonomous objects interacting with one another in real-time; Instead of proposing a one-size-fits-all platform, the Continuum platform is based on an open software framework from which several “profiles” can be derived (e.g. for collaborative applications, for real-time online games etc.), each addressing a specific application domain. Our design approach is to some extent similar to Application Level Framing approach [3] proposed for building communication protocols. Our platform enables the application programmer to integrate closely the semantics of the application with the mechanisms of the platform. The rest of this paper is outlined as follows. Section 2 describes the object and perception model which lies as the foundation of our architecture. The next section explains the replication model and its implication on the scalability of the platform. Section 4 describes the event model of the platform. The platform architecture is outlined in section 5. Related work is summarized in section 6. Finally we conclude and point to future directions. The Continuum platform is strongly Java technology centered and assumes that the Java programming language is used both at the application and system level. All code fragments provided in this paper are in Java.
2
Object and Perception Model
A shared information space consists of a collection of objects. Each of these object encapsulates a state and a behavior. The behavior of an object determines how the object will change either spontaneously or as the result of some interaction with its environment. We distinguish: – Passive objects whose behavior is degenerate and which are under the concurrent control of active objects. Examples of passive objects include all “inert” objects making up the scenery of a virtual environment: chairs, trees etc. – Active objects endowed with a behavior and which can evolve autonomously within the simulated world. Computer-controlled opponents in games (“bots”) are typical examples of active objects. Note that an active object is not necessarily a “material” visible object. Force fields (gravity, wind etc.) are examples of such objects. However every simulation object has a well-defined spatial extent. Interactions between simulation objects make the simulation progress over time. In the context of synthetic environments, we define a perception model which quantifies and constraints inter-object interactions. Each simulation object is endowed with an “aura” that describes how far an object can perceive its environment and how far it can influence its environment:
An Object-Oriented Software Framework for Large-Scale NVEs
1063
class Aura { public Bounds getInfluenceZone(); public Bounds getPerceptionZone(); } A simulation object is endowed with a collection of auras, each dedicated to a specific medium (visual, aural senses etc.).
3
Replication and Persistence Model
The major challenge faced by our platform is to be able to handle large-scale simulation spaces populated with several thousands of simulated objects and accessible in real-time. Centralized architectures which rely on a single server for hosting the simulation space suffer from obvious scalability problems, the server being a natural bottleneck for communication and local processing. The Continuum platform is based on a completely distributed architecture in which a collection of simulations cooperate with one another in order to simulate the conceptually unique simulation space. We distinguish: – autonomous simulations without the control of human participants. Note that the existence of autonomous simulations guarantees the persistency of (part of) the simulation space, – human-driven simulations that include functionality necessary to allow human participants to take part into the simulation: graphical rendering, I/O interfaces. It is worth noting that the core of the Continuum platform is not dependent on any graphical/audio rendering APIs and focus only on distributed systems issues. The task of simulating the large population of objects is spread among these simulations. A given logical simulation object is broken up into a certain (and dynamic) number of replicas. The SimObject abstraction is the root class of all simulation objects that can be shared as part of a virtual environment: abstract class SimObject implements EventListener , Serializable { public Aura getAura(); ... } A SimObject represents both: – a reference to a conceptually unique simulation object which is fragmented over several simulations. In other words, from the outside, one can interact with a SimObject instance without being aware of the fact that this instance is a replica. – an individual replica. Internally a SimObject “knows” that it is a replica that needs to synchronize with its peers.
1064
Fr´ed´eric Dang Tran and Anne G´erodolle
A SimObject must provide the way to create replicas of itself. Java serialization is used for this purpose. Any replica must be ready to be passed by value to another simulation in order to create a new replica of the simulation object it represents. Note that, except for immutable objects, this mechanism, by itself, does not solve all consistency issues raised by the replication architecture and needs to be complemented by an object-specific consistency protocol. At one given instant, a logical simulation object will be replicated on N simulations. Within the group of N replicas that represent this object, we distinguish a master replica, for which the platform guarantees the following properties: – The master replica is initially located within the simulation which created and added the object to the virtual environment (i.e. the birthplace of the object) – a simulation object exists as long as its master replica exists – at any moment there is one and only one master replica for any simulation object – the mastership of an object can be transferred from one simulation to another – the shutdown of a simulation entails the destruction of all simulation objects for which it holds a master copy This master-slave distinction is quite similar to the notion of proxy in clientserver architectures. A slave object acts as a local representative of the master object. How master and slave replicas communicate with one another is covered in the next section. We assume, although this can be qualified to a large extent, that the “intelligence” of the object is located within its master copy whereas the slave replicas are “dumb” and just reflect the state of the master object. A given simulation hosts a certain number of master replicas (typically those which has been created locally) as well as slave copies of other simulation objects. Instead of replicating the whole simulation space on each simulation, the perception capabilities of objects (as quantified by their auras) are used as criteria to determine the actual subset of the object population which needs to be present locally: Only objects whose zone of influence intersects the zone of perception of at least one master object need to be replicated locally. The SimSpace abstraction provides access to the simulation space. It is a container of SimObjects and allows a client application to add or remove objects to/from the shared environment and to be notified of changes in its population: interface SimSpace { void addObject(SimObject obj); void removeObject(SimObject obj); void addSimSpaceListener( SimSpaceListener listener); void removeSimSpaceListener( SimSpaceListener listener); void makePersistent(SimObject obj); }
An Object-Oriented Software Framework for Large-Scale NVEs
1065
interface SimSpaceListener { void handleAddObject(SimObject obj); void handleRemoveObject(SimObject obj); }
4
Event Model and Synchronization
Replicas which collectively make up a logical simulation object need to keep themselves synchronized in order to provide a consistent local simulation space as hosted by each simulation and, as possibly viewed and manipulated by human users. A SimObject instance interacts with its peers by exchanging events over a many-to-many event channel which is provided by the Continuum platform on a per-object basis: abstract class SimObject implements EventListener , Serializable { protected PeerChannel getPeerChannel(); ... } The sending of events is performed via the PeerChannel interface which abstracts this event channel linking replicas to one another (see figure 1): public interface PeerChannel { public void sendToAll(Object event, Object hints); public void sendToMaster(Object event, Object hints); } This interface allows one SimObject to emit an event to all other replicas (sendToAll) or only to the master replica (sendToMaster). The event argument is any serializable Java object. It is expected that the PeerChannel will offer different sendEvent semantics. The stronger the semantics, the less work has to been carried out by the SimObject proper. The hints parameter is currently a placeholder for providing specific requirement on a per-event basis. We expect the following semantics to be offered (the first two have been implemented so far): – best-effort, no delivery guarantee, no sequenced delivery guarantee – ordered and reliable delivery with respect to the source – causal delivery for all events emitted within the interaction group the sender is part of. Each SimObject is a listener for events coming from its peers: interface EventListener { public void eventReceived(Event event); }
1066
Fr´ed´eric Dang Tran and Anne G´erodolle
EventListener
SimObject
Master Object
PeerChannel
SendToMaster
SendToAll
Fig. 1. Inter-replica event communication
Specializations of the SimObject class need to implement the eventReceived method following a chain of responsibility design pattern [1] along the inheritance tree. In other words, if a class cannot handle an incoming event, it should pass it on to its parent class and so on all the way up to the SimObject root class. An Event represents a change in the state of the object. The Event abstraction is defined as follows: interface Event { long getRealTimeStamp(); Object getCausalTimeStamp(); Object getArgument(); } Each Event is associated with the wall clock physical time of its emission (getRealTimeStamp) and a causal timestamp (getCausalTimeStamp) which is currently a placeholder for providing access to the sequence of events causally related to this one (provided that the platform is extended with consistency protocols meeting both real-time and causality ordering constraints between objects). Consistency Policy Examples Various types of consistency policies have been implemented on top of the event primitives covered above: – for highly dynamic mobile objects, the master replica “pushes” events containing the kinematic state of the object on a continuous basis. In this case, the loss of one event is not critical since it will be rapidly superseded by later events in the flow. The bandwidth requirement of such event flows is
An Object-Oriented Software Framework for Large-Scale NVEs
1067
reduced by employing dead-reckoning techniques. Dead-reckoning is a way to hide network latencies by predicting the kinematic state of an object based on an extrapolation algorithm. Thus the master object will emit an event only if it detects that its real position differs from the approximated position above a certain threshold. – some attributes of a simulation are static and will never change over the life of the object (e.g. the graphical description of “visible” objects). Such data are usually requested on demand by a slave replica using a “pull” approach. – concurrency control issued for passive objects are currently handled using a “centralized” approach. All replicas forward the request to alter an object (say to change its color) to the master replica which takes a decision according to a FIFO policy and announces the change back to its replicas.
5
Platform Architecture
application
libraries of reusable objects
object management
object replication & life-cycle
event management
consistency policies event ordering, synchronization
Aura Management
event filtering, communication channel mux
group communication protocols
reliability, group management (PGM...)
network layer
Multicast IP, UDP/IP, TCP/IP
Fig. 2. Continuum platform architecture
The Continuum middleware architecture is depicted in figure 2. The layered architecture reflects a reasonable separation between mostly orthogonal tasks within the system. Worthy of note is the fact that each layer makes a clear separation between policy and mechanisms and can be extended and adapted to fit the needs of the target application. – The network layer allows the integration of arbitrary network protocols. Of particular interest are multicast protocols well adapted to the communication patterns of shared virtual worlds applications. – On top of the basic network layer, group communication services are offered which provide various trade-off between reliability and timeliness.
1068
Fr´ed´eric Dang Tran and Anne G´erodolle
– The Aura Management layer is responsible for filtering the flow of events in accordance with the partial replication model described previously, which relies on object auras. Various types of Aura management implementations are possible. The current implementation relies on a combination of networklevel filtering (through a virtual world partition and the allocation of separate IP multicast addresses for each region) and local filtering. – The Event Management layer provides various levels of consistency and involves tasks such as ordering, timestamping or buffering events. – The object management layer covers the correct handling of replicated objects (bookkeeping, lifecycle, garbage collection). – The application layer consists of simulation objects containing specific information such as graphical models, behavior, rules of interaction between entities (e.g. physical simulation ). The Continuum platform expects to offer along with the software framework a toolkit of reusable components from which an application programmer can construct complex objects. The Continuum platform relies on the open distributed object platform Jonathan [4]. Jonathan offers both a CORBA-compliant and a RMI-like programming personalities. The client-server functionality of the RMI personality has been used for most point-to-point communications in Continuum (initial access and connection to a world server etc.). In the context of Continuum, this ORB has been extended with many-to-many event propagation facilities relying on IP multicast.
6
Related Work
Several platforms for large-scale shared virtual environments have been developed in the past years [6,10,9,2]. These systems are based on a shared database approach and focus on the replication of passive objects. They assume that the behavior associated with objects is handled outside the scope of the replication platform proper. Among them, very few can qualify as open middleware platforms. They usually make fixed design choices in terms of communication architecture, consistency protocols etc. For example, using a reliable multicast protocol for event dissemination on a systematic basis might not be appropriate for all types of events. As a result, their extensibility is limited and they are biased toward certain types of applications at the expense of others. A lot of work has been done in the areas of replication and cache consistency. Group communication systems relying on strong consistency criteria [13], standard distributed database or shared memory approaches [11] are ill-suited for real-time interaction and will scale only if one accepts a performance penalty or a significant delay. Resorting to partial replication in order to reduce the degree of sharing is an approach adopted by many systems in the field of collaborative work. The notion of aura is introduced in [7] as a spatial notion to support interaction models. This notion is further refined in [8] to quantify the mutual awareness between two objects. Other approaches [14,9] rely on a static partitioning of the shared information space into a set of disjoint interaction spaces.
An Object-Oriented Software Framework for Large-Scale NVEs
1069
The design of real-time consistency criteria and protocols is still an open research area. Resorting to application-level prediction techniques such as dead-reckoning is an approach adopted by many virtual environment platforms [12,10].
7
Conclusion
Distributed multi-user applications on the Internet (and shared 3D worlds in particular) require an underlying infrastructure which takes into account their specific needs and which can adapt to their specific semantics. By designing the Continuum middleware as an open software framework with a clear-cut separation between policies and mechanism, we offer an adaptable and flexible platform which both (i) insulates the application programmer from low-level aspects and (ii) enables the system programmer to introduce alternate infrastructure components (network protocols, consistency protocols etc.). The use of the Java programming language, beyond its cross-platform portability, significantly contributes to the flexibility of the platform with the ability to dynamically download and run code. Several test applications have been developed and experimented over a WAN using IP multicast. In particular an underwater world simulation involving several hundred of autonomous fishes has been developed. The computation of the behavior of the fishes is spread across several simulations. Thanks to aura-directed event filtering, a given user only “sees” a small subset of the simulation space at a given time, which significantly decreases the bandwidth requirement of the network link connecting his client simulation. We are currently assessing the performance of our platform in more detail. We intend to extend it toward the following directions: – integration of our platform with the Java-based reactive programming framework described in [5]. This approach avoids the use of Java threads and the not-clear semantics and non deterministic scheduling associated with them. – real-time consistency protocols based on consistency criteria which take into account both real-time and causality relationships between events. – scalability issues need to be investigated further regarding, e.g., placement, caching and load-balancing issues.
References 1. Gamma Erich, Helm Richard, Johnson Ralph, Vlissides John: “Design Patterns, Elements of Reusable Object-Oriented Software”, Addision-Wesley, published October 1994. 2. High Level Architecture Web Site: http://hla.dmso.mil/ 3. Floyd S., Jacobson V., Liu C., McCanne S., Zhang L. “A Reliable Multicast Framework for Light-weight Sessions and Application Level Framing”, Proc. ACM SIGCOMM’95, 1995. 4. Dang Tran F., Dumant B. , Horn F. , Stefani J.B., “Jonathan: an open distributed processing environment in Java”, Middleware’98, IFIP International Conference on distributed Systems Platforms and Open Distributed Processing, Lake District, UK, September 1998. Jonathan Web site: http://www.objectweb.org
1070
Fr´ed´eric Dang Tran and Anne G´erodolle
5. Hazard L., Susini J.F., Boussinot F., “Distributed Reactive Machines”, RTCSA’98, Hiroshima, October 98. 6. Hagsand O., “Interactive Multiuser VEs in the DIVE System”, IEEE Multimedia, Vol.3, Number 1, 1996. 7. Benford S., Fahlen L., Greenhalge C., Bowers J., “Managing mutual awareness in collaborative virtual environments”, Proceedings of the ACM SIGCHI conference on Virtual Reality and Technology, August 1994, Singapore, ACM press. 8. Greenhalgh C., Benford S., “MASSIVE: a Distributed Virtual Reality System Incorporating Spatial Trading”, Proceedings of the 15th International Conference on Distributed Computing Systems, Vancouver, Canada, May 30 - June 2, 1995, IEEE Computer Society. 9. Anderson D., Barrus J., Howard J., Rich C., Shen C., Waters R., “Building MultiUser Interactive Multimedia Environments in MERL”, IEEE Multimedia, Volume 2, Number 4, 1995. 10. Macedonia M., Zyda M., Pratt D., Barham P., Zeswitz S., “NPSNET: A Network Software Architecture for Large Scale Virtual Environments”, Presence, Volume 3, Number 4, 1990. 11. Li K., Hudak P., “Memory Coherence in shared virtual memory systems”, ACM Transactions on Computer Systems 7(4), 1989. 12. Roberts D., Sharkey P., “Maximizing concurrency and scalability in a consistent, causal, distributed virtual reality system, whilst minimizing the effect of network delays”, IEEE Sixth Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, MIT, June 1997. 13. Birman K., Schiper A., Stephenson P., ”Lightweight causal and atomic group multicast”, ACM Trans. on Computer Systems, Vol. 9 no 3, 1991. 14. Kindberg T., Coulouris G., Dollimore J., Heikkinen Jyrki, “ Sharing Objects over the Internet: the Mushroom Approach”, Proceedings of Global Internet 96, IEEE, London, November 1996.
TACO — Dynamic Distributed Collections with Templates and Topologies J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa Real World Computing Partnership Tsukuba Mitsui Bldg. 16F, 1-6-1 Takezono Tsukuba 305-0032, Ibaraki, Japan {jon,msato,ishikawa}@trc.rwcp.or.jp
Abstract. High-level data-parallel programming with distributed object sets eases many aspects of parallel programming. In this paper we describe the design and implementation of Taco, a template library that provides higher-order operations on polymorphic regular and irregular distributed object sets by means of reusable topology classes and C++ function templates. Keywords: dynamic collections, distributed data-parallel processing, global communication
1
Introduction
Collective operations are a powerful means to implement globally coordinated operations on distributed data sets. Despite this fact the acceptance of dataparallel programming languages is still limited. However, while efficient finegrained data-parallel computing is hard to achieve without proper compiler support and appropriate optimization techniques, efficient coarse-grained as well as medium-grained data-parallel computing is well possible on a library basis only. Thus we can introduce the flavour of data-parallel programming without language extensions and data-parallel programming can be conveniently combined with common parallel programming techniques where appropriate. In addition, the inheritance mechanisms of object oriented languages allow us to control, extend and adapt the behaviour of distributed collections in a well structured manner. Taco is therefore a pure template library that extends our basic Multiple Threads Template Library (MTTL) [8] with Topologies and Collections. Topology classes are used to describe the relation between members of distributed object groups and template classes as well as function templates are applied to address such data sets collectively.
2
The Multiple Threads Template Library
The MTTL is a very efficient communication and threading library on top of either the PM [9] or MPI communication layer. Amongst several other features A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1071–1080, 2000. c Springer-Verlag Berlin Heidelberg 2000
1072
J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa
the MTTL provides a concept for global pointers, multi threaded remote method invocation and global synchronization variables. Global pointer templates store both the network address as well as the local address of an object, such that the object can be read and written remotely using overloaded operators. Objects can be created remotely resulting in a global pointer that can also be applied for remote method invocation. Remote method invocation is implemented by means of function templates that take care of argument marshaling, communication and method invocation. A method can either be invoked synchronously or asynchronously. In case of asynchronous invocations, global pointer based synchronization variables can be used for lazy synchronization. A synchronization variable is a kind of future [7], that blocks any reading access until the variable is (remotely) initialized. These few prerequisites were already sufficient to implement distributed collections and efficient higher-order operations. 2.1
Global Object Pointers and Remote Method Invocation
The original implementation of the MTTL did not support polymorphism. In the MTTL a global pointer to a derived class cannot be assigned or safely converted to a pointer to a base class like in standard C++. Likewise a method belonging to a base class cannot be invoked using a global pointer to a derived class. Therefore we first extended the MTTL with a very lean remote method invocation layer (RMI) that implements fully polymorphic global pointers and remote method invocation mechanisms. We designed an ObjectPtr class for the MTTL that implements statically type safe compatibility between global object pointers and remote method invocation according to C++ type rules. class Base { public: virtual void f(int n); }; class Derived : public Base { public: Derived(int i); void f(int n); }; // allocate new object at node 17 and pass 77 as argument to // the constructor ObjectPtr
// OK, statically checked. // remote Derived::f(77) is called.
Note that &Base::f does not evaluate to the machine address of a method as one might expect but to an index into the virtual table of class Base because
TACO — Dynamic Distributed Collections with Templates and Topologies
1073
f() is declared to be a virtual method in Base. The various invocation methods of the ObjectPtr class take a pointer to the member function to be invoked as well as an arbitrary number of arguments being passed to the method. All methods are implemented as various overloaded member function templates being distinguished automatically by the compiler by means of the actual type of the passed arguments. Parameter passing semantics is strictly call-by-value and global object pointers can be passed as well. ObjectPtr
Remote objects are created by means of the allocate() template at the specified node, the specified arguments are passed to the constructor of Class. The call methods are synchronous, while the apply() methods are asynchronous. The most simple variant of apply() is only applicable to methods returning void while the second version of apply() allows for lazy synchronization of the result of the method. This is particularly useful for the implementation of global reductions. The implementation of class ObjectPtr is based on the MTTL’s monomorphic GlobalPtr class. The compatibility of global object pointers is defined by a conversion constructor: template
Any time an ObjectPtr instance is used in a context that expects a different type of global object pointer this conversion constructor is called automatically to perform the type coercion. The global object pointer is effectively split into a local pointer component and node address which are now both assigned separately. Thus a type coercion on the local pointer component is automatically performed according to C++ type rules. The coercion fails when the types of
1074
J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa
the local pointer components are not compatible to each other and these errors are detected at compile time. For performance reasons we also provide an overloaded assignment operator, that basically performs the same operations as the conversion constructor. This operator helps to avoid the creation of temporary objects and additional copies in assignments. The ObjectPtr class is now powerful enough to support various kinds of dynamic as well as polymorphic distributed data-structures.
3
Collections and Topologies
In Taco the structure of a collection is defined by a topology class. In principle, a collection can have any user defined topology such as n-dimensional grids, lists or trees (fig. 1).
Fig. 1. Topologies
Topologies are implemented as distributed linked object sets by means of global pointers to objects (refer to section 2) These sets can be collectively addressed by issuing a method to a leader object that represents the entire collection. All objects that are reachable from this leader object automatically participate in collective operations. Note that multiple objects of a collection might be mapped to the same computing node. Taco will apply local invocation mechanisms in this case. Thus the size of collections is not limited by the number of available nodes. 3.1
Collective Methods
We implemented only those collective operations that are fundamental to most collective operation patterns. An asynchronous global map() operation is provided to initiate data parallel computations and invoke a void method on all members of the collection. The asynchronous map() operation is complemented with a synchronous reduce() operation that executes a method on all members of the collection in parallel and applies an associative and commutative binary
TACO — Dynamic Distributed Collections with Templates and Topologies
1075
function to combine all results. Collective methods are again implemented as member function templates of the ObjectPtr class. ObjectPtr
All collective methods assume that the members of a collection define an STL style iterator [12] that (recursively) iterates all members of a collection. Members are identified uniquely by means of global object pointers (see section 2) and thus the iterators need to evaluate to global object pointers. Taco’s topology classes define such iterators and thus application classes only need to be derived from a topology class to inherit the iterators as well. The iterator determines the order in which collective methods are passed to members of the collection. Each collective method is first propagated asynchronously to the other members of the collection in the order defined by the iterator before it is called on the local object instance. In case of reductions we use the asynchronous apply() method in conjunction with lazy synchronization to synchronize on the availability of results. All results are collected in exactly the same order in which the invocation method has been spread. Thus, the behaviour remains deterministic and application programmers can get control over collective operations either through the iterator method of the topology class or by means of custom iterators defined by application classes. 3.2
Creation of Collections
Collections with a known initial size can be created and addressed by means of a GroupOf template class as follows: class Sheep: public BinTree<Sheep> { public: int weight(); // return sheep’s weight ... }; Cyclic map(8); // cyclic mapping to 8 nodes GroupOf<Sheep> flock(128, map); // 128 members // a reduction (global sum) int add(int a, int b) { return a+b; } int sum = flock.reduce(&add, &Sheep::weight);
In this example a collection of 128 Sheep is created and mapped to 8 processing nodes using a Cyclic mapping strategy. Block cyclic as well as random strategies are also available. Mapping classes provide a subscript operator that delivers for a given rank in a collection a corresponding node number. The mapping argument of the GroupOf class is generic and thus even an array of node numbers can be specified to control mapping explicitly. Class Sheep can be any arbitrary C++ class, while BinTree is a topology class that describes a distributed binary tree
1076
J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa
of objects of any kind. After the construction we can sum up the weight of the whole flock by means of a global reduction that applies an add() function to reduce the results. 3.3
Design Considerations
In the previous example class Sheep is aware that its instances will become members of a collection being organized as a binary tree. Being aware of collections and topologies is beneficial to those classes that need to determine topology related data at runtime, such as the rank of objects within the topology, the addresses of neighbouring objects in case of grid topologies and similar data. This approach is intrusive and thus this collection concept cannot directly be applied to arbitrary classes. However, collection aware classes can easily be separated from other classes by means of inheritance. Therefore a Sheep class that is aware of being in a flock can easily be described be means of multiple inheritance: class FlockMember: public Sheep, public BinTree
The FlockMember class inherits all abilities from the Sheep class and additionally knows about its group relationship. The methods of FlockMember can now easily deal with conditional collective operations without any interference with the basic Sheep class. We automated this approach and provide additionally a non-intrusive CollectionOf template that does not require its members to be derived from a topology class. The topology is specified to the CollectionOf template instead and members are wrapped into containers that are derived from the specified topology class. CollectionOf<Sheep,BinTree> flock(...);
Note that collective operations are applied in the same manner to both intrusive and non-intrusive collections, only the declarations of the collections as such differ. Furthermore, constructors of collections might have an arbitrary number of arguments that are passed to the members for initialization. When the scope of a collection object becomes invalid (or the collection object is deleted in case it has been allocated on the heap) all members of the distributed collection will also automatically be destroyed. 3.4
Dynamic Collections
The implementation of dynamic collections is a straightforward task since Taco’s collections are inherently based on distributed dynamic data structures that can be constructed in the same way as any corresponding local data structure. Therefore the implementation of highly dynamic schemes is often possible with a few lines of code only.
TACO — Dynamic Distributed Collections with Templates and Topologies
1077
Fig. 2. An adaptive Grid
All Taco topology classes provide a join() method that allows to add either new members to the collection or to join existing collections. To illustrate this further, consider a class Quadrant that adaptively refines a mesh or grid as depicted in figure 2 based on some runtime condition. The following short code fragment then describes the dynamic construction of the corresponding collection: class Quadrant: public QuadTree
The QuadTree class is a 4-ary tree topology. Depending on the state of the object and/or some specific runtime condition the refine() method may create new members and join them to the tree. The allocate() template is applied to create a new remote Quadrant instance according to a random placement strategy and returns a global object pointer to the newly created object.
1078
J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa
When refine() is now called as a collective operation the members of the collection that meet the condition for refinement will create new members in parallel and join them to the collection dynamically. // allocate first group member ObjectPtr
By means of such schemes highly irregular collections as shown in figure 2 can be created. Since we also retain full polymorphism, these dynamic collections can be heterogenous as well.
4
Performance
We measured the performance of our collective operations on the RWC PC Cluster II consisting of 128 Pentium Pro nodes (200MHz CPU) interconnected by a 160MByte/sec Myrinet network running the Score [9] system software on top of a Linux System. All benchmarks have been run 100 times in a loop to
250
par. red-2 mpi-bc-red par. red-4
Time (microsec)
200 150 100 50 0
0
50
100 150 Number of Objects (128PE)
200
Fig. 3. reduce() vs. MPICH-PM/CLUMP
ensure, that the benchmark code was mostly in the cache and that the benchmark times were long enough to get a meaningful measurement using the standard
TACO — Dynamic Distributed Collections with Templates and Topologies
1079
gettimeofday() system call. The resulting times have been divided then by the number of runs. For comparison with a standard communication library we measured the computation of global sums with our reductions against collective operations of MPI (fig. 3) using both a binary tree (par. red-2) as well as a 4-ary tree (par. red4) to study the impact of different tree topologies (which are both not optimal for software multicasts [6]). A MPI bcast() followed by a MPI Reduce() (mpibc-red) is closest to our reduce() semantics. The MPI implementation we used is MPICH-PM/CLUMP [13] which is a very efficient port of MPICH for Myrinet networks. Figure 3 shows that even our very first implementation is by all means in the competitive range. In case of the 4-ary tree topology our implementation shows even better performance than the corresponding MPI implementation.
5
Conclusion
Collections and collective operations are neither new nor did we invent the notion of topologies as such. In fact our work has been strongly inspired by the collection concept of pC++ [3] and ICC++ [5], the communities in Ocore [10], groups in HPC++ [1] and – last but not least – the topology concept of Promoter [2]. However, in most data-parallel languages as well as commonly used communication libraries like MPI the notion of a group is usually an abstract concept and the programmer has to rely on an efficient implementation of this concept without any reasonable means of control. Since it is notoriously hard to find suitable implementations of collections that perform well on all use cases and all network architectures, Taco gives the programmer a powerful means of control over collections through reusable topology classes, that reveal the internal structure of a collection. This is of utmost importance with regard to performance tuning. Furthermore, most data-parallel approaches traditionally concentrate strongly on regular array structures like the Amelia Vector Template Library [11]. Amelia is a single-paradigm approach that solely relies on data parallelism similar to a vector machine. The only supported data structure is a distributed vector of potentially very fine-grained elementary data types like doubles or integers. Because of this fine-grained library approach Amelia needs to provide a rich set of specific algorithms on vectors to achieve acceptable performance. Compared to Amelia, Taco is a fairly general purpose library for medium grained parallelism. Taco focuses on a simple, yet powerful and easy to understand group concept that allows programmers to construct more complex application specific libraries with modest effort. We deliberately based our collections on flexible graphs similar to Arts [4]. Therefore existing collections can easily be changed dynamically at run time and the implementation of adaptive schemes is straightforward. Due to Taco’s full support for polymorphism heterogenous collections are possible as well. Although fine grained parallel computing is also possible, we do not expect satisfactory performance results since we cannot apply compiler level optimiza-
1080
J¨ org Nolte, Mitsuhisa Sato, and Yutaka Ishikawa
tions like the language-based approaches. However, the performance for fine grained structures can be significantly improved when the members of Taco’s collections are themselves containers holding entire sets of small objects. Therefore Taco can serve well as a basis for domain specific libraries as well as runtime systems for fine-grained data-parallel processing.
References [1] Peter Beckman, Dennis Gannon, and Elizabeth Johnson. Portable Parallel Programming in HPC++. Technical report, Department of Computer Science, Indiana University , Bloomington, IN 47401. [2] M. Besch, H. Bi, P. Enskonatus, G. Heber, and M. Wilhelmi. High-Level Data Parallel Programming in PROMOTER. In Proc. Second International Workschop on High-level Parallel Programming Models and Supportive Environments HIPS’97, Geneva, Switzerland, April 1997. IEEE-CS Press. [3] Francois Bordin, Peter Beckman, Dennis Gannon, Srinivas Narayana, and Shelby X. Yang. Distributed pC++: Basic Ideas for an Object Parallel Language. Scientific Programming, 2(3), Fall 1993. [4] Lars B¨ uttner, J¨ org Nolte, and Wolfgang Schr¨ oder-Preikschat. Arts of Peace – A High-Performance Middleware Layer for Parallel and Distributed Computing. Journal of Parallel and Distributed Computing, 59(2):155–179, Nov 1999. [5] A. Chien, U.S. Reddy, J.Plevyak, and J. Dolby. ICC++ – A C++ Dialect for High Performance Parallel Computing. In Proceedings of the 2nd JSSST International Symposium on Object Technologies for Advanced Software, ISOTAS’96, Kanazawa, Japan, March 1996. Springer. [6] J. Cordsen, H. W. Pohl, and W. Schr¨ oder-Preikschat. Performance Considerations in Software Multicasts. In Proceedings of the 11th ACM International Conference on Supercomputing (ICS ’97), pages 213–220. ACM Inc., July 1997. [7] R. H. Jr. Halstead. Multilisp: A Language for Concurrent Symbolic Computation. ACM Transactions on Programming Languages and Systems, 7(4), 1985. [8] Y. Ishikawa. Multiple threads template library. Technical Report TR-96-012, Real World Computing Partnership, 1996. [9] Yutaka Ishikawa, Hiroshi Tezuka, Atsuhi Hori, Shinji Sumimoto, Toshiyuki Takahashi, Francis O’Carroll, and Hiroshi Harada. RWC PC Cluster II and SCore Cluster System Software – High Performance Linux Cluster. In Proceedings of the 5th Annual Linux Expo, pages 55 – 62, 1999. [10] H. Konaka, M. Maeda, Y. Ishikawa, T. Tomokiyo, and A. Hori. Community in Massively Parallel Object-based Language Ocore. In Proc. Intl. EUROSIM Conf. Massively Parallel Processing Applications and Development, pages 305– 312. Elsevier Science B.V., 1994. [11] Thomas J. Sheffler. The Amelia Vector Template Library. In Parallel Programming using C++, pages 43–90. MIT Press, 1996. [12] A. Stepanov and M. Lee. The Standard Template Library. Technical Report HPL-94-34, Hewlett Packard Laboratories, 1994 revised 1995. [13] Toshiyuki Takahashi, Francis O’Carroll, Hiroshi Tezuka, Atsushi Hori, Shinji Sumimoto, Hiroshi Harada, Yutaka Ishikawa, and Peter H. Beckman. Implementation and Evaluation of MPI on an SMP Cluster. In Parallel and Distributed Processing – IPPS/SPDP’99 Workshops, volume 1586 of Lecture Notes in Computer Science. Springer-Verlag, April 1999.
Object-Oriented Message-Passing with TPO++ Tobias Grundmann, Marcus Ritt, and Wolfgang Rosenstiel Wilhelm-Schickard-Institut f¨ ur Informatik, University of T¨ ubingen Department of Computer Engineering, Sand 13, 72076 T¨ ubingen {grundman,ritt,rosen}@informatik.uni-tuebingen.de
Abstract. Message-passing is a well known approach for parallelizing programs. The widely used standard MPI (Message passing interface) also defines C++ bindings. Nevertheless, there is a lack of integration of object-oriented concepts. In this paper, we describe our design of TPO++1 , an object-oriented message-passing library written in C++ on top of MPI. Its key features are easy transmission of objects, typesafety, MPI-conformity and integration of the C++ Standard Template Library.
1
Motivation and Design Goals
With MPI, a widely accepted standard for message-passing has been established. At the same time, object-oriented programming concepts gain increased acceptance in scientific computing (see, for example [1]). The MPI-2 standard [6, 7] defines C++ language bindings for MPI-1 and MPI-2, but these bindings are no significant improvement compared to the C bindings. The interface is not type-safe, does not simplify the MPI calls and defines no way for transmitting objects. Other approaches such as the mpi++ system [4, 5], OOMPI [8] and Para++ [2] improve certain aspects of message-passing in C++, but, besides some other topics, none of them integrates the STL (Standard Template Library), an important part of the current C++ standard [3]. Further they often don’t support user-defined types very well or there is a significant difference to MPI conventions, even if not necessary to introduce object-oriented concepts. In the design of TPO++, we try to address these problems: A major goal in providing a class library for C++ is a tight integration of objectoriented concepts, in particular the capability of transmitting objects in a simple, efficient, and type-safe manner. Also, an implementation in C++ should take into account all recent C++ features, like the usage of exceptions for error handling and the integration of the Standard Template Library by supporting the STL containers as well as adopting the STL interface conventions. The interface design should conform to the MPI conventions and semantics as closely as possible without violating object-oriented concepts. This helps migrating from C bindings and eases porting of existing C or C++ code. Further, the implementation should not differ much from MPI in terms of communication and memory 1
This project is funded by the DFG within SFB 382. T¨ ubingen Parallel Objects
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1081–1084, 2000. c Springer-Verlag Berlin Heidelberg 2000
1082
Tobias Grundmann, Marcus Ritt, and Wolfgang Rosenstiel
efficiency. Another goal is to guarantee thread-safety to provide maximum flexibility for application software and parallel runtime systems. This topic will not be further discussed here since thread-safety is optional in the MPI-Standard and depends mostly on the underlying MPI-Implementation.
2
Interface and Examples
The basic structure given in the C++ bindings of MPI is similar to our approach. All common MPI objects, i.e. communicators, groups, are implemented as separate classes. After initialization, the user can use the global Communicator object CommWorld. Transmission of predefined C++ types In the case of sending a C++ basic type, the send call reduces to: double d; CommWorld.send(d, dest_rank, message_tag); STL containers can be sent using the same overloaded communicator method. The STL conventions require two iterators specifying begin and end of a range, which also allows to send subranges of containers: vector<double> vd; CommWorld.send(vd.begin(), vd.end(), dest_rank, message_tag); The application can also use the blocking, synchronous and ready send modes defined in MPI by calling the communicator methods bsend, ssend and rsend, respectively. Asynchronous communication methods return an object of class Request which can be used for the application to test or wait for the completion. On the receiver side, a receive-call is done as follows (basic type): Status status; status=CommWorld.recv(d); Note that the status object, different from MPI, is a return parameter, because error handling is done via exceptions. This simplifies the receiver code, if no error checking is necessary and makes send and receive calls more symmetric. The receive methods take two optional arguments, the senders rank and a message tag for selecting particular messages. If omitted, they default to any sender and any tag, respectively. To receive a container, a single argument, specifying the insertion point is sufficient. Conforming to the STL, the data can be received into a container which is large enough by means of an iterator, or into any container by means of an inserter. The example shows both approaches: vector<double> vd1(x); // must provide enough space vector<double> vd2; CommWorld.recv(vd1.begin()); CommWorld.recv(tpo_back_insert_iterator(vd2)); //allocates space
Object-Oriented Message-Passing with TPO++
1083
Transmission of user-defined types To enable a class for transmission, the user has to declare its marshalling category. The library distinguishes user-defined objects having a trivial copy constructor and complex user-defined objects. Enabling a class with a trivial copy constructor for transmission reduces to the statement TPO TRIVIAL(User type). On transmission, the memory block occupied by such an object will be copied directly to the net. For transmitting complex objects (i.e. without a trivial copy constructor), the user has to define the marshalling methods serialize and deserialize, as part of the class definition. The presence of these methods must be signaled by a declaration of TPO MARSHALL(User type). They are supplied with a serializer object. In a serialize method, insert() is called repeatedly for every member to prepare the object for transmission. The serializer object does not copy the data, but records its memory layout for later transmission. Similarly, a received message can be unpacked to user-provided memory locations by calling extract() in the deserialize method. Usually, user-defined objects do not have to inherit from any special “message” class. Also note, that the code given in the marshalling methods can be reused in derived classes. For applications relying on virtual methods and generic interfaces an abstract base class Message is also provided.
3
Comparison with MPI
The tests have been done on Sun Ultra 5 machines (Solaris 2.7) using MPICH 1.1.2 and on a Cray T3E a Cray T3E (512 nodes at 450 MHz) using the native MPI implementation. We measured the efficiency of our library using a ping test and compared the achieved latencies and bandwidths of MPI and TPO. Bandwidth
Latency
CRAY T3E
CRAY T3E
2
2
1
10
0
10
10
0
10
10
1
10 10
0
10
2
10 Message size [B]
4
10
6
10
3
10
3
10
4
10
4
TPO Latency [s]
1
MPI Latency [s]
10
TPO Bandwidth [MB/s]
MPI Bandwidth [Mb/s]
10
1
10
0
10
2
10 Message size [B]
4
10
6
Fig. 1. Comparison of MPI (solid curves) and TPO (dashed curves) on a Cray T3E.
As shown in Figure 1 communication using TPO achieves the same bandwidth as MPI for messages larger than approximately 16 KB. The loss in bandwidth
1084
Tobias Grundmann, Marcus Ritt, and Wolfgang Rosenstiel
below this size is mainly due to the increased latency. Latencies of MPI and TPO converge as messages are getting larger. For small messages TPO shows a constant latency overhead of 15µs compared to MPI.
4
Conclusions
We have presented our implementation of an object-oriented message-passing system. It exploits object-oriented and generic programming concepts, allows the easy transmission of objects and makes use of advanced C++ techniques and features as well as supporting these features, most notably it supports STL datatypes. The system introduces object-oriented techniques and type-safety to message-passing while preserving MPI semantics and naming conventions as far as possible. This simplifies the transition from existing code. In contrast to other implementations the code to marshall an object can be reused in derived classes. Also, our library is able to handle arbitrary complex and dynamic datatypes. An object-oriented interface can be implemented with almost identical performance compared to MPI. The library is designed as a base for parallelizing scientific applications in an object-oriented environment.
References [1] F. Bassetti, K. Davis, and B. Mohr, editors. Proceedings of the Workshop on Parallel/High-Performance Object-Oriented Scientific Computing (POOSC’99), European Conference on Object-Oriented Programming (ECOOP’99), Technical Report FZJ-ZAM-IB-9906. Forschungszentrum J¨ ulich, Germany, June 1999. [2] O. Coulaud and E. Dillon. Para++: C++ bindings for message-passing libraries. Users guide. Technical report, INRIA, 1995. [3] International Standards Organization. Programming languages – C++. ISO/IEC publication 14882:1998, 1998. [4] D. Kafure and L. Huang. mpi++: A C++ language binding for MPI. In Proceedings MPI developers conference, Notre Dame, IN, June 1995. [5] D. Kafure and L. Huang. Collective communication and communicators in mpi++. Technical report, Department of Computer Science Virginia Tech, 1996. [6] Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. Technical Report CS-94-230, Computer Science Department, University of Tennessee, Knoxville, TN, May 1994. [7] Message Passing Interface Forum. MPI-2: Extensions to the Message-Passing Interface, July 1997. [8] J. M. Squyres, B. C. McCandless, and A. Lumsdaine. Object Oriented MPI: A Class Library for the Message Passing Interface. In Proceedings of the POOMA conference, 1996.
Topic 17 Architectures and Algorithms for Multimedia Applications Manfred Schimmler Local Chair
The emergence of multimedia technology in recent years is strongly driven by an enormous commercial potential. For the scientific community this development is interesting because a number of attractive disciplines for computer science and engineering flow together into the multimedia mainstream: image processing, computer graphics, data compression, encoding, cryptography, and broadband communication, to mention just a few of them. These fields have always been driving forces behind the design of massively parallel architectures and algorithms as well as special purpose processors and storage systems. The area provides an additional challenge due to the fact that the time interval from the scientific idea to the implementation becomes shorter and shorter. Take the following example as an indication of this statement on the one hand, and, on the other hand, as a proof of the high quality of this conference series: In the sessions [1, 2] corresponding to this one of Euro-Par’98, and Euro-Par’99 we have seen the presentation of ideas that have meanwhile found their way into successful commercial products. For this topic four papers in this tempting area have been selected: The first one gives a new approach to the design of array processors for DCT algorithms as used in MPEG-style encoders or decoders. The second paper proposes a massively parallel combination of SIMD array and ISA for volume rendering. An automated approach to design an architecture for image processing algorithms is discussed in the third paper. The last article of this topic provides a parallel storage architecture capable to deliver reliable video data while supporting a maximum number of video streams even in case of failure. In contrast to the earlier papers on multimedia focussing on special purpose architectures, one may notice that the systems presented in this topic are designed for flexibility in order to cope with future standards. It will be interesting to observe if this trend will hold on in the future.
References [1] [2]
Workshop 10+17+21+22: Theory and Algorithms for Parallel Computation. Proc. Euro-Par’98, LNCS 1470, Springer Verlag, pp. 863-966 (1998) Topic 12: Architectures and Algorithms for Vision and other Senses. Proc. EuroPar’99, LNCS 1685, Springer Verlag, pp. 939-1018 (1999)
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1085–1085, 2000. c Springer-Verlag Berlin Heidelberg 2000
Design of Multi-dimensional DCT Array Processors for Video Applications Shietung Peng1 and Stanislav Sedukhin2 1
2
Hosei University,Tokyo 184-8584, Japan, [email protected] The University of Aizu, Aizu-Wakamatsu City, Fukushima 965-8580, Japan [email protected]
Abstract. In this paper, we propose array processors for computing the 2-D and 3-D DCTs. We first introduce a new method, called dimensional splitting method, for the design of array processors for multi-dimensional image transforms. The method can be applied to any multi-dimensional image transforms with separable kernels such as DFT and DCT. Then, we propose a new coding scheme for the 1-D DCT in which the need for generating the kernel matrix in advance is eliminated. Finally, we show array processors for computing r-D DCT (r ≥ 2) which are scalable, regular, locally-connected, and fully-pipelined.
1
Introduction
Recent advances in various aspects of digital technology have made possible many applications of digital video such as HDTV, teleconference, and multimedia communications. These applications require high-speed transmission of vast amounts of video data. Most video standards use discrete cosine transform (DCT) as a standard transform coding scheme [4,7]. The DCT is very computationally intensive. To realize high-speed and cost-effective DCT for video coding, one needs a specially designated array processor so that the high throughout requirements can be matched. There has been considerable research in mapping efficient algorithms to practical and feasible VLSI implementations in the recent past [2,5,6]. However, these designs employed irregular butterfly structures with global communications resulting in complex layout, timing, and reliability concerns which severely limit the operating speed and expandability in VLSI implementation. In this paper, we propose new array processors for computing multi-dimensional DCT/IDCT based on a new decomposition method for image transforms. The main feature of this method is that no transposition of any intermediate matrix is needed during processing. Moreover, our method can be applied to any r-D (r ≥ 2) image transform. Using this method, the design of the array processors for the r-D DCT depends heavily on the design of corresponding linear array processors for the 1-D DCT. We develop effective DCT coding schemes for the 1-D DCT. Our schemes eliminates the need for generating the whole kernel A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1086–1094, 2000. c Springer-Verlag Berlin Heidelberg 2000
Design of Multi-dimensional DCT Array Processors for Video Applications
1087
matrix of the 1-D DCT. The coefficients are created locally on the fly whenever needed. This I/O effective recursive scheme is of particular interests when the real-time systems such as HDTV are considered [1,3].
2
A Dimensional Splitting Method
The 1-D image transforms can be expressed in the terms of the following relation T (u) =
N −1
f (x)g(x, u),
x=0
where 0 ≤ u ≤ N − 1, T (u) is the transform of f (x), and g(x, u) is the transformation kernel. The properties of the transformation kernel determine the nature of a transform. Similarly, the 2-D and the 3-D image transforms can be expressed as N −1 N −1 f (x, y)g(x, y, u, v), T (u, v) = x=0 y=0
and T (u, v, w) =
N −1 N −1 N −1
f (x, y, z)g(x, y, z, u, v, w),
x=0 y=0 z=0
where 0 ≤ u, v, w ≤ N − 1, g(x, y, u, v) and g(x, y, z, u, v, w) are the kernels for the 2-D and the 3-D transforms, respectively. A 2-D kernel is said to be separable if g(x, y, u, v) = g1 (x, u)g2 (y, v), and it is symmetric if g1 is functionally equal to g2 . The separability and symmetry of a 3-D kernel are defined similarly. Some samples of transforms with separable, symmetric kernels are discrete Fourier transform, discrete cosine transform, Walsh-Hadamard transform, and Haar transform. A 2-D (3-D) transform with separable kernel can be computed in two (three) steps, each requiring a 1-D transform. In the 2-D case, we can first take the 1-D transform along each row of f (x, y) to get z(u, y), and then, take the 1-D transform along each column of z(u, y) to get the result T (u, v). That is, T (u, v) =
N −1 N −1
(
y=0 x=0
f (x, y)g1 (x, u))g2 (y, v) =
N −1
z(u, y)g2 (y, v).
y=0
Similarly, in the 3-D case, the 1-D transforms are performed three times, each on different dimension, to get the result T (u, v, w). The idea of the proposed method to construct r-D array processors for computing the r-D image transform with separable kernel is based on the above approach. However, to realize this approach, several different I/O formats are enforced.
1088
Shietung Peng and Stanislav Sedukhin
1. In the 1-D transforms along the first dimension, the elements of the initial image vector are pipelined into the array processor and the elements of the intermediate transform vector should be kept inside the processing elements (PEs) of the array processor. This type of array processor is called type I array. 2. In the 1-D transforms along the last dimension, the intermediate data are in the PEs of the array processor and the final results should be pipelined out of the array processor. This type of array processor is called type II array. 3. In the 1-D transforms along all other dimensions, both the intermediate data and the transformation results should stay in the PEs of the array processor. This type of array processor is called type III array. In order to construct linear array processors for computing the 1-D image transform with different I/O formats, we first derive a 2-D localized datadependency graph (LDG) of the 1-D image transform (see Fig. 1). Note that the
i
T (0)
T (1)
T (2)
T (3)
f (3)
a3,0
a3,1
a3,2
a3,3
f (2)
a2,0
a2,1
a2,2
a2,3
f (1)
a1,0
a1,1
a1,2
a1,3
f (0)
a0,0
a0,1
a0,2
a0,3
j
0
0
0
0
Fig. 1. LDG of 1-D image transform (N = 4) 1-D image transform is just a vector-matrix multiplication T = fA, where A is an N × N transformation matrix with aij = g(i, j), 0 ≤ i, j ≤ N − 1, f is the image vector, and T is the transformation vector. If we project the LDG along i direction, we get a type I array (see Fig. 2). Similarly, if we project the LDG along j direction, we get a type II array (see Fig. 3). A type III array can be constructed by modifying the type I array as follows. Assume that f (x) is in the xth PE initially. Each PE of the type III array has three input and two output channels. The two additional I/O channels are for shifting data f (x), 0 ≤ x ≤ N − 1. Datum f (x) is left-shifted to the leftmost PE in x steps, and then, is used for computing T (u) and then right-shifted to the next PE repeatedly at each time step as in type I array. The type III array is shown in Fig. 4.
Design of Multi-dimensional DCT Array Processors for Video Applications T (0)
T (1)
T (2)
1089
T (3)
f (3) f (2) f (1) f (0) T (u) in1
a0,0 a1,0 a2,0 a3,0
out1
a0,1 a1,1 a2,1 a3,1
a0,2 a1,2 a2,2 a3,2
a0,3 a1,3 a2,3 a3,3
T (u) := T (u) + in1 * in2; out1 := in1
in2
Fig. 2. Type I array for 1-D image transform (N = 4) 0
f (0)
a0,0 a0,1 a0,2 a0,3
f (1)
a1,0 a1,1 a1,2 a1,3
f (2)
f (3)
T (3) T (2) T (1) T (0)
in1 a2,0 a2,1 a2,2 a2,3
a3,0 a3,1 a3,2 a3,3
f (u)
out1
in2 out1 := in1 + f (u) * in2
Fig. 3. Type II array for 1-D image transform (N = 4) The computing time is 2N − 1 for the first two array processors. The computing time for the third array processor is 2N since one additional time step is needed to get the initial data f (x), 0 ≤ x ≤ N − 1, from the registers inside the PEs of the array.
3
Array Processor Designs for 1-D DCT
In this section, we will refine the three types of linear array processors in the previous section for the 1-D DCT. The main task is to reduce the I/O overhead T (0)
T (1)
T (2)
T (3)
f (0)
f (1)
f (2)
f (3)
a0,0 a1,0 a2,0 a3,0
a0,1 a1,1 a2,1 a3,1
a0,2 a1,2 a2,2 a3,2
a0,3 a1,3 a2,3 a3,3
T (u) out1 in2
f (u)
in1 out2
First step: in3 out1 := f (u); Other steps: out1 := in1; out2 := in2; T (u) := T (u) + in2 * in3
Fig. 4. Type III array for 1-D image transform (N = 4)
1090
Shietung Peng and Stanislav Sedukhin
using the special structure of the DCT kernel. The cost for precomputing the transformation matrix outside the array is very high, and the precomputing also incurs high I/O overhead. Therefore, designs which generate the matrix entries locally while needed are of practical importance. The 1-D DCT uses the following kernel: 1 uπ 2 cos(x + ) , g(x, u) = C(u) N 2 N √ where C(u) = 1/ 2 if u = 0, otherwise 1. The 2-D and 3-D DCT kernels are shown below. g(x, y, u, v) = C(u)C(v)
1 uπ 1 vπ 2 cos(x + ) cos(y + ) , N 2 N 2 N
and g(x, y, z, u, v, w) = 1 vπ 1 wπ C(u)C(v)C(w)( N2 )3/2 cos(x + 12 ) uπ N cos(y + 2 ) N cos(z + 2 ) N . For video applications, the size of the kernel matrix can be extremely large. How to generate the needed entries of the kernel matrix in the local buffers when perform local DCT computations in each of the active PEs is the key for the design of efficient linear array processor. For simplicity, we omit the scale-factor, 2 N,
in our designs. The scaling can be done easily at the end of processing.
First, we notice that when we perform f (x) × cos(x + 12 ) uπ N in PE(u), the 1 u π values cos(x + 2 ) N , u < u, were created and used in the PE(u ) already. Some of these values can be delivered through pipelining to PE(u) in order to generate cos(x + 12 ) uπ N . The following formula can be used for this purpose 2 cos uδ cos δ = cos(u + 1)δ + cos(u − 1)δ. This formula shows that the value cos(u + 1)δ can be obtained from three values: cos uδ, cos(u − 1)δ, and cos δ. Based on this idea, we develop linear array processors which, instead of creating the whole kernel matrix externally, need only an N -vector, 1, cos θ, cos 3θ, . . . , cos(2N −1)θ, as input. The required entries of the matrix will be calculated locally on the fly inside each PE. How this vector is delivered depends on the types of the linear array processors proposed in the previous section. For the type I array, this vector will be pipelined into the array from left to right as the input vector f (0), f (1), . . . , f (N − 1). Besides, there are two more channels for the transmissions of intermediate values, cos uδ and cos(u − 1)δ, needed for computing cos(u + 1)δ at PE(u + 1). Since cos uδ and cos(u − 1)δ had been computed at PE(u) and PE(u − 1), respectively, they can be made available in PE(u + 1) through two additional channels, and the proper transmission scheme for right timing during the pipelining computations. The details of the array structure (and its PE functions) for computing 1-DCT based on type I array is depicted in Fig. 5.
Design of Multi-dimensional DCT Array Processors for Video Applications T(0)
f(3)
T(1)
T(2)
1091
T(3)
f(2) f(1) f(0)
cos7q cos5q cos3q cosq
PEu: 1£ u £ N-1
PE0:
T(u) in1
< in2, in3, in4 >
T(0)
out1
< out2, out3 , out4>
in1
out1
in2
< out2, out3 , out4>
initialization:
initialization:
T(u) ¬ 0; continuation:
T(0) ¬ 0; continuation:
T(u) ¬ T(u) + in1 ´ in2; out1 ¬ in1; < out2, out3 , out4> ¬ < 2 in2 ´ in4 in3, in2, in4>
T(0) ¬ T(0) + in1; out1 ¬ in1; < out2, out3 , out4> ¬ < in2, 1, in2 >
Fig. 5. DCT-I, Type I array for the 1-D DCT (N = 4) The linear array processor of type II for computing 1-D DCT is simple (no additional channels are needed). The two values cos uδ and cos(u − 1)δ are kept now inside PE(u), and are updated and used locally in the next time step. The array structure and the PE functions of the type II array for computing 1-D DCT (DCT-II) are shown in Fig. 6. In each PE of DCT-II, three registers c0 , c1 , and c2 , are used to hold values cos δ, cos(u − 1)δ, and cos uδ. An additional register c3 is used while updating the values of c1 and c2 . Finally, the array structure and the PE functions of the type III array for computing 1-D DCT, called DCT-III, are shown in Fig. 7. It follows the same design as in DCT-I.
4
Array Processor for Multidimensional DCT
The 2-D array processor for the 2-D DCT can be constructed using DCT-I and DCT-II on i and j dimensions, respectively. An example array processor, where N = 4, is shown in Fig. 8. This array processor contains N × N PEs. Each PE can be divided into two parts: functions in left (right) part is the same as the PEs in DCT-I (DCT-II). The input data, [f (x, y)]N ×N , are fed into the array column-by-column, and the transformation data, [T (u, v)]N ×N , are generated row-by-row. This design allows the intermediate results z(u, y), the results of the 1-D DCTs along the i dimension, to be kept inside the PEs of the array processor. Matrix transposition is not needed. The size of the array is N 2 , and the total computing time is 2(N − 1) + 2(N − 1) = 4N − 2 time steps.
1092
Shietung Peng and Stanislav Sedukhin
f(1)
f(0)
0
1
cosq
PEi : 0£ i £ N-1
f(2)
f(3)
cos3q
cos5q
T(3) T(2) T(1) T(0)
initialization: in2 ¬ if i = 0 then 1 else cos(2i - 1)q
in1
f(i)
out1
c0 ¬ c2 ¬ in2; c1 ¬1; continuation: out1 ¬ in1 + f (x) ´ c2; c3 ¬ c2; c2 ¬ 2c2´ c0 - c1; c1 ¬ c3;
in2
Fig. 6. DCT-II, Type II array for the 1-D DCT (N = 4)
cos5q cos3q cosq
T(0)
T(1)
T(2)
T(3)
f(0)
f(1)
f(2)
f(3)
PE0:
PEu: 1£ u £ N-1
T(0)
T(u) out1 in5
f(u)
< in2, in3, in4 >
in1 out5
< out2, out3 , out4>
out1 in5
in1 out5
f(0)
in2
< out2, out3 , out4>
initialization:
initialization:
T(u) ¬ 0; continuation:
T(0) ¬ 0; continuation:
first step: out1 ¬ f(u); other steps:
first step: out1 ¬ f(u); other steps:
T(u) ¬ T(u) + in5 ´ in2;
T(0) ¬ T(0) + in5;
out1 ¬ in1; < out2, out3 , out4> ¬ < 2 in2 ´ in4 in3, in2, in4>
out1 ¬ in1; < out2, out3 , out4> ¬ < in2, 1, in2 >
out5 ¬ in5;
out5 ¬ in5;
Fig. 7. DCT-III, Type III array for the 1-D DCT (N = 4)
Design of Multi-dimensional DCT Array Processors for Video Applications
0
1093
T(3,3) T(3,2) T(3,1) T(3,0)
0
T(2,3) T(2,2) T(2,1) T(2,0)
0
T(1,3) T(1,2) T(1,1) T(1,0)
0
T(0,3) T(0,2) T(0,1) T(0,0)
1 cosq f(0,0) cos3q f(1,0) cos5q f(2,0) cos7q f(3,0)
cosq f(0,1) f(1,1) f(2,1) f(3,1)
cos3q
cos5q
f(0,2) f(1,2) f(2,2) f(3,2)
f(0,3) f(1,3) f(2,3) f(3,3)
Fig. 8. An example array processor for the 2-D DCT
The 3-D array processor for the 3-D DCT can be constructed similarly using DCT-I, DCT-III, and DCT-II on i, j and k dimensions, respectively. Similar to the 2-D case, the PEs of the 3-D array processor are divided into three parts. The functions of each part is equal to that of corresponding PEs in DCT-I, DCT-II, and DCT-III. All proposed array processors can be scaled to different size of problems. Notice that further refinement of the proposed array processors is possible. In this paper, the emphasis is on the key issues of design.
5
Concluding Remarks
In this paper, we proposed a new method for designing array processors for computing multi-dimensional image transforms. We applied this method for the multi-dimensional DCTs which are used widely in video applications. The proposed array processors are highly-regular, time-efficient, fully-pipelined, scalable, and have a small I/O overhead. The advance in VLSI technology will make it possible to develop the 2-D and 3-D array processors proposed in this paper.
References 1. Gonzalez, R.C. and Woods, R.E., Digital Image Processing, Addison-Wesley, 1992. 2. Miyazaki, T., Nishitani, T., Edahiro, M., Ono, I., and Mitsuhashi, K., ”DCT/IDCT Processor for HDTV developed with DSP Silicon Compiler”, Journal of VLSI Signal Processing, No. 5, pp. 151–158, 1993.
1094
Shietung Peng and Stanislav Sedukhin
3. Pratt, W.K., Digital Image Processing, John Wiley & Sons, 1991. 4. Rao, K.R. and Yip, P., Discrete Cosine Transform: Algorithms, Advantages, and Applications, Academic Press, 1990. 5. Slawecki, D. and Li, W., ”DCT/IDCT Processor Design for High-data Rate image Coding”, IEEE Trans. Circuits Syst. Video Techn., Vol. 2, pp. 135–146, 1992 6. Sun, M.-T., Chen, T.C., and Gottlieb, A.M., ”VLSI Implementation of a 16x16 Discrete Cosine Transform”, IEEE Trans. Circuits Syst., Vol. 36, pp. 610–617, 1989. 7. Tonge, G., ”Image Processing for Higher Definition Television”, IEEE Trans. Circuits Syst., Vol. 34, pp. 1385–1398, 1987.
Design of a Parallel Accelerator for Volume Rendering Bertil Schmidt School of Applied Science, Nanyang Technological University, Singapore 639798, [email protected] Abstract. We present the design of a flexible massively parallel accelerator architecture with simple processing elements (PEs) for volume rendering. The underlying parallel computer model is a combination of the SIMD mesh with the instruction systolic array (ISA), an architectural concept suited for easy implementation in very high integration technology. This allows the parallel accelerator unit to be built as a programmable low cost co-processor, that suffices to render volumes with up to 16 million voxels (2563) at 30 frames per second (fps).
1
Introduction
Volume visualisation [4] is a key technology for the interpretation of 3D scalar data generated by acquisition devices such as biomedical scanners, by supercomputer simulation, or by voxelising geometric models. Especially important for the exploration and understanding of the data are sub-second display rates and instantaneous visual feedback during change of rendering parameters. This is a challenging task due to its rigorous requirements. Firstly, the datasets are very large, typically over 16 MBytes and sometimes exceeding 150 MBytes. Secondly, to be useful the system must be able to produce images at interactive frame rates, preferably at 30 fps, but at least greater than 10 fps. These tremendous storage and processing requirements have limited the widespread use of volume visualisation. Consequently, research has been conducted towards the development of dedicated volume rendering architectures [11,12,14]. VolumePro [12] is the first single-chip real-time accelerator for consumer PCs. However, the disadvantage of these special-purpose systems is the lack of flexibility with respect to the implementation of different algorithms, e.g. interactive segmentation, feature extraction and other tasks which are to be performed on volume datasets before rendering cannot make use of the special-purpose architecture. ISAs combine the speed and simplicity of systolic arrays with flexible programmability [6, 8], i.e. they achieve extremely high performance cost ratio and can at the same time be used for a wide range of applications, e.g. scientific computing, image processing, multimedia video compression, computer tomography, and cryptography [15-18]. Thus, the ISA architecture fits well for performing high-speed visualisation and processing of 3D datasets at low cost. In this paper we present an ISA architecture that can solve all components of a volume rendering application efficiently by taking advantage of their high degree of inherent parallelism. It has been designed in order to render volumes with up to 2563 voxels at real-time. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1095-1104, 2000. Springer-Verlag Berlin Heidelberg 2000
1096
Bertil Schmidt
This paper is organised as follows. Section 2 gives an introduction to volume rendering algorithms. In Section 3 previous SIMD implementations of volume rendering are described. The concept of the ISA is explained in Section 4. Section 5 presents the new accelerator architecture. The parallel algorithms for volume rendering are explained in Section 6 and their performance is evaluated in Section 7. The outlook to further research topics concludes the paper in Section 8.
2
Volume Rendering Algorithms
Volume rendering involves the direct projection of an entire 3D dataset onto a 2D image plane. The data is sampled on a rectilinear grid, represented as a 3D array of volume elements, or voxels. Volume visualisation algorithms can simultaneously reveal multiple surfaces, amorphous structures, and other internal structures. These algorithms can be divided into two categories: forward-projection and backward projection. Forward projection algorithms iterate over the dataset during the rendering process projecting voxels onto the image plane. A common forward-projection algorithm is splatting [21]. Backward-projection iterates over the image plane during the rendering process by resampling the dataset at evenly spaced intervals along each viewing ray. Ray casting [10] is a common backward-projection algorithm. In ray casting, rays are cast into the dataset. Each ray originates at the viewing position (eye), penetrates a pixel in the image plane (screen), and passes through the dataset. At evenly spaced intervals along the ray, samples are computed using interpolation. The sample values are mapped to display properties such as opacity and colour. A local gradient is combined with a local illumination model at each sample point to provide realistic shading of the object. Final pixel values are found by compositing colour and opacity values along a ray. Composition models the physical reflection and absorption of light. Ray casting offers room for algorithmic improvements by still allowing for high image quality. Several variants of traditional ray casting have been introduced, e.g. [7,23]. The modifications to the original ray casting algorithm to make it more suitable for our parallel accelerator architecture are presented in Section 6.
3
Previous SIMD Volume Rendering Work
Schröder and Stoll proposed an algorithm for the Connection Machine CM2 where the volume is stored one beam per PE. However, the inherent latency of CM2 limited their performance to 4 fps for a 1283 volume [19]. Yoo et al. presented a method to perform volume rendering on the Pixel Planes 5 machine partly utilising the 2D SIMD mesh and partly the MIMD Graphic processors [24]. They achieved 20 fps for a 128x128x56 volume. Hsu designed a segmented ray casting approach for the DECmpp SIMD mesh [3]. However it distributed the volume in subblocks and only achieved 4-5 fps. Both Vezina [21] and Wittenbrink [23] proposed algorithms for the MASPAR MP-1 (a SIMD 8-connected mesh). Yet, neither achieved frame rates better than 2-5 fps. All of those methods suffered because of the latency inherent in large general-purpose machines. Dogett [2] presented a special-purpose architecture with a 2D array of PEs for volume rendering. However, his PEs are not programmable
Design of a Parallel Accelerator for Volume Rendering
1097
ASICs. The PAVLOV design presented in [5] achieves 30 fps for a 2563 volume on a 64x64 torus of 8-bit-parallel PEs. This architecture is close to our approach since it is a 2D mesh with simple PEs. However, its communication mechanism assumes enough memory to store two times the complete volume on-chip. This is extremely costly in terms of area requirements for PEs as compared to our design.
4
Principle of the ISA
The ISA is a quadratic array of identical processors, each connected to its four direct neighbours by data wires. The array is synchronised by a global clock. The processors are controlled by instructions, row selectors, and column selectors. The instructions are input in the upper left corner of the processor array, and from there they move step by step in horizontal and vertical direction through the array. This guarantees that within each diagonal of the array the same instruction is active during each clock cycle. In clock cycle k+1 processor (i+1,j) and (i,j+1) execute the instruction that has been executed by processor (i,j) in clock cycle k. The selectors also move systolically through the array: the row selectors horizontally from left to right, column selectors vertically from top to bottom. Selectors mask the execution of the instructions within the processors, i.e. an instruction is executed if and only if both selector bits, currently in that processor, are equal to one. Otherwise, a no-operation is executed. This construct leads to a flexible structure, which creates the possibility of very efficient solutions for a large variety of applications, e.g., numeric, image processing, video compression, and cryptography [16-19].
instructions ×
− +
1 1 1 1 1 1 1 1 1 column 0 1 selectors 1
1 0 1 1 1 1 1 1 1 0 1 1
row selectors
Fig. 1: Control flow in an ISA
Every processor has read and write access to its own memory. Besides that, it has a designated communication register (C-register) that can also be read by the four neighbour processors. Within each clock phase reading access is always performed before writing access. Thus, two adjacent processors can exchange data within a single clock cycle in which both processors overwrite the contents of their own Cregister with the contents of the C-register of its neighbour. This convention avoids read/write conflicts and also creates the possibility to broadcast information across a whole row or column with one single instruction. This property can be exploited for an efficient calculation of row broadcasts, row ringshifts, and row sums, which are the key-operations in many algorithms.
1098
5
Bertil Schmidt
Accelerator Architecture Design
RAM RAM
RAM RAM
RAM RAM
ISA ISA
RAM RAM
ISA ISA
ISA ISA
RAM RAM
ISA ISA
ISA ISA
RAM RAM
ISA ISA
ISA ISA
RAM RAM
ISA ISA
ISA ISA
RAM RAM
RAM RAM RAM RAM
ISA ISA
ISA ISA
RAM RAM
ISA ISA
ISA ISA
RAM RAM
RAM RAM RAM RAM
Program Program Memory Memory
ISA ISA
RAM RAM
Conroller Conroller
ISA ISA
RAM RAM
Since our aim is to develop a special-purpose architecture for multimedia applications, it is highly desirable to design hardware that can be installed on PCs within the price range of a PC, i.e. to design add-on-boards for PCs. Due to experience gathered in the cause of designing and fabricating the Systola 1024 (also an add-on-board for PCs, with a 32x32 ISA built out of 16 processor chips of 64 processors each [9]), we are in the position of being able to make reliable performance predictions by extrapolation, based on the change of technology parameters. While the Systola 1024 is based on 1.0-micron technology, we are now able to use .25-micron technology, so that on the same chip area that contains 64 processor in case of Systola 1024 we can now place 1024 processors and on a single PC-board we can place 16K processors (together with memory chips, memory multiplexers and a controller with program memory). As on-chip communication can be clocked at significantly higher frequency than chip-to-chip communication, we have decided to use the 16 processor chips relatively independently, i.e. we assume that the applications allow simple data partitioning. Chip-to-chip communication is done exclusively via locally shared memory, i.e. each processor chip is connected to a memory chip via a simple multiplexer that also allows access to the memories of the four direct neighbours (NEWS) -- here we assume a torus architecture in order to be able to perform easily horizontal and vertical ringshift of data (see Fig. 2). Thus, by avoiding direct off-chip communication we can assume an on-chip clock cycle time of 200 MHz.
ISA ISA
Fig. 2: Data paths of the accelerator architecture
The analysis of ray casting and its processing efforts leads to a fixpoint PE architecture (see Section 6). The PE needs a small local memory for the storage and fast supply of local voxel data. For 8-bit input voxels an intermediate operand length of 16 bits in most computations provides enough accuracy for ray casting [1]. Thus, the wordlength of the data items is set to 16 bits. To allow flexible use of the architecture the PEs must also be able to process longer operands and shorter operands efficiently, e.g. adding two 32-bit numbers in two instructions or adding two
Design of a Parallel Accelerator for Volume Rendering
1099
8-bit numbers in one instruction. This idea is incorporated in the design of our computational units. Figure 3 depicts the PE architecture for volume rendering. column selector in
instruction in
data links north
R63
external/ internal write
global bus
external/ internal read
R48 R47
internal write
64
internal read
RAM R0 R0
data links west
rsrs cs cs
row selector out
instruction instructiondecoder decoder conditional conditional unit unit ALU ALU
Op0-Mux Op0-Mux Op1-Mux Op1-Mux
row selector in
communication communication register register
data links east
flag flag manager manager shifter shifter multiplier multiplier
column selector out
instruction out
clock clock
data links south
Fig. 3: Block diagram of the processor architecture
Due to the limited chip area the processor has to be very compact. This leads to our choice of a bit-serial data organisation. The bit-serial design allows a higher number of PEs per chip and a higher clock frequency than a corresponding bit-parallel design. The main components of the PE are a set of 64 data registers, a C-register, an ALU, a conditional unit, a multiplier, and a shifter. In addition to the registers there are flags (zero flag, negative flag, activation flag) that control the processing units depending on the state of the processor and several special registers. The wordlength of data items is 16 bits. Because the data is processed bit-serially, the execution of each instruction takes exactly 16 clock cycles. After receiving an instruction, the PE stores it in the instruction register, decodes the two operand addresses and the destination address, retrieves the operands from the register file, executes the instruction, writes the result back to the destination register, and passes the instruction to the next processor. The corresponding instruction set consists of 44 instructions. Since all this is done bit-serially, it can be pipelined on bit-level, such that a new instruction can be fetched and processed every 16 clock cycles. Extrapolating the design parameters used for Systola 1024 allows us to predict that a 32x32 array of these PEs on a 1cm2 chip is realistic for a .25-micron CMOS process with a 200 MHz true single phase clock. For a word format of 16 bits the theoretical peak performance for one chip is 12.8 GIPS and for the complete board 204.8 GIPS.
1100
Bertil Schmidt
There is already a SIMD single chip architecture with a 32x32 array of bit-serial PEs in .25-micron technology on the market [20]. But the architecture proposed in this paper achieves twice the clock frequency due to adhering as closely as possible to local communication, and its main advantage it gets through its unique control structure that allows the execution of aggregate/reduction functions in a fraction of time as compared to conventional SIMD architectures. For the fast exchange of data with the processor array each PE has two memory banks. Each memory bank contains 8 interface registers. One of these banks is always assigned to the corresponding processor, the other to a neighbouring memory chip by means of a fast data channel. The exchange of data between ISA and the memory chip is done by bank switching. Both memory banks can be active at the same time, i.e. data transfer can be done concurrently to the execution of an ISA program.
6
Mapping of Ray Casting to the Accelerator Architecture
Fig. 4 shows three possible approaches to parallelising ray casting. According to the form of parallelism that is exploited, we call them ray, beam, and slice parallel.
a) ray parallel
b) beam parallel
c) slice parallel
Fig. 4: Three different approaches to parallelising ray casting. Shaded voxels are processed simultaneously. The thick arrows indicate the direction in which the algorithm proceeds.
In the ray parallel approach, all voxels along a ray are processed simultaneously (the shaded voxels in Fig. 4a). The algorithm proceeds ray by ray in scanline order (the thick arrow in Fig. 4a). However, simultaneous access to all voxels along a ray requires irregular data transfer patterns between volume memory and PEs. An alternative to operating on all samples of a single ray is to simultaneously operate on samples of several neighbouring rays. Depending on how the algorithm proceeds, we call these approaches beam parallel (Fig. 4b) and slice parallel (Fig. 4c). A beam is a line of voxels that is parallel to a principle axis of the dataset. The beam parallel ray casting approach follows a group of rays by fetching consecutive beams in major viewing direction. However, the stepping along slanted planes of rays requires complicated addressing mechanisms. The slice parallel approach processes consecutive data slices that are parallel to the base plane of the volume dataset (Fig. 4c) and achieves a uniform data access. The base plane is the face of the volume that is closest to perpendicular to the major component of the viewing direction. A 2D array of ISA PEs can inherently process slice order algorithms very efficiently, since an entire slice of the volume can be processed in parallel. Therefore,
Design of a Parallel Accelerator for Volume Rendering
1101
we choose the slice order approach to be mapped on our architecture. Our implementation combines the slice order ray casting approach [1] with segmented ray casting [3] for parallel projections: The volume is partitioned into subcubes. These subcubes are distributed evenly across the memory modules. Each ISA chip computes the colour and opacity values of the portion of the rays, which lie inside the subblock, and writes them into its adjacent memory module. After all subcubes have been processed the segments are composited using chip-to-chip communication. The algorithm consists of the following steps: Subcube partitioning: Determined by the memory capabilities of PEs, the size of the non-overlapping subcubes is set to 643. Each slice is mapped onto a 32x32 ISA by loading 2x2 voxels into each PE. As the algorithm requires a small local neighbourhood of each voxel, three slices are stored in the processor array at any time and processors at the borderline need some data from neighbouring subcubes. Gradient estimation: The first computing step is the determination of gradients to approximate surface normals for classification and shading. x-, y-, and z-gradient are computed for a voxels sample value Pi,j,k at location (i,j,k) using central differences: Gx = Pi+1,j,k - Pi-1,j,k, Gy = Pi,j+1,k - Pi,j-1,k,, Gz = Pi,j,k+1 - Pi,j,k-1. Each PE can compute gradients for its 2x2 voxel samples of the current slice in parallel using neighbouring samples. Because the processor array holds three slices at the same time, samples needed from the ahead and behind slice are stored locally in each PE. Samples needed in the two dimensions within the current slice are either also stored locally or in one of the four neighbouring PEs. Other algorithms that use larger neighbourhoods and produce higher quality gradients at additional computational costs can also be mapped efficiently on our architecture. Afterwards gradient magnitude computation continues locally by taking the sum of the squares of the gradient components and then a Newton-Raphson iteration to compute the square root of this value, resulting in an approximation of the gradient magnitude. Classification: Classification maps a colour and opacity to sample values. Opacity values range from 0.0 (transparent) to 1.0 (opaque). On special-purpose architectures [11,12,14] classification is typically implemented using look-up tables (LUTs). These LUTs are addressed by sample value and gradient magnitude and they output sample opacity and colou. In our architecture using LUTs is not appropriate, as the local memories of PEs are very small. Thus, we are using few low degree polynomials depending on the sample value (for colour) and the product of sample value and gradient (for opacity). Shading: The Phong shading algorithm [13] is often used in shading subsystems within volume rendering architectures. It requires gradients, light, and reflection vectors to calculate the shaded colour for each sample location. The shading calculation can be expressed as: I = A + D (L*N) + S (R*V)s, where N is the (normalised) gradient vector, I is the light vector, R is the reflection vector, V is the viewing vector, A, D, L represent ambient, diffuse, and specular material components, and s is the specular exponent. The shading equation can be computed in each PE locally. To normalise the gradient vector we compute the reciprocal of the gradient magnitude by a Newton-Raphson iteration, followed by three multiplications. Parallel view and light vectors are assumed in order to make the reflection independent of the place. Thus, L and V can be stored as constants within each processor. In this case also the computation of the reflection vector can be avoided by using the halfway vector between L and V instead.
1102
Bertil Schmidt
Compositing: Compositing is responsible for summing up colour and opacity contributions from interpolated sample locations along a ray into a final pixel colour for display. The front-to-back formulation for composting is Cacc = (1.0 - Aacc) Csample + Cacc and Aacc = (1.0-Aacc) Asample+ Aacc, where Cacc is the accumulated colour, Aacc is the accumulated opacity, Csample is the interpolated samples colour, and Asample is the interpolated samples opacity. Since we are processing in slice order fashion, the data is sampled in each slice at the point the ray would intersect the current slice by using bilinear interpolation. Because all the points needed for bilinear interpolation are contained within the slice of voxels currently being processed, it is simpler than trilinear interpolation performed in traditional ray casting. It is also more accurate than nearest neighbour interpolation. As the algorithm moves through the dataset, the point where the ray intersects the current slices moves off the current PE position. This offset is stored and accumulated. Once the ray moves closer to another PE position, the compositing information is shifted to be stored in the corresponding neighbouring PE. In other words, the compositing information of each ray is stored in the PE closest to the ray intersection with the current slice. For parallel projections the corresponding data movement pattern is regular. Thus, whenever a ray attempts to shift to another PE, all the rays in the entire slice buffer shift together. The rays are cut into segments by the planes that separate the subcubes. These planes are either parallel to the x-y-plane, or the x-z-plane, or the y-z-plane. We refer to these planes as x-y-planes, x-z-planes, and y-z-planes. Without loss of generality we assume that the main viewing axis is the z-axis. Firstly, we composite rays that pierce the x-z-plane between subblocks and then we composite rays that pierce the y-z-plane. Due to the fact that z is the main axis, this can be done in one step. Afterwards compositing has to happen at the x-y-planes. This can be done for k x-y-planes in log2k steps using a binary tree approach. Finally, a 2D warp depending on the viewing vector is computed to produce the image for display. Since this is only a 2D operation, it does not influence overall computing time significantly and can be neglected.
7
Performance Evaluation
We execute ray casting within a 2563 volume by firstly executing ray casting within 643 subblocks (subblock processing) and secondly compositing results of rays that move through neighbouring subblocks (final compositing). We have written a C++ cycle accurate simulation of our architecture. During subblock processing we produce the rays slice by slice. Each slice needs 1385 instructions as shown in Table 1. (Table 1 also shows the number of instructions for each substep.) Table 1: Instruction count (IC) for the ray casting algorithm of Section 6 of one 64x64 slice of a 643 subblock with 8-bit voxels on a 32x32 ISA module. For intermediate operands we mostly use a length of 16 bits. Task IC
Gradient 369
Classification 208
Shading 504
Compositing 304
Sum 1385
Design of a Parallel Accelerator for Volume Rendering
1103
Assuming an instruction cycle of 80 ns and computing 64 slices per subblock and 4 subblocks per ISA module, leads to total execution time of 28.4 ms for subblock processing. The data I/O for these steps (based on 150 MBytes/s throughput between each ISA module and RAM) is totally dominated by above computing time and thus can be ignored (see Section 5). Because the final composition step does not require bilinear interpolation it is dominated by the data transfer time. In the worst case (45° viewing angles) it requires 392 KByte per module. The runtime for a 2563 volume is shown in Table 2. The processing time for larger volumes scales linear with the volume size. Table 2: Runtime for the rendering of a 2563 volume with 8-bit voxels on the introduced accelerator architecture. It includes computing time on the ISA and data transfer time between ISA modules and RAM. Task Runtime Accelerator
8
Subblock 28.4 ms
Final Compositing 2.9 ms
Sum 31.3 ms
Conclusions
In this paper we have presented a massively parallel architecture for volume rendering combining the SIMD computing model with the ISA concept. The accelerator unit has been designed as a co-processor to fit into an inexpensive PC class machine. The global architecture of the accelerator engine has been discussed as well as the detailed implementation of PEs. It has been shown how a volume rendering application can be mapped on the new architecture in order to render a 2563 volume in real-time. The introduced architecture is faster, cheaper, and smaller than previous generalpurpose SIMD mesh arrays. Different from special-purpose designs, it provides more functionality, e.g. it allows multiple rendering algorithms, and, more importantly, it allows volume processing such as segmentation and feature extraction. The design will give be benefits to a medical or scientific PC where normally users wish to do more than merely render volumetric data. Future work would include identifying applications that profit from this type of processing power. For example, some users may wish to analyse the frequency of local density patterns of a volume and subsequently visualise these measurements. It would be also interesting to study the performance of the new architecture in totally different application areas like scientific computing and multimedia video processing.
References 1. 2. 3.
Bitter, I., Kaufman, A.: A Ray-Slice-Sweep Volume Rendering Engine, Proc. SIGGRAPH/Eurographics’97, ACM (1997) 121-130 Doggett, M.: An array based design for Real-Time Volume Rendering, Proc. Eurographics’95, Eurographics (1995) 93-101 Hsu, W. M.: Segmented Ray Casting for Data Parallel Volume Rendering, Parallel Rendering Symposium, IEEE (1993) 7-14
1104 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24.
Bertil Schmidt
Kaufman, A.: Volume Visualization, IEEE CS Press (1991) Kreeger, K., Kaufman, A.: PAVLOV: A Programmable Architecture for Volume Processing, Proc. SIGGRAPH/Eurographics’98, ACM (1998) 77-86 Kunde, M., et al.: The Instruction Systolic Array and its Relation to Other Models of Parallel Computers, Parallel Computing 7 (1988) 25-39 Lacroute, P.: Analysis of a Parallel Volume Rendering System Based on the Shear-Warp Factorization, IEEE Trans. on Visualization and Comp. Graphics 2 (3) (1996) 218-231 Lang, H.-W.: The Instruction Systolic Array, a parallel architecture for VLSI, Integration, the VLSI Journal 4 (1986) 65-74 Lang, H.-W., Maa,ß R., Schimmler, M.: The Instruction Systolic Array - Implementation of a Low-Cost Parallel Architecture as Add-On Board for Personal Computers, Proc. HPCN 94, LNCS 797, Springer Verlag (1994) 487-488 Levoy, M.: Display of Surfaces from Volume Data, IEEE Computer Graphics and Applications 5 (3) (1988) 29-37. Meinß er, M., Kanus, U., Straeß r, W.: VIZARD II: A PCI-Card for Real-Time Volume Rendering, Proc. SIGGRAPH/Eurographics’98, ACM (1998) 61-67 Pfister, H., et al.: The Volume Pro Real-Time Ray-Casting System, Proc. SIGGRAPH’99, ACM (1999) 251-260 Phong: Illumination for Computer Generated Pictures, Comm. ACM 18(6) (1975) 311-317 Ray, H., et al.: Ray Casting Architectures for Volume Visualization, IEEE Trans. On Visualization and Computer Graphics, 5 (3) (1999) 210-223 Schimmler, M., Lang, H.-W.: The Instruction Systolic Array in Image Processing Applications, Proc. Europto 96, SPIE 2784 (1996) 136-144 Schmidt, B., Schimmler, M.: A Parallel Accelerator Architecture for Multimedia Video Compression, Proc. EuroPar’99, LNCS 1685, Springer Verlag (1999) 950-959 Schmidt, B., Schimmler, M., Schrdö er, H.: Long Operand Arithmetic on Instruction Systolic Computer Architectures and Its Application to RSA cryptography, Proc. EuroPar’98, LNCS 1470, Springer Verlag (1998) 916-922 Schmidt, B., Schimmler, M., Schrdö er, H.: The Instruction Systolic Array in Tomographic Image Reconstruction Applications, Proc. PART’98, Springer Verlag (1998) 343-354 Schrdö er, P., Stoll, G.: Data Parallel Volume Rendering on the MasPar MP-1, Workshop on Volume Visualization, ACM (1992) 25-32 Teranex Inc.: Parallel Processing Solves the DTV Format Conversion Problem, http://www.teranex.com/whitepapers.html (1999) Vezina G., Fletcher, P. A., Robertson, P. K.: Volume Rendering on the MasPar MP-1, Workshop on Volume Visualization, ACM (1992) 3-8 Westover, L.A.: Splatting: A Parallel, Feed-Forward Volume Rendering Algorithm, PhD thesis, Dept. of Computer Science, Univ. of South Carolina in Chapel Hill (1991) Wittenbrink, C.M., Somani, A.K.: Time and Space Optimal Data Parallel Volume Rendering using Permutation Warping, Parallel and Distrib. Comp. 46(2) (1997) 148-164 Yoo, T.S. et al.: Direct Visualization of Volume Data, IEEE Computer Graphics and Applications 12 (4) (1992) 63-71
Automated Design of an ASIP for Image Processing Applications Henjo Schot and Henk Corporaal Delft University of Technology Department of Electrical Engineering Section Computer Architecture and Digital Technique P.O. Box 5031, 2600 GA Delft, The Netherlands [email protected], [email protected]
Abstract. This paper presents the design of highly optimized TTA architectures for image processing applications. An automatic processor design framework as described in [2] is used. Specialized hardware is used to improve the performance-cost ratio of the processors. An explorer searches the design space for solutions that are good in terms of cost and performance. We show that architectures can be found that efficiently execute very different algorithms at low cost. A hardware feasible architecture is presented that efficiently executes a set of image processing algorithms and performs almost equally or better than alternative, commercial-available solutions do.
1
Introduction
In this paper, we show the design of an application specific instruction set processor (ASIP) for a set of image processing algorithms. Processors and code are generated, trying to exploit the instruction level parallelism of image processing algorithms. We show that processors can be generated that efficiently execute very different algorithms at low cost. We add application specific hardware and functionality to improve the performance cost of the processors. The architecture of the ASIP we develop is a Transport Triggered Architecture (TTA) [2]. An automated design framework, called the MOVE framework [], is used for the development of the VLIW like processor. It tries to find an architecture with an optimal cost/performance ratio. The designer can use Special Function Units (SFUs) to improve the cost-performance ratio of an architecture. These SFUs can be incorporated in the MOVE framework. Our work on the design of a highly optimized processor architecture differs from others [3] in that we use TTAs and application specific hardware (SFUs) in order to find architectures with higher performance-cost ratios. The next section describes the image processing algorithms we used. Section 3 shows the mapping of the algorithms to TTAs. Section 4 presents the results and conclusions.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1105-1109, 2000. Springer-Verlag Berlin Heidelberg 2000
1106
2
Henjo Schot and Henk Corporaal
Image Processing Algorithms
Four image processing algorithms were mapped to TTAs. A color conversion algorithm, and three gray-scale neighborhood algorithms: convolution and two edge detection algorithms on a 3x3 area. Here we concentrate on the color conversion algorithm. Color conversion is an operation from the area of color image processing. It is used to convert between color representations e.g. a color in RGB has to be converted to represent the same color using CMYK color components. There are several methods to perform color conversion. In our case we uses lookup tables (LUTs) and triinterpolation. Using this method, the conversion of a color P starts with searching the lookup table for the eight nearest points. Seven linear interpolations are performed on these points. Figure 1 shows point P and its eight nearest points that correspond with the corners of a cube. The interpolations are shown as bold lines. The final interpolation is performed on V and W. The distance a of point V to point P and the distance 1 - a of point W to point P are used as interpolation coefficients. 001
011
111 a B
V
P = (1-a)W + aV
W
0
P G
000
R
100
110
Fig. 1. Tri-linear interpolation of point P in the RGB space.
Since the calculation of each pixel is independent of other pixels, in principle, all pixels can be processed in parallel. The amount of parallelism that can be attained by TTAs is determined by the maximum number of available resources and the ability of the compiler to exploit the parallelism. In our implementation, we use a LUT of 4 K entries. This size results in reasonable interpolation accuracy at low cost. Higher accuracy requires larger lookup tables. The LUT is addressed using the 4 most significant bits of each color component. The 4 least significant bits are used for the interpolation distance giving interpolation coefficients of 0, 1/16, ... 16/16. We aim to achieve a performance that is comparable or better than that of available solutions (12.5 Mpixels/s [7]), at lower cost.
Automated Design of an ASIP for Image Processing Applications
3
1107
Mapping the Algorithms to TTAs
The main components of the MOVE framework are a retargetable C/C++ compiler, a processor generator and hardware modeller and a design explorer. The explore tool searches the design space for architecture solutions using hardware cost and performance as its main design criteria. The explore tool drives both the compiler and the hardware modeler in order to find architecture solutions with a good performance/cost ratio. A pareto-curve with the resulting architecture solutions is produced, from which the designer chooses an architecture configuration. The mapping of the algorithm starts with analyzing the solutions that are found in case we use basic operations only. The curve ‘w/o SFUs’ in figure 2 shows that the latency for high cost architectures remains quite long. E.g.. an architecture of cost 300 (in integer units) does color conversion of a single pixel in 11 cycles.
Fig. 2. The TTA design space for the color conversion application.
An enormous improvement in performance is achieved when SFUs are used. Parts of the algorithm that are implemented in hardware are the lookup operation and a linear interpolation. Figure 3 shows the dataflow symbols of different implementations of the interpolation. They are implemented by extending a multiplier FU with these specific functions. The impact of the SFUs on architectures cost/performance is also shown in figure 2. Solutions with 2 to 60 times better performance at equal cost are found. x
a
y
1 6-a x
*
y
a
x .y
a
* Interpol
packed Interpol
+ o
o
o
Fig. 3. Dataflow symbols of the interpolation functions
Architectures that execute a set of algorithms are found by combining multiple algorithms in an application. Resulting architectures showed a small loss in
1108
Henjo Schot and Henk Corporaal
performance for each algorithm, but overall they performed very well, as can be seen in figure 4.
Fig. 4. The TTA design space for both color conversion and neighborhood operations.
4
Results
A feasible processor configuration which efficiently executes the whole set of algorithms is selected from the curve in figure 4. In this configuration, marked with ‘+’, the in- and outputs of each FU and register file are connected to all buses, which is impractical. We therefore remove as much connections as possible without performance losses. The resulting processor is shown in figure5. It can do color conversion at 5.3 cycles/pixel. It contains 8 buses and 11 functional units (FUs). The register file, as shown, contains many read and write ports. However, the tools allow to split up this file into multiple small register files [4][5].
Fig. 5. A processor configuration that efficiently executes the color conversion algorithm and the whole repertoire of neighborhood operations.
Automated Design of an ASIP for Image Processing Applications
1109
Table 1 gives an overview of the performance of commercial available solutions [6]-[11] and our solution. It is seen that the Imagine and the C6x perform better for the convolution application than our solution does. For the other algorithms our solution performs significantly better. Table 1. Performances of the applications for other available solutions and our solution. Algorithm convolution Min-max operation Edge detection Color conversion
5
Imagine @ 66 MHz 37 < 15 < 15 3.0
Performance (Mpixels/s) PixelMagic 44 TI C6201 Chameleon @ 75 MHz @ 200 MHz 22.8 40 n.a. 11.8 n.a. < 11.8 n.a. 12.5
MOVE @ 100 MHz 27 27 23 19
Conclusion
In this paper, we showed that the MOVE framework can be used to find solutions for digital image processing algorithms. Solutions can be found for applications containing several algorithms, including very different ones. Furthermore, hardware feasible solutions are found that perform almost equally or better than alternative, commercially available solutions. A large part of the processor design is done automatically. However, a lot of manual interaction is required in the identification and application of Special Function Units. Automation of this part of the design trajectory is currently being researched [1].
References 1.
Marnix Arnold and Henk Corporaal, Automatic Detection of Recurring Operation Patterns, Codes ’99, May 1999 2. Henk Corporaal, Microprocessor Architectures; from VLIW to TTA, John Wiley, 1998, ISBN 0-471-97157-X 3. Joseph A. Fisher, Paolo Faraboschi and Guiseppe Desoli, Custom-Fit Processors: Letting Applications Define Architectures, 4. Jan Hoogerbrugge, Code Generation for Transport Triggered Architectures, Delft University of Technology, 1996 5. Johan Janssen and Henk Corporaal, Partitioned Register File for TTAs, Delft University of Technology 6. Redford, J., Iler, J. and Berger, E., The PM44: A Single-Chip SIMD GigaOp DSP for Imaging, Pixel Magic Inc, Andover MA 7. The Barco Chameleon ASIC; A very high speed, very high accuracy, color correction utility, Barco Graphics, 1993 8. Redford, J., Iler, J. and Berger, E., The PM44: A Single-Chip SIMD GigaOp DSP for Imaging, Pixel Magic Inc, Andover MA 9. Evaluation of the Performance of the C6201 Processor & Compiler, Loughborough Sound Images plc., 1996 10. TMX320C6201 DIGITAL SIGNAL PROCESSOR, Texas Instruments Inc., 1997 11. The Imagine engine; Documentation & User Manual, Arcobel Graphics B.V., March 1994
A Distributed Storage System for a Video-on-Demand Server Alice Bonhomme and Lo¨ıc Prylli LHPC/ ENS Lyon, 69364 Lyon, France {Alice.Bonhomme, Loic.Prylli}@ens-lyon.fr
Abstract. The aim of this paper is to present the design of a distributed storage system for a video server. The main goal is to support good faulttolerance capabilities (no single point of failure, and no perturbation of the clients at the time the failure occurs) while supporting a high-number of video streams.
1
Introduction
The aim of this paper is to present the design of a distributed storage system for a video server. A distributed architecture is quite natural for a video server, given the intrinsic parallelism provided by independent clients on one hand, and the possibility of easily fetching multiple blocks in parallel for a given stream on the other hand. We use a “PC type” cluster, which provides a good performance/price ratio. An important aspect of video servers is the continuity of service even in case of a hardware failure, a distributed architecture provides the required redundancy to handle such failure. Our goal was to minimize the resource overhead, so a parity strategy is used to manage redundancy rather than mirroring blocks. Then to handle failures without any visible perturbation in the service, the parity block is systematically read just in case (the cost of doing this is low enough). For the intended target of this video-on-demand server, no cache strategy seems possible, so all data to the clients is systematically fetched from disk.
2
Related Works
The subject of video servers (see [GVK+ 95, GM98, TMDV96]) has been explored in both theory and practice. This is still a active subject because of the diversity of goals. We list below different distributed implementations: RIO: at UCLA [MSB97] is based on random allocation of striping nodes. This makes it possible to support a wide range of multimedia needs. MITRA: at USC [GZS+ 97] focuses on optimizing the throughput by precisely controlling the data placement to optimize the movements of the disk heads. Server Array: at Eurecom [BB96a] is able to cope with many types of heterogeneity : number of nodes or disks, striping strategy, reliability schemes. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1110–1114, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Distributed Storage System for a Video-on-Demand Server
1111
Tiger: at Microsoft [BB+ 96b] implements a distributed stream scheduling, replicates data blocks and distributes each copy among a subset of nodes. From a fault tolerance point of view, these video server implementations exhibits various strategies (parity/mirroring, cf. [GLP98]) and different degrees of fault tolerance (disk, node, network). Tiger uses a distributed replication of the blocks. Mitra chooses to replicate the whole disk and to systematically read on both disks. The Server Array is designed to support any kind of failures using a combination of both strategies. In RIO, the replication scheme of the blocks seems to be more adapted. However, excepted Tiger, all the other prototypes suffer from a single point of failure due to the presence of a meta server, which is in charge of the client connection and the stream scheduling.
3
Overview of the Complete Video Server
The video server (cf. Figure 1) is a cluster of PCs Client Client Client Client with local storage units. These PCs are interconnected using a Myrinet network. This internal network is independent from the distribution Distribution Network network. The video server is divided into three parts. The first module manages RTSP client sessions via the distribution network. The second module schedules all real time IO operations and gathers data from the third module: the cluster file sysIntern Network : Myrinet tem (CFS). This CFS module relies on the inter1 nal network (using the GM communication library) to manage the distributed storage (open, Fig. 1. The video server close, read and write operations to perform on hardware architecture the cluster nodes). It mainly consists of one IOM (IO Manager) on each node. In fact, all the software components are distributed among all the server nodes. PC
PC
PC
disks
4
The Cluster File System
Video data placement and management. We use a striping strategy. A video file is divided into blocks of equal size, distributed among the nodes of the cluster or among a subset of the cluster nodes. Furthermore, for fault tolerance issues, we have parity blocks stored on a node of the subset. When a file is stripped among n nodes, this requires 1/n additional space to store the parity blocks. The distribution information for each file is stored in a global table. This table is persistent, and is replicated on each node. To keep it consistent across the nodes, it is accessed through a mutual exclusion scheme. This table also permits 1
GM: message passing system from Myricom, Inc (http://www.myri.com/GM)
1112
Alice Bonhomme and Lo¨ıc Prylli
to deal with permission problems using a counter for read sessions and write sessions. A local persistent table on each node gives the information about the location of data stored on the local disks. Implementation of distributed IO. The read function posts an asynchronous read on the distributed file. This function returns an operation descriptor that allows the user to check for the completion of the operation. A read operation is posted from a client to its local IOM, which distributes it among the involved nodes. If necessary, the IOM posts a local read using the local IO subsystem. Remote requests are handled through the internal network. Besides the file’s data blocks, the parity block is also systematically requested. If a failure occurs, the parity is already available and no additional delay is needed to get the parity information from the parity node. A status function allows to check for completion of the operation. If all the data has been retrieved, the function returns with a completed status. If one response is missing, the missing data is reconstructed using the parity information and the function returns with the completed status. If more than one response is missing, the function returns with a “not ready” status. The write operation is essentially similar to the read operation. The difference is mainly the addition of the parity blocks generation before sending the request to the remote nodes. The IOM scheme. The IOM manages at Client Manager the same time the client requests, the Call to the remote IOMs requests and the local acStriping CFS API Information Local cesses. The client requests and the loFiles Table cal accesses are stored into two interGlobal Local data structures nal queues, the remote IOM requests are Files Table IOM stored into a queue fed to the internal network. The IOM scheme consists of polling on each queue and handling the requests. Thus, depending on the request, the IOM local IO subsystem Communication Local sends messages to remote IOMs, calls the Module File System local IO subsystem to perform local acLocal Disks cesses or updates some local information. Myrinet Network The IOM must also keep the context of each client and each operation. Finally it Fig. 2. Interactions between the is responsible of the fault tolerance man- modules of the CFS agement (cf. Section 5)
5
Fault Tolerance Management
Within the distributed storage system, it is important that the failure of some component (node, disk, network link) does not cause a server crash. Therefore, we designed a strategy to support any failure (no single point of failure).
A Distributed Storage System for a Video-on-Demand Server
1113
Distinction between transient problem and permanent failure. In case of a transient problem lasting less than n seconds (node disconnected from the network for example), the corresponding node is considered as potentially failed, the missing data are computed thanks to parity blocks. Then if the problem disappears, the node comes back in the server without any special treatment. In case a fault is lasting more than n seconds, we rely on the unreachable notification mechanisms of GM to avoid false detection, and the other nodes can consider the unreachable node as permanently failed (each node regularly sends probe messages in order to detect failures even if there are no other useful communications required between the nodes). Mutual exclusion algorithm. We use a classical algorithm based on the logical clocks defined by Lamport. In order to support a node failure, we modify this algorithm by checking the state of a node that is not responding to mutual exclusion requests or that has failed while owning an exclusive access. Node recovery. When a node fails, it is important to repair this node without having to stop the server. The recovering node proceeds in several steps: it gets the current global files table and striping table after gaining exclusive access to them, then it can use the striping table and can compare it to its local table to detect video data that needs to be reconstructed, finally it sends read requests to the other nodes to reconstruct the missing data using the parity blocks.
6
Experimental Results
We validated the storage subsystem on without fault with fault a 3-nodes cluster made up of Intel bi1 client node 34 55 processors connected with a Myrinet 2 client nodes 2 × 32 2×35 network. Each node has four UltraW3 × 23 ide/7200 RPM disks in a RAID 0 con- 3 client nodes figuration, accessed using the NTFS file system. What we want to compute Fig. 3. Number of 4 Mbit/s streams is the maximum number of streams for supported on a 3-nodes cluster. a given bit-rate. We use a test program that uses the CFS to retrieve streams and checks that the real time constraints are met for each block. This allows to experimentally compute the maximum number of streams supported. This experiment is done by executing client requests from one or several nodes, and in presence of a node fault or not (in case of one failure, there can be at most 2 client nodes). The results are shown in Table 3. The maximum number of streams globally supported by the server is around 70 which can be shown to match the above hardware using the computation given in [BP00]. The lower part of the array is close to the maximum capacity of the server limited by the local IO capacity. The performance with a fault is better than without, because the redundancy of getting one extra remote block is avoided.
1114
7
Alice Bonhomme and Lo¨ıc Prylli
Conclusion
The aim of this paper was to describe the overall design of a distributed storage system targeted at video. The choices that have been made are consistent with the intended target: reliably delivering video while supporting as many streams as possible even in case of failure. If redundancy strategies and fault recovery have been well studied in previous works, in practice the problem of fault detection of a host must be correctly handled: the design should guarantee a consensus between all other hosts before deciding that another one has failed (for the purpose of mutual exclusion), but this consensus can take time. The solution proposed here allows the delivery of ongoing streams without perturbation (handling parity blocks does not depend on mutual exclusion and thus on host fault detection), independently of the time needed to detect reliably an host failure.
References C. Bernhardt and E. Biersack. High-Speed Networking for Multimedia Applications, chapter The Server Array: A Scalable Video Server Architecture. Kluwer, 1996. [BB+ 96b] William J. Bolosky, Joseph S. Barrera, et al. The Tiger video fileserver. In Proceedings of the Sixth International Workshop on Network and Operating System Support for Digital Audio and Video. IEEE Computer Society, April 1996. [BP00] Alice Bonhomme and Loic Prylli. A distributed storage system for a videoon-demand server. Technical Report 2000-16, LIP, ENS Lyon, France, April 2000. [GLP98] L. Golubchik, J. Lui, and M. Papadopouli. A survey of approaches to fault tolerant design of VOD servers: Techniques, analysis and comparison. Parallel Computing, 24(1):123–155, 1998. [GM98] S. Ghandeharizadeh and R. Muntz. Design and implementation of scalable continuous media servers. Parallel Computing, 24:91–122, 1998. [GVK+ 95] D.J. Gemmel, H.M. Vin, D.D. Kandlur, P.V. Rangan, and L. Rowe. Multimedia storage servers: A tutorial and survey. IEEE Computer, 28(5):40– 49, November 1995. [GZS+ 97] S. Ghandeharizadeh, R. Zimmermann, W. Shi, R. Rejaie, D. Ierardi, and A.W Li. Mitra : A scalable continuous media server. Multimedia Tools and Applications Journal, 5(1):79–108, July 1997. [MSB97] R. Muntz, J.R Santos, and S. Berson. RIO: A real-time multimedia object server. ACM Performance Evaluation Review, 25(2):29–35, September 1997. [TMDV96] R. Tewari, R. Mukherjee, D.M. Dias, and H.M. Vin. Design and performance tradeoffs in clustered video servers. In the IEEE international Conference on Multimedia Computing and Systems (ICMCS’96), pages 144–150, May 1996. [BB96a]
Topic 18 Cluster Computing Rajkumar Buyya, Mark Baker, Daniel C. Hyde, and Djamshid Tavangarian Topic Chairmen
Recent Advances in Cluster Computing Cluster computing can be described as a fusion of the fields of parallel, highperformance, distributed, and high-availability computing. It has become a popular topic of research among the academic and industrial communities, including system designers, network developers, algorithm developers, as well as faculty and graduate researchers. The use of clusters as a application platform is not just limited to the scientific and engineering area; there are many business applications, including E-commerce, that are benefiting from the use of clusters. There are many exciting areas of development in cluster computing. These include new ideas as well as hybrids of old ones that are being deployed in production and research systems. There are attempts to couple multiple clusters, either located within one organisation or situated across multiple organisations forming what is known as a federated clusters or hyperclusters. The exploitation of federated clusters (clusters of clusters) as an infrastructure can seem to be approaching the area of the increasingly popular GRID infrastructure. The concept of portals that offers web-based access to applications running on clusters is becoming widely accepted. Such computing portals, offering access to scientific applications online, are known as scientific portals. PAPIA (Parallel Protein Information Analysis system) developed by the Japanese Real World Computing Partnership (RWCP) is an example of one. PAPIA allows scientists to have online access to a Protein Data Bank (PDB) in order to perform protein analysis on clusters. Anyone can perform the analysis of protein molecules and genetic DNA sequence from anywhere, at any time using any platform. In the future, we will see many such applications that exploit clustering and the Internet technologies for scientific discovery. The TopClusters project is TFCC collaboration with the TOP500 team. Plans are underway to build a database to record the performances of the most powerful cluster systems in different areas. These is great interest in this area as clusters are being used as platforms to host a diverse range applications, including scientific computing, web serving, database, and mission critical systems. Each of these areas has its own specific requirements. For example scientific applications are driven by floating-point performance whereas database applications are driven primarily by system I/O performance. The TopClusters project will measure these parameters (Mflop/s, I/O, TPC, MTBF, etc.) and use this to try and understand more fully the key aspects that need to addressed when building clusters for new and emerging applications. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1115–1117, 2000. c Springer-Verlag Berlin Heidelberg 2000
1116
Rajkumar Buyya et al.
The IEEE Task Force on Cluster Computing (TFCC ) is acting as a focal point and guide to the current cluster computing community. The TFCC has been actively promoting the field of cluster computing with the aid of a number of novel projects, for example we have an educational activity that has a book donation programme, holds forums for informal discussion, helps guide R&D work both in academic and industrial settings through workshops, symposiums and conferences. The recent developments in high-speed networking, middleware and resource management technologies have pushed clusters into the mainstream as general purpose computing system. This is clearly evident from the use of clusters as a computing platform for solving problems in number of disciplines. It also raises a number of challenges that cluster systems need to address with respect to their ability to support for example: – – – – –
System architecture. Heterogeneity. Single system image. System scalability. Resource management.
– – – – –
System administration. Performance. Reliability. Application scalability. Management and administration of hyperclusters.
Based on the aspects allready mentioned, a number of challenges listed above are among the issues addressed by the 12 research papers that we have selected from 27 contributions for the Euro-Par 2000 Cluster Computing Workshop. The program of the Workshop presents articles which demonstrate both theoretical and practical results of research works and new developments regarding cluster computing. The following papers have been accepted for presentation and discussion in the workshop, they cover: – F. Rauch, C. Kurmann and T. M. Stricker presents an analytical model that guides an implementation towards an optimal configuration for any given PC cluster. The model is validated by measurements on a cluster using Gigabitand Fast Ethernet links. – W. Hu, F. Zhang, and H. Liu describe topics belonging to the SMP and DSMP topics. They introduce an SMP protocol for the home-based software DSM system JIAJIA. – R. Cunniffe and B. A. Coghlan proposes a framework for cluster management which enables a cluster to be more efficiently utilized within a research environment. – V. Shurbanov, D. Avresky, P. Mehra and W. Watson describe in their paper the performance implications of several end-to-end flow-control schemes clusters based on the ServerNet system-area network. – The authors of the next paper H. Pedroso and J. G. Silva introduce the system WMPI as the first implementation of the MPI standard for Windows based machines. – The goal of the paper of F. Solsona, F. Gin´e, P. Hern´ andez and E. Luque is to build a NOW that runs parallel applications with performance equivalent
Topic 18: Cluster Computing
–
– –
– –
–
1117
to a MPP as well as executing sequential tasks as a dedicated uniprocessor with acceptable performance. Z. Juhasz and L. Kesmarki investigates the possible use of Jini technology for building Java-based metacomputing environments and gives an overview of Jini and highlights those features that can be used effectively for metacompting. The implementation of a skeleton library allowing the C programmer to write parallel programs using skeleton abstractions to structure and exploit parallelism is given by M. Danelutto and M. Stigliani. The contributions of the paper of C. Wagner and F. Mueller are twofold. First, a protocol for distributed mutual exclusion is introduced using a tokenbased decentralized approach, which allows either multiple concurrent readers or a single writer to enter their critical sections. Second, this protocol is evaluated in comparison with another protocol that uses a static structure instead of dynamic path compression. W. Schreiner, C. Mittermaier and F. Winkler describe a parallel solution to the problem of reliably plotting a plane algebraic curve based on Distributed Maple, a distributed programming extension written in Java. An Application Programming Interface (PCI-DDC) is described by E. Renault, P. David and P. Feautrier which provides different levels of integration in the kernel depending on the security and the performances expected by the administrator. A Clustering Approach for Improving Network Performance in Heterogeneous Systems is the topic of the presentation of V. Arnau, J. M. Ordu˜ na, S. Moreno, R. Valero and A. Ruiz. They propose on one hand a clustering algorithm that, given a network topology, provides a network partition adapted to the communication requirements of the set of applications running on the machine. On other hand, they propose a criterion to measure the quality of each one of the possible mappings of processes to processors that the provided network partition may generate.
All in all, we think we have an interesting program for this scientific event on cluster computing. We are grateful to the authors for their contributions and the reviewers who provided many useful comments in a very short time. We are also grateful to the organizers of the Euro-Par Conference 2000 for their helpful support regarding the organization of the Workshop.
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets in PC Clusters Felix Rauch, Christian Kurmann, and Thomas M. Stricker Laboratory for Computer Systems ETH - Swiss Institute of Technology CH-8092 Z¨urich, Switzerland {rauch,kurmann,tomstr}@inf.ethz.ch
Abstract. Multicasting large amounts of data efficiently to all nodes of a PC cluster is an important operation. In the form of a partition cast it can be used to replicate entire software installations by cloning. Optimizing a partition cast for a given cluster of PCs reveals some interesting architectural tradeoffs, since the fastest solution does not only depend on the network speed and topology, but remains highly sensitive to other resources like the disk speed, the memory system performance and the processing power in the participating nodes. We present an analytical model that guides an implementation towards an optimal configuration for any given PC cluster. The model is validated by measurements on our cluster using Gigabit- and Fast Ethernet links. The resulting simple software tool, Dolly, can replicate an entire 2 GByte Windows NT image onto 24 machines in less than 5 minutes.
1 Introduction and Related Work The work on partition cast was motivated by our work with the Patagonia multi-purpose PC cluster. This cluster can be used for different tasks by booting different system installations [12]. The usage modes comprise traditional scientific computing workloads (Linux), research experiments in distributed data processing (data-mining) or distributed collaborative work (Linux and Windows NT) and computer science education (Windows NT, Oberon). For best flexibility and maintenance, such a multi use cluster must support the installation of new operating system images within minutes. The problem of copying entire partitions over a fast network leads to some interesting tradeoffs in the overall design of a PC cluster architecture. Our cluster nodes are built from advanced components such as fast microprocessors, disk drives and high speed network interfaces connected via a scalable switching fabric. Yet it is not obvious which arrangement of the network or which configuration of the software results in the fastest system to distribute large blocks of data to all the machines of the cluster. After in depth analytical modelling of network and cluster nodes, we create a simple, operating system independent tool that distributes raw disk partitions. The tool can be used to clone any operating system. Most operating systems can perform automatic installation and customization at startup and a cloned partition image can therefore be used immediately after a partition cast completes. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1118–1131, 2000. c Springer-Verlag Berlin Heidelberg 2000
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1119
For experimental verification of our approach we use a meta cluster at ETH that unites several PC clusters, connecting their interconnects to a dedicated cluster backbone. This cluster testbed offers a variety of topologies and networking speeds. The networks include some Gigabit networking technology like SCI [7, 5], Myrinet [3] with an emphasis on Fast and Gigabit Ethernet [13]. The evaluation work was performed on the Patagonia sub-cluster of 24 Dell 410 Desktop PCs configured as workstations with keyboards and monitors. The Intel based PC nodes are built around a dual Pentium II processor configuration (running at 400 MHz) and 256 MB SDRAM memory connected to a 100 MHz front side bus. All machines are equipped with 9 GB Ultra2 Cheetah SCSI harddisk drives which can read and write a data stream with more than 20 MByte/s. Partition cloning is similar to general backup and restore operations. The differences between logical and physical backup are examined in [8]. We wanted our tool to remain operating system and file system independent and therefore we work with raw disk partitions ignoring their filesystems and their content. Another previous study of software distribution [9] presents a protocol and a tool to distribute data to a large number of machines while putting a minimal load on the network (i.e. executing in the background). The described tool uses unicast, multicast and broadcast protocols depending on the capabilities and the location of the receivers. The different protocols drastically reduce the network usage of the tool, but also prevent the multicast from reaching near maximal speeds. Pushing the protocols for reliable multicast over unreliable physical network towards higher speeds leads to a great variation in the perceived bandwidth, even with moderate packet loss rates, as shown in [11]. Known solutions for reliable multicast (such as [6]) require flow-control and retransmission protocols to be implemented in the application. Most of the multicast protocol work is geared to distribute audio and video streams with low delay and jitter rather than to optimize bulk data transfers at a high burst rate. The model for partition cast is based on similar ideas presented in the throughputoriented copy-transfer model for MPP computers [14]. A few commercial products are available for operating system installation by cloning, such as Norton Ghost 1 , ImageCast 2 or DriveImagePro 3 . All these tools are capable of replicating a whole disk or individual partitions and generating compressed image files, but none of them can adapt to different networks or the different performance characteristics of the computers in PC clusters. Commercial tools also depend on the operating- and the file system, since they use knowledge of the installed operating- and file systems to provide additional services such as resizing partitions, installing individual software packages and performing customizations. An operating system independent open source approach is desired to support partition cast for maintenance in Beowulf installations [2]. Other applications of our tool could include presentation-, database- or screen image cast for new applications in distributed data mining, collaborative work or remote tutoring on clusters of PCs. An early 1 2 3
c Symantec, http://www.symantec.com/ Norton Ghost, c Innovative Software Ltd., http://www.innovativesoftware.com/ ImageCast, c PowerQuest, http://www.powerquest.com/ DriveImagePro,
1120
Felix Rauch, Christian Kurmann, and Thomas M. Stricker
survey about research in that area including video-cast for clusters of PCs was done in the Tiger project [4].
2 A Model for Partition-Cast in Clusters In this Chapter we present a modelling scheme that allows to find the most efficient logical topology to distribute data streams. 2.1 Node Types We divide the nodes of a system into two categories, active nodes which duplicate a data stream and passive nodes which can only route data streams. The two node types are shown in Figure 1. Active Node A node which is able to duplicate a data stream is called an active node. Active nodes that participate in the partition cast store the received data stream on the local disk. An active node has at least an in-degree of 1 and is capable of passing the data stream further to one or more nodes (out-degree) by acting as a T-pipe. Passive Node A passive node is a node in the physical network that can neither duplicate nor store a copy of the data stream. Passive nodes can pass one or more streams between active nodes in the network.
Fig. 1. An active node (left) with an in-degree of 1 and an out-degree of 2 as well as a passive node (right) with an in- and out-degree of 3. Partition cast requires reliable data streams with flow control. Gigabit Ethernet switches do only provide unreliable multicast facilities and must therefore be modelled as passive switches that do only route TCP/IP point-to-point connections. Incorporating intelligent network switches or genuine broadcast media (like Coax Ethernet or Hubs) could be achieved by making them active nodes and modelling them at the logical level. This is only an option for expensive Gigabit ATM switches that feature multicast capability on logical channels with separate flow control or for simple switches that are enhanced by a special end-to-end multicast protocol that makes multicast data transfers reliable.
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1121
2.2 Network Types The different subsystems involved in a partition-cast must be specialized to transfer long data streams rather than short messages. Partitions are fairly large entities and our model is therefore purely bandwidth-oriented. We start our modelling process by investigating the topology of the physical network and taking a note of all the installed link and switch capacities. Physical Network The physical network topology is a graph given by the cables, nodes and switches installed. The vertices are labeled by the maximal switching capacity of a node, the edges by the maximal link speeds. The model itself captures a wide variety of networks including hierarchical topologies with multiple switches. Figure 2 shows the physical topology of the meta cluster installed at ETH and the topology of our simple sub-cluster testbed. The sub-cluster testbed is built with a single central Gigabit Ethernet switch with full duplex point-topoint links to all the nodes. The switch has also enough Fast Ethernet ports to accommodate all cluster nodes at the low speed. Clusters of PCs are normally built with simple and fast layer-2 switches like our Cabletron Smart Switch Routers. In our case the backplane capacity for a 24 port switch is at 4 GByte/s and never results in a bottleneck.
COPS 16 Nodes
Patagonia 8 Nodes
...
...
...
...
...
Fast Ethernet Gigabit Ethernet
Cabletron Cabletron SSR 8600 SSR 9000
Linneus 16 Nodes
COPS Cluster 16 Nodes
Cabletron Cabletron SSR 8000 SSR 8600
Math./Phys. Beowulf 192 Nodes
Fig. 2. Physical network topologies of the ETH Meta-Cluster (left) and the simple subcluster with one central switch (right). Our goal is to combine several subsystems of the participating machines in the most efficient way for an optimal partition-cast, so that the cloning of operating system images can be completed as quickly as possible. We therefore define different setups of logical networks. Logical Network The logical network represents a connection scheme, that is embedded into a physical network. A spanning tree of TCP/IP connections routes the stream of a partition cast to all participating nodes. Unlike the physical network, the logical network must provide reliable transport and flow control over its channels.
1122
Felix Rauch, Christian Kurmann, and Thomas M. Stricker S S
S
S
S
S
Fig. 3. Logical network topologies (top) describing logical channels (star, n-ary spanning tree, multi-drop-chain) and their embedding in the physical networks. Star A logical network with one central server, that establishes a separate logical channel to all n other nodes. This logical network suffers heavy congestion on the outgoing link of the server. n-ary Spanning Tree Eliminates the server-bottleneck by using an n-ary spanning tree topology spanning all nodes to be cloned. This approach requires active T-nodes which receive the data, store it to disk and pass it further to up to n next nodes in the tree. Multi-Drop-Chain A degenerated, specialized tree (unary case) where each active node stores a copy of the stream to disk and passes the data to just one further node. The chain is spanning all nodes to be cloned. Figure 3 shows the above described topologies as well as their embedding in the physical networks. We assume that the central switch is a passive node and that it cannot duplicate a partition cast stream. 2.3 Capacity Model Our model for maximal throughput is based on capacity constraints expressed through a number of inequalities. These inequalities exist for active nodes, passive nodes and links, i.e. the edges in the physical net. As the bandwidth will be the limiting factor, all subsystems can be characterized by the maximal bandwidth they achieve in an isolated transfer. The extended model further introduces some more constraints e.g. for the CPU and the memory system bandwidth in a node (see Section 2.5). Reliable transfer premise We are looking for the fastest possible bandwidth with which we can stream data to a given number of active nodes. Since there is flow control, we know that the bandwidth b of the stream is the same in the whole system. Fair sharing of links We assume that the flow control protocol eventually leads to a stable system and that the links or the nodes, dealing with the stream, allocate the bandwidth evenly and at a precise fraction of the capacity.
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1123
Both assumptions hold in the basic model and will be slightly extended in the refined model that can capture raw and compressed streams at different rates simultaneously. Edge Capacity defines a maximum streaming capacity for each physical link and logical channel (see Figure 4). As the physical links normally operate in full duplex mode, the inbound- and outbound-channels can be treated separately. If the logical-to-physical mapping suggests more than one logical channel over a single physical link, its capacity is evenly shared between them. Therefore the capacity is split in equal parts by dividing the link capacity through the number of channels that are mapped to the same physical link. Example: For a binary tree with in-degree 1 and out-degree 2 mapped to one physical Gigabit Ethernet link the bandwidth of a stream has to comply with the following edge inequality: 125 (1) 1 E2 : b < 125, 2b < 125 → b < 2
Passive Node Switching Capacity 4 GByte/s
Physical Link 125 MByte/s
2 Logical Channels b < 62.5 MByte/s each
4 Streams b < 256 MByte/s each
Active Node Switching Capacity b < 30 MByte/s
3 Streams b < 10 MByte/s each
Fig. 4. Edge capacities exist for the physical and logical network, node capacities for each in- and out-stream of a node.
Node Capacity is given by a switching capacity a node can provide, divided through the number of streams it handles. The switching capacity of a node can be measured experimentally (by parameter fitting) or be derived directly from data of the node computer through a detailed model of critical resources. The experimental approach provides a specific limit value for each type of active node in the network, i.e. the maximal switching capacity. Fitting all our measurements resulted in a total switching capacity of 30 MB/s for our active nodes running on a 400 MHz Pentium II based cluster node. The switching capacity of our passive node, the 24 port Gigabit Ethernet switch is about 4 GByte/s - much higher than needed for a partition cast. 2.4 Model Algorithm With the model described above we are now able to evaluate the different logical network alternatives described earlier in this Section of the paper. The algorithm for evaluation of the model includes the following steps:
1124
Felix Rauch, Christian Kurmann, and Thomas M. Stricker
algorithm basicmodel 1 chose the physical network topology 2 chose the logical network topology 3 determine the mapping and the edge congestions 4 for all edges determine in-degree, out-degree of nodes attached to edge evaluate channel capacity (according to logical net) 5 for all nodes determine in-degree, out-degree and disk transfer of the node evaluate node capacity 6 solve system of inequalities and find global minimum 7 return minimum as achievable throughput Example: We compare a multi-drop-chain vs. the n-ary spanning tree structure for Gigabit Ethernet as well as for Fast Ethernet. The chain topology with all active nodes with in-degree ι and out-degree ω of exactly one (except for the source and the sink) can be considered as a special case of an unary tree (or hamiltonian path) spanning all the active nodes receiving the partition cast. – Topology: We evaluate the logical n-ary tree topology of Figure 3 with 5 nodes (and a streaming server) mapped on our simple physical network with a central switch of Figure 2. The out-degree shall be variable from 1 to 5, multi-drop-chain to star. – Edge Capacity: The in-degree is always 1. The out-degree over one physical link varies between 1 for the multi-drop-chain and 5 for the star which leads to the following inequality: 1 Eo
: ob < 125 → b <
125 o
for Gigabit Ethernet
(2)
12.5 for Fast Ethernet (3) o – Node Capacity N: For the active node we take the evaluated capacity of 30 MByte/s with the given in-degree and out-degree and a disk write: 1 Eo
: ob < 12.5 → b <
N1,o,1 : (1 + o + 1)b < 30 → b <
30 (1 + o + 1)
(4)
We now label all connections of the logical network with the maximal capacities and run the algorithm to find a global minimum of achievable throughput. The evaluation of the global minimum indicates that for Gigabit Ethernet the switching capacity of the active node is the bottleneck for the multi-drop-chain and for the n-ary trees. But for the slower links of a Fast Ethernet the n-ary tree the network rapidly becomes a bottleneck as we move to higher branching factors. Section 4 gives a detailed comparison of modelled and measured values for all cases considered. 2.5 A More Detailed Model for an Active Node The basic model considered two different resources: Link capacity and switching capacity. The link speeds and the switch capacity of the passive node were taken from the
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1125
physical data sheets of the networking equipment, while the total switching capacity of an active node was obtained from measurements by a parameter fit. Link and switching capacity can only lead the optimization towards a graph theoretical discussion and will only be relevant to cases that have extremely low link bandwidth and high processing power or to systems that are trivially limited by disk speed. For clusters of PCs with high speed interconnects this is normally not the case and the situation is much more complex. Moving data through I/O buses and memory systems at full Gigabit/s speed remains a major challenge in cluster computing. The systems are nearly balanced between CPU performance, memory system and communication speed and some interesting tradeoffs can be observed. As indicated before, several options exist to trade off the processing power in the active node against a reduction of the load on the network. Among them are data compression or advanced protocol processing that turns some unreliable broadcast capabilies of Ethernet switches into a reliable multicast. For a better model of an active node we consider the data streams within an active node and evaluate several resource constraints. For a typical client the node activity comprises receiving data from the network and writing partition images to the disk. We assume a “one copy” TCP/IP protocol stack as provided by standard Linux. In addition to the source and sink nodes the tree and multi-drop chain topologies require active nodes that store a data stream and forward one or more copies of the data streams back into the network. Figure 5 gives a schematic data flow in an active node capable of duplicating a data stream.
DMA SCSI
System buffer T-connector
Copy
Copy
System buffer
DMA
Copy
System buffer
DMA
User buffer
Network
Fig. 5. Schematic data flow of an active node running the Dolly client.
2.6 Modelling the Limiting Resources in an Active Node The switching capacity of an active node is modelled by the two limits of the network and four additional resource limits within the active node. Link capacity as taken from the physical specifications of the network technology (125 MB/s (Gigabit Ethernet) or 12.5 MB/s (Fast Ethernet) on current systems). Switch capacity of passive nodes as taken from the physical specifications of the network hardware (2 or 4 GB/s depending on the Cabletron Smart Switch Router model, 8000 or 8600). Disk system similar to a link capacity in the basic model (24 MB/s for a Seagate Cheetah 10’000 RPM disk). I/O bus capacity the sum of data streams traversing the I/O bus must be less than its capacity (132 MB/s on current, 32 bit PCI bus based PC cluster nodes).
1126
Felix Rauch, Christian Kurmann, and Thomas M. Stricker
Memory system capacity the sum of the data streams to and from the memory system must be less than the memory system capacity (180 MB/s on current systems with the Intel 440 BX chipset). CPU Utilization the processing power required for the data streams at the different stages. For each operation fraction coefficient 1/a1, 1/a2 , 1/a3 , . . . indicates the maximal speed of the stream with exclusive use of 100% CPU. The sum of the fractions of CPU use must be < 1(= 100%) (Fractions considered: 80 MB/s SCSI transfer, 90 MB/s internal copy memory to memory, 60 MB/s send or receive over Gigabit Ethernet, 10 MB/s to decompress a data stream for a current 400 MHz single CPU cluster node). Limitations on the four latter resources result in constraint inequalities for the maximal throughput achievable through an active node. The modelling algorithm determines and posts all constraining limits in the same manner as described in the example with a single switching capacity. The constraint over the edges of a logical network can be evaluated then into the maximum achievable throughput considering all limiting resources. 2.7 Dealing with Compressed Images Partition images or multimedia presentations can be stored and distributed in compressed form. This reduces network load but puts an additional burden on the CPUs in the active nodes. Compressing and uncompressing is introduced into the model by an additional data copy to a gunzip process, which uncompresses data with an output data rate of about 10 MByte/s (see Figure 6).
DMA SCSI
System buffer T-connector
Copy Uncomp
gunzip
Copy
Copy
System buffer
DMA
Copy
System buffer
DMA
User buffer
Network
Fig. 6. Schematic data flow of a Dolly client with data decompression. The workload is defined in raw data bytes to be distributed and the throughput rates are calculated in terms of the uncompressed data stream. For constraints inequalities involving the compressed data stream the throughput must be adjusted by the compression factor c. Hardware supported multicast could be modeled in a similar manner. For multicast, the network would be enhanced by newly introduced active switches, but a reliable multicast flow control protocol module must be added at the enpoints and would consume a certain amount of CPU performance and memory system capacity (just like a compression module). Example: Modelling the switching capacity of an active node for a binary-spanning tree with fast Ethernet and compression.
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1127
From the flow chart (similar to Figure 6 but with an additional second output stream from the user buffer to the network) we see two network sends, one network receive, one disk write, four crossings of the I/O bus, eleven streams from and to buffer memory, one compression module and five internal copies of the data stream. This leads to the following constraints for the maximal achievable throughput b: b c 2b c
< 12.5 MB/s < 12.5 MB/s b < 24 MB/s 3b c + b < 132 MB/s 8b c + 3b < 180 MB/s 3 1 4 1 ( 45c + 80 + 90c + 90 + 9c )b < 1 (100%)
link for receive link for send SCSI Disk i/o bus, PCI memory system CPU utilization
For a compression factor of c = 2, an active node in this configuration can handle 5.25 MB/s. The limiting resource is the CPU utilization.
3 Differences in the Implementations Our first approach to partition-cast was to use a simple file sharing service like NFS4 with transfers over a UDP/IP network resulting in a star topology. The NFS server exports partition images to all the clients in the cluster. A command line script reads the images from a network mounted drive, possibly decompresses the data and writes it to the raw partitions of the local disk. Because of the asymmetric role of one server and many clients, this approach does not scale well, since the single high speed Gigabit Ethernet can be saturated even serving a small number of clients (see performance numbers in Section 4). Although this approach might look a bit naive to an experienced system architect, it is simple, highly robust and supported by every operating system. A single failing client or a congested network can be easily dealt with. As a second setup, we considered putting together active-clients in a n-ary spanning tree topology. This method works with the standard TCP point-to-point connections and uses the excellent switching capability of the Gigabit Ethernet switch backplane. A partition cloning program (called Dolly) runs on each active node. A simple server program reads the data from disk on the image server and sends the stream over a TCP connection to the first few clients. The clients receive the stream, write the data to the local disk and send it on to the next clients in the tree. The machines are connected in an n-ary spanning tree, eliminating the bottleneck of the server link accessing the network. Finally for the third optimal solution the same Dolly client program can be used with a local copy to disk and just one further client to serve. The topology turns into a highly degraded unary spanning tree. We call this logical network a multi-drop chain. An obvious topological alternative would be a true physical spanning tree using the multicasting feature of the networking hardware. With this option the server would only source one stream and the clients would only sink a single stream. The protocols and schemes required for reliable and robust multicast are neither trivial to implement 4
Network File System
1128
Felix Rauch, Christian Kurmann, and Thomas M. Stricker
nor included in common commodity operating systems and often depend on the multicast capabilities of the network hardware. In a previous study [11] one of the authors implemented several well known approaches ([6, 10]). Unfortunately the performance reached in those implementation was not high enough to make an application to the partition cloning tool worthwhile.
4 Evaluation of Partition-Cast In this section we provide measurements of partition casts in different logical topologies (as described in Section 2) with compressed and uncompressed partition images. The partition to be distributed to the target machines is a 2 GByte Windows NT partition. The compressed image file is about 1 GByte in size, resulting in a compression factor of 2. Star (NFS)
Execution Time [s]
1200
800
✩ ✩
600
❍
200
Multidrop Chain (Dolly Client)
1400
✩
1200
● ●
1000
400
3-Tree (DollyClient)
2500
★
★ ✩ ✩❍ ❍ ★
Execution Time [s]
1400
1000 800
❍
✩ ✩ ❍
★ ★ ✩ ✩❍ ❍ ★ ●●
✩ ❍ ●
★ ✩ ❍
★ ✩ ❍
●
●
600 ✩ ❍ ✩❍ ● ★ ●★
✩ ❍ ★ ●
✩ ❍ ★ ●
✩ ❍ ★ ●
400 200 0
Number of Nodes
Number of Nodes
15
10
5
1 2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
15
10
5
1 2
0 Number of Nodes
✩ Fast Ethernet compressed ★ Fast Ethernet raw ❍ Gigabit Ethernet compressed ● Gigabit Ethernet raw
Fig. 7. Total execution times for distributing a 2 GByte Windows NT Operating System partition simultaneously to 1, 2, 5, 10, 15 machines by partition cloning with NFS based star topology, the Dolly based 3-tree and multi-drop-chain topologies on the Patagonia cluster. The star topology run with 10 clients using raw transfers over Fast Ethernet resulted in execution times around 2500 seconds as the NFS servers disk started thrashing. A first version of our partition-cast tool uses only existing OS services and therefore applies a simplistic star topology approach. It consists of a NFS server which exported the partition images to all the clients. The clients accessed the images over the network using NFS, possibly uncompressing the images and finally writing the data to the target partition. The results of this experiment are shown on the left side of Figure 7 (the execution time for each client machine is logged to show the variability due to congestion). The Figure shows two essential results: (1) The more clients need to receive the data, the more time the distribution takes (resulting in a lower total bandwidth of the system). The bandwidth is limited by the edge capacity of the server. (2) Compression helps to increase the bandwidth for a star topology. As the edge capacity is the limiting factor, the nodes have enough CPU, memory and IO capacity left to uncompress the incoming stream at full speed, thereby increasing the total bandwidth of the channel.
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1129
180
●
160 140 100
★
Fast Ethernet multi-drop/raw
❍
Gigabit Ethernet tree/raw
✩
✩
Fast Ethernet tree/raw
● ★
●
Gigabit Ethernet star/compressed
★
Fast Ethernet star/compressed
● ❍ ★
60 ● ✩ ★ 5
✩ ● ★
● ★
20
● ❍ ★ ❍ ● ●✩ ★ ★ ✩ ● ★
10
❍ ● ★
40
✩
15
80
0
Gigabit Ethernet multi-drop/raw
● ❍ ★
120
20
● ★
1
Aggregate Bandwidth [MByte/s]
A second approach is to use an n-ary spanning tree structure. This topology was implemented in the small program Dolly which acts as an active node. The program reads the partition image on the server and sends it to the first n clients. The clients write the incoming data to the target partition on disk and send the data to the next clients. The out-degree is the same for all nodes (if there are enough successor-nodes) and can be specified at runtime. The results for a 3-ary tree are shown in Figure 7 in the middle. For Fast Ethernet the execution time increases rapidly for a small number of clients until the number of clients (and therefore the number of successor-nodes of the server) reaches the out-degree. As soon as the number of clients is larger than the out-degree, the execution times stay roughly constant. For this network speed, the edge capacity is again the bottleneck, resulting in increasing execution times for higher outdegree. In the case of Gigabit Ethernet, the link speed (the edge capacity) is high enough to satisfy an out-degree of up to 5 without the edge capacity becoming the bottleneck. The bottleneck in this case is still the nodes memory capacity.
Number of Nodes
Fig. 8. Total (aggregate) transfer bandwidth achieved in distributing a 2 GByte Windows NT Operating System partition simultaneously to a number of hosts by partition cloning in the Patagonia cluster. For the third experiment we use Dolly to cast the partition using a multi-drop chain. The results are shown in Figure 7 on the right. They indicate that the execution time for this partition cast is nearly independent on the number of clients. This independence follows from the fact that with the multi-drop chain configuration, the edge capacity is no longer a bottleneck as every edge carries at most one stream. The new bottleneck is the nodes’ memory system. The memory bottleneck also explains why compression results in a lower bandwidth for the channel (decompressing data requires more memory copy operations in the pipes to the gunzip process). Figure 8 shows the total, aggregate bandwidth of data transfers to all disk drives in the system with the three experiments. The figure indicates that the aggregate bandwidth of the NFS-approach increases only modestly with the number of clients while the multi-drop chain scales perfectly. The 3-ary-tree approach also scales perfectly, but increases at a lower rate. The numbers for the NFS approach clearly max out with the transfer bandwidth of the servers network interface reaching the edge capacity: Our NFS-server can deliver a maximum of about 20 MByte/s over Gigabit Ethernet and
1130
Felix Rauch, Christian Kurmann, and Thomas M. Stricker
about 10 MByte/s over Fast Ethernet (note that we are using compressed data for the NFS approach in the above figure thereby doubling the bandwidth). The predicted bandwidths are compared with measured values in our cluster in Table 1. Network
Fast Ethernet Bandwidth Gigabit Ethernet Bandwidth Out- Com- Extended Extended Topology Degree pression Model Measured Model Measured Multi-Drop-Chain 1 no 11.1 8.8 11.1 9.0 Multi-Drop-Chain 1 yes 6.1 4.9 6.1 6.1 2-Tree 2 no 6.3 5.4 8.1 8.2 3-Tree 3 no 4.2 3.8 6.4 8.0 Star 5 no 2.5 2.3 5.3 6.3 Star 5 yes 5.0 3.6 5.0 4.1
Table 1. Predicted and measured bandwidths for a partition cast over a logical chain and different tree topologies for uncompressed and compressed images. All values are given in MByte/s.
5 Conclusion In this paper we investigated the problem of a partition-cast in clusters of PCs. We showed that optimizing a partition-cast or any distribution of a large block of raw data leads to some interesting tradeoffs between network parameters and node parameters. In a simple analytical model we captured the network parameters (link speed and topology) as well as the basic processor resources (memory system, CPU, I/O bus bandwidth) at the intermediate nodes that are forwarding our multicast streams. The calculation of the model for our sample PC cluster pointed towards an optimal solution using uncompressed streams of raw data, forwarded along a linear multi-drop chain embedded into the Gigabit Ethernet. The optimal configuration was limited by the CPU performance in the nodes and its performance was correctly predicted at about one third of the maximal disk speed. The alternative of a star topology with one server and 24 clients suffered from heavy link congestion at the server link, while the different n-ary spanning tree solutions were slower due to the resource limitations in the intermediate nodes, that could not replicate the data into multiple streams efficiently enough. Compression resulted in a lower network utilization but was slower due to the higher CPU utilization. The existing protocols for reliable multicast on top of unreliable best-effort hardware broadcast in the Ethernet switch were not fast enough to keep up with our multi-drop solution using simple, reliable TCP/IP connections. The resulting partition casting tool is capable of transferring a 2 GByte Windows NT operating system installation to 24 workstations in less than 5 minutes while transferring data at a sustained rate of about 9 MByte/s per node. Fast partition cast permits the distribution of entire installations in a short time, adding flexibility to a cluster of PCs to do different tasks at different times. A setup for efficient multicast also results
Partition Cast — Modelling and Optimizing the Distribution of Large Data Sets
1131
in easier maintenance and enhances the robustness against slowly degrading software installations in a PC cluster.
References [1] Henri Bal. The Distributed ASCI Supercomputer (DAS). http://www.cs.vu.nl/∼bal/das.html. [2] D. J. Becker, T. Sterling, D. Savarese, J. E. Dorband, U. A. Ranawake, and C. V. Packer. Beowulf: A Parallel Workstation for Scientific Computation. In Proceedings of 1995 ICPP Workshop on Challenges for Parallel Processing, Oconomowc, Wisconsin, U.S.A., August 1995. CRC Press. [3] Nanette J. Boden, Robert E. Felderman, Alan E. Kulawik, Charles L. Seitz, Jakov N. Seizovic, and Wen-King Su. Myrinet — A Gigabit per Second Local Area Network. IEEEMicro, 15(1):29–36, February 1995. [4] William J. Bolosky, Joseph S. Barrera III, Richard P. Draves, Robert P. Fitzgerald, Garth A. Gibson, Michael B. Jones, Steven P. Levi, Nathan P. Myhrvold, and Richard F. Rashid. The Tiger Video Fileserver. In Sixth International Workshop on Network and Operating System Support for Digital Audio and Video, Zushi, Japan, April 1996. IEEE Computer Society. [5] Dolphin Interconnect Solutions. PCI SCI Cluster Adapter Specification, 1996. [6] S. Floyd, V. Jacobson, S. McCanne, L. Zhang, and C-G. Liu. A Reliable Multicast Framework For Lightweight Sessions and Application Level Framing. In Proceedings of ACM SIGCOMM ’95, pages 342–356, August 1995. [7] H. Hellwagner and A. Reinefeld, editors. SCI Based Cluster Computing. Springer, Berlin, Spring 1999. [8] Norman C. Hutchinson, Stephen Manley, Mike Federwisch, Guy Harris, Dave Hitz, Steven Kleiman, and Sean O Malley. Logical vs. Physical File System Backup. In Proceedings of the 3rd Symposium on Operating Systems Design and Implementation, New Orleans, Louisiana, pages 239–249. The USENIX Association, February 1999. [9] Steve Kotsopoulos and Jeremy Cooperstock. Why Use a Fishing Line When you Have a Net? An Adaptive Multicast Data Distribution Protocol. In Proccedings of the USENIX 1996 Annual Technical Conference, San Diego, California, January 1996. The USENIX Association. [10] Sanjoy Paul, Krishan K. Sabnani, and David M. Kristol. Multicast Transport protocols for High Speed Networks. In Proceedings of International Conference on Network Protocols, pages 4–14. IEEE Computer Society Press, 1994. [11] F. Rauch. Zuverl¨assiges Multicastprotokoll. Master’s thesis, ETH Z¨urich, 1997. English title: Reliable Multicast Protocol. See also http://www.cs.inf.ethz.ch/. Contains a survey about reliable IP multicast. [12] Felix Rauch, Christian Kurmann, Thomas Stricker, and Blanca Maria M¨uller. Patagonia — A Dual Use Cluster of PCs for Computation and Education. In 2. Workshop Cluster Computing, Karlsruhe, March 1999. [13] Rich Seifert. Gigabit Ethernet: Technology and Applications for High-Speed LANs. Addison-Wesley, May 1998. ISBN: 0201185539. [14] T. Stricker and T. Gross. Optimizing Memory System Performance for Communication in Parallel Computers. In Proc. 22nd Intl. Symposium on Computer Architecture, pages 308–319, Santa Marguerita di Ligure, June 1995. ACM.
A New Home-Based Software DSM Protocol for SMP Clusters Weiwu Hu, Fuxin Zhang, and Haiming Liu Institute of Computing Technology Chinese Academy of Sciences, Beijing 100080 [email protected]
Abstract. This paper introduces an SMP protocol for the home-based software DSM system JIAJIA. In the protocol, intra-node processes in an SMP node share their home pages through hardware coherent sharing so as to take the full advantage of the home effect of home-based software DSMs. In contrast, cached remote pages of a process are not shared by its intra-node partners to avoid cache page conflict within an SMP. Besides, JIAJIA also implements the shared memory communication among processes within the same SMP node to accelerate intra-node communication. Performance evaluation with some well accepted benchmarks and real applications in a cluster of four two-processor nodes shows that the SMP protocol of JIAJIA reduces remote accesses, diffs, and consequently message amounts in all of the ten benchmarks and as a result obtains noticeable performance improvement in seven.
1
Introduction
With the wide spread of symmetric multiprocessor (SMP) systems, cluster of SMPs has been emerging as an attractive parallel processing platform to provide high performance with certain connectivity and affordable costs. Given the convenient shared address programming model of SMPs, it is natural to extend this convenience to the SMP clusters with shared virtual memory. The shared virtual memory has the obvious advantages over the message passing alternative on SMP clusters in that it can take the advantage of the efficient hardware-based shared memory within an SMP. In a software DSM on cluster of SMPs, the SMP hardware transparently provides shared memory at the cache line granularity for intra-node sharing, while the software protocol is responsible for providing shared memory at the page granularity for inter-node sharing. Previous software DSMs on SMP clusters include the softFLASH system[3] which implements a single-writer protocol for sequential consistency, the Shasta system[12] which performs coherence through code instrumentation at cache-line granularity, the Cashmere-2L[13] which is based on the DEC Memory Channel
The work of this paper is supported by National Natural Science Foundation of China (Grant No. 69703002) and National High Technology (863) Program (Grant No. 863-306-ZD01-02-2).
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1132–1142, 2000. c Springer-Verlag Berlin Heidelberg 2000
A New Home-Based Software DSM Protocol for SMP Clusters
1133
network interface and network, the HLRC-SMP[11] which is an implementation of a lazy, home-based, multiple-writer protocol across SMP nodes and which uses the Virtual Memory Mapped Communication (VMMC-2) library for the Myrinet network, and the modified version of TreadMarks[7] which uses POSIX threads instead of processes to implement parallelism within a multiprocessor. This paper introduces design and implementation of a new software DSM protocol for SMP clusters. The protocol is designed and implemented on the home-based software DSM system JIAJIA[4] and hence has a similar goal to the HLRC-SMP protocol. The main difference between HLRC-SMP and the SMP protocol of JIAJIA is that processes in an SMP share both home and cached pages in HLRC-SMP, while only home pages are shared by intra-node processes in JIAJIA. Sharing home pages among intra-node processes in SMP clusters helps to take the full advantage of the home effect of home-based software DSMs, i.e., the ability to dispense with page faults for references made by the home node of a given page. Though sharing cached pages is helpful for all processes within a node to benefit from a page fetch performed by one, it also causes cache page conflict such as when a process is writing a cached page, another process in the same node may covers this page due to a page fetch. Another optimization JIAJIA makes for SMP clusters is the optimization of intra-node communication. With this optimization, processes within an SMP node communicate through shared memory. The effect of the SMP protocol is evaluated on a Myrinet cluster of four twoprocessor Ultra-2 nodes with ten benchmarks, include Water, Barnes, Ocean, and LU from SPLASH2, MG and 3DFFT from NAS Parallel Benchmarks, SOR, and TSP from TreadMarks benchmarks, and two real applications EM3D for magnetic field computation and IAP18 for climate simulation. Evaluation results show that the SMP optimization achieves a speedup of 4% − 5% in LU, SOR, and EM3D, about 10% in MG and 3DFFT, and more than 20% in Ocean and IAP18. The rest of this paper is organized as follows. The following Section 2 briefly introduces the JIAJIA software DSM system. Section 3 illustrates the SMP protocol of JIAJIA. Section 4 presents experiment results and analysis. The conclusion of this paper is drawn in Section 5.
2
The JIAJIA Software DSM System
In JIAJIA, each shared page has a home node and homes of shared pages are distributed across all nodes. References to home pages hit locally, references to non-home pages cause these pages to be fetched from their home and cached locally. A cached page may be in one of three states: Invalid (INV), Read-Only (RO), and Read-Write (RW). When the number of locally cached pages is larger than the maximum number allowed, some aged cache pages must be replaced to its home to make room for the new page. This allows JIAJIA to support shared memory that is larger than physical memory of one machine.
1134
Weiwu Hu, Fuxin Zhang, and Haiming Liu
JIAJIA implements the scope memory consistency model. Multiple writer technique is employed to reduce false sharing. In JIAJIA, the coherence of cached pages is maintained through requiring the lock-releasing (or barrier arriving) processor to send to the lock write notices about modified pages in the associated critical section, and the lock-acquiring (or barrier leaving) processor to invalidate cached pages that are notified as obsolete by the associated write-notices in the lock. This protocol maintains coherence through write notices kept on the lock and consequently eliminates the requirement of directory. The following optimizations[4,6] are made to the protocol. Single-Writer Detection. In the optimization, if a page is modified only by its home processor during a synchronization interval, then it is unnecessary to send the associated write notice to the lock manager at the end of the interval. If a page is modified only by one remote processor during an interval, then the processor who makes the modification need not invalidate the page on next acquire or barrier. Incarnation Number[8] Technique. With this optimization, each lock is associated with an incarnation number which is incremented when the lock is transferred. A processor records the current incarnation number of a lock on an acquire of the lock. When the processor acquires the lock again, it tells the lock manager its current incarnation number of the lock. With this knowledge, the lock manager knows which write notices have been sent to the acquiring processor on previous lock grants and excludes them from write notices sent back to the acquiring processor at this time. Lazy Home Page Write Detection. Normally, home pages are writeprotected at the beginning of a synchronization interval so that writes to home pages can be detected through page faults. The lazy home page write detection delays home page write-protecting until the page is first fetched in the interval so that home pages that are not cached by remote processors do not need to be write-protected. Write Vector Technique. The write vector optimization is motivated by the idea of fetching diffs only on a page fault in homeless protocols. It avoids fetching the whole page on a page fault through dividing a page into blocks and fetches only those blocks that are dirty with respect to the faulting processor on a page fault. A write vector table is maintained for each shared page in its home to record for each processor which block(s) has been modified since the processor fetched the page last time. Home Migration. The home migration optimization adaptively migrates home of a page to the processor most frequently writes to the page, to reduce diff overhead at the end of an interval because write to home pages does not produces twin and diff in home-based protocol. In the home migration scheme, pages that are written by only one processor between two barriers are recognized by the barrier manager and their homes are migrated to the single writing processor. Migration information is piggybacked on barrier messages and no additional communication is required for the migration.
A New Home-Based Software DSM Protocol for SMP Clusters
P1
P2
···
Pn−1
cache
···
cache
home
···
home
P1
P2
···
Pn−1
Pn
cache
cache
···
cache
cache
Pn
Interconnection Network (a) HLRC-SMP
home
···
1135
home
Interconnection Network (b) JIAJIA-SMP
Fig. 1. Memory Organization of HLRC-SMP and JIAJIA-SMP
3 3.1
SMP Protocol for JIAJIA Design Alternatives
Processors within an SMP can share data either through threads or through processes. The thread model provides an intuitive and efficient method of sharing data within an SMP. However, threads may cause some unexpected side effects. First, all threads within an SMP implicitly share all global variables, which makes it difficult to provide a uniform sharing memory between threads within a node and threads in different nodes. Second, all threads within an SMP have the same view of shared data, which means if a thread decides to invalidate a shared page, the page is invalidated for other threads as well. The alternative is to use processes within an SMP node to share memory. One approach is to use shmget() and shmat() system calls. The major disadvantage of this approach is that both the number of shared segments and segment size are limited by the system, preventing it from being used by JIAJIA which can support a large shared memory. Besides, the system overhead of shmget() increases linearly with the number of segments[2]. Another and more efficient approach of sharing memory among processes with an SMP is to rely on virtual memory management and the mmap() system call (anonymous mapping with MAP SHARED parameter). JIAJIA adopts this approach after comparing the portability, programmability, ability to support large memory, and implementation simplicity of all candidate alternatives. The disadvantage of this approach is that shared pages should be mapped before processes are forked in an SMP node. To meet this requirement, JIAJIA reserves a shared space before processes are forked in an SMP node at the initialization stage.
1136
3.2
Weiwu Hu, Fuxin Zhang, and Haiming Liu
SMP Protocol
Figure 1 shows the memory organization of HLRC-SMP and SMP protocol of JIAJIA. As can be seen from Figure 1, the main difference between HLRC-SMP and the SMP protocol of JIAJIA is that processes in an SMP share both home and cached pages in HLRC-SMP, while only home pages are shared by processes in an SMP in JIAJIA. Home-based software DSMs benefit from the home effect, the ability to dispense with page faults for references made by the home node of a given page. In SMP clusters, there are several processes running on an SMP node, each process is the home of part of the total shared pages. Sharing home pages among processes within a node combine their home pages together so that each process in the node “sees” a larger home. Sharing cached pages within an SMP seems not so attractive to JIAJIA. Though with shared cache a processor may benefit from a remote page fetch carried out by another processor in the same SMP node, shared cache pages cause sharing violation among processors within the same SMP. For example, when a process fetches a page from its home, it has to ensure that no other processes in the same node are currently writing to the page, otherwise the incoming page may overwrite these writes. Besides, to support shared space larger than the physical memory of a machine, a cached page is dynamically mapped and unmapped to the faulting address when the page is cached and flushed by a process in JIAJIA. The dynamic mapping and unmapping of cached pages violates the requirement that pages physically shared by intra-node processes through mmap() should be mapped before intra-node processes are forked. The decision of shared home and separated cache for intra-node processes greatly simplifies the SMP protocol of JIAJIA. Each processor in an SMP maintains cache coherence as if in cluster of single-processor nodes. Sharing home pages among intra-node processes is very simple and requires least modification to JIAJIA. Though all processes share their home pages, each process has its separate view (such as protect state) of a given home page and operates on the page as if it is the unique home host of the page. To service remote page fetch and diff request, one process is appointed as the real home host of a given page and a page fetch or diff request is always sent to the real home host of the faulting page. Real home hosts of shared pages in an SMP node are distributed across all processes within the node to avoid bottleneck problem. The process that regards a page as its home page, but does not service remote request about the page, are called co-home host of the page. Though the SMP protocol of JIAJIA is simplified in the absence of some intra-node processes cache sharing related complexity, non-trivial issues still arise when coordinating with some optimization methods of JIAJIA. In the lazy home page write detection, the home page write-protecting that are performed at the beginning of an interval is delayed until the page is first fetched in the interval so that home pages that are not cached by remote processors do not need to be write-protected. In the SMP protocol, a page fetch request is only sent to the real home host of the page, co-home hosts do not
A New Home-Based Software DSM Protocol for SMP Clusters
1137
know when the page is first fetched at an interval. Therefor, the lazy home page write detection technique is applied only to the real home host of a given page, and the co-home hosts of a page write-protect the page at the beginning of each interval as normal. With the observation that only the process that modifies a page needs to detect the write, the real home host of a given page can be dynamically shifted to the process that write the page most frequently. In the write vector technique, a home page is divided into blocks and a write vector table is maintained for each shared page in its home to record for each processor which block(s) has been modified since the processor fetched the page last time, so that only modified blocks are sent back when the processor requires the page. In the normal protocol, updates by remote processors to a page are recorded in the write vector table when diffs are applied, and updates by the home node are recorded with the twin mechanism. In the SMP protocl, the modifications made by the co-home hosts of a page should also be known by the real home host. JIAJIA solves this problem through making the write vector table of any given page shared among all processes within a node, so that modifications to the write vector table made by co-home hosts are always visible to the real home host. The home migration technique also causes problem to the SMP protocol of JIAJIA. As has been mentioned, the mmap() mechanism JIAJIA uses to share physical memory among intra-node processes requires that shared pages are mapped before intra-node processes are forked. In contrast, the home migration algorithm of JIAJIA maps and unmaps pages dynamically during runtime. Compromise has to be made here that if the home of a shared page is migrated from one node to another, then the advantage of sharing home page among intra-node processes is given up for this page. The new home host is the real home host of the page, no co-home host exists for this page. 3.3
Intra-node Communication
Another optimization JIAJIA makes for SMP clusters is the optimization of intra-node communication. In the old version of JIAJIA which regards processors of an SMP as separate nodes, UDP protocol is used for communication between two intra-node processes. With the SMP communication support, processes within an SMP node communicate through shared memory. To support shared memory intra-node communication, JIAJIA allocates a shared communication buffer for each pair of intra-node processes. When a process wants to send a message to another process within the same node, the sending process first copies the message to the associated shared buffer. It then sends a SIGIO signal to the receiving process through kill() system call. On receiving the SIGIO signal, the receiving process directly reads the message from the shared communication buffer and then sets a tag in the buffer to indicate that the message has been read out. The sending process proceeds on observing this receiving tag. To keep the simplicity of implementation, the current version of JIAJIA does not include some other SMP-related optimizations such as intra-node synchro-
1138
Weiwu Hu, Fuxin Zhang, and Haiming Liu
Table 1. Characteristics and Sequential Run time of Benchmarks Appl. Size Mem. Barrs Locks Seq. Time(seconds) Water 1728 mole. 0.5MB 70 1040 491.99 Barnes 16384 bodies 1.6MB 28 64 321.83 Ocean 514 ∗ 514 60MB 858 1568 54.88 LU 2048 ∗ 2048 32MB 128 0 133.45 SOR 2048 ∗ 2048 16MB 200 0 89.44 TSP -f20 -r15 0.8MB 0 1167 216.70 MG 256 ∗ 256 ∗ 256 443MB 592 0 398.56∗ 3DFFT 128 ∗ 128 ∗ 128 96MB 12 0 74.05 EM3D 120 ∗ 60 ∗ 416 160MB 20 0 101.46 IAP18 144 ∗ 91 ∗ 18 20MB 5400 0 1994.95 ∗: Estimated from 128 × 128 × 128 MG sequential time (49.82 seconds). Sequential time of 256 × 256 × 256 MG is unavailable due to memory size limitation.
Table 2. Runtime Statistics of Parallel Execution Appl. Water Barnes Ocean LU SOR TSP MG 3DFFT EM3D IAP18
Msg. amt.(MB) JIA JIAsmp 36 29 81 65 891 563 25 18 12 5.3 7.4 6.3 553 260 149 129 2.4 1.1 2943 1355
Get pages JIA JIAsmp 3420 2870 9076 7408 107073 67733 12072 8292 2823 1216 4965 4237 32508 18025 17976 15408 1200 528 276643 136474
Diffs JIA JIAsmp 1897 1373 510 353 1294 1132 0 0 0 0 3236 2780 43192 17016 56 48 990 420 131136 49920
nization and locating fault pages in the caches of intra-node processes on page faults.
4
Performance Evaluation
The evaluation is done in four Ultra-2 nodes connected by a Myrinet. Each node has two 200MHz Ultra-sparc processor and 256MB memory. The benchmarks include Water, Barnes, Ocean and LU from SPLASH2[14], MG and 3DFFT from NAS Parallel Benchmarks[1], SOR and TSP from TreadMarks benchmarks[10], and two real applications EM3D for magnetic field computation and IAP18 for climate simulation[5]. Table 1 gives characteristics and sequential run time of these benchmarks.
A New Home-Based Software DSM Protocol for SMP Clusters
1.1 Execution Time(%) 51.16 24.46 1.0 .67.54 ......... ............. ........ ......... .... ....... ........ ........... .... 0.9 .......... ... ........ ........ ..... ....... 0.8 .................. ............. ..... ..... 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 smp smp smp Water Barnes LU
63.90 13.98 35.16 ... ... ............ ..... ... ... ... ... ... ... ............... ........ ... .. . . ... .. ..... ... .......... ........ .......... ........ ........ ........
......... ...... ...... .........
105.30 22.71 .14.25 .......... ..... ........ .............. ......... . . ........ ............. ....... ........... .... ........ ........ ........ .......... ........... ..... ...... ......... ..... ........... ........... .............. ............ ... .... ....... ......... ............. ...... ...... ........ ............ ...... ............... .......... ............. ..... ..... ... ............... ....... ............... .................... .......... .................... ............... ...............
1139
1.1 572.36 1.0 0.9
....... 0.8 .......... .......... .... ............ .......... 0.7 .............. .......... 0.6 ...... Server .......... ........... .... .......... ............. 0.5 .............Syn.
............. .. SEGV 0.4 ........ Comp. 0.3 0.2 0.1 smp smp smp smp smp smp smp 0.0 Ocean SOR TSP MG 3DFFT EM3D IAP18 ................. ............. ............... ............... ............... .................... .................... .......... ..........
Fig. 2. Parallel Execution Time Breakdown
Each benchmark is run under JIAJIA both with (JIAsmp ) and without (JIA) SMP optimization. The four nodes of two-processor SMPs are configured as eight separate hosts in JIA, and as four SMPs in JIAsmp . Table 2 gives some statistics of the parallel execution. Message amount, remote get page request count, and diff count are listed in Table 2 for each run of each benchmark. Figure 2 shows parallel execution time results of the execution. For each benchmark, the parallel execution time of JIA rides on the top of the corresponding bar, and the parallel execution time of JIAsmp is represented as percentages of the parallel execution time of JIA. In Figure 2, the execution time of each parallel run is broken down into four parts: page fault (SIGSEGV service) time, synchronization time, remote request (SIGIO) service time, and computation time. The first three parts of time are collected at runtime as the overhead of the execution, and the computation time is calculated as the difference of the total execution time and the total overhead. It can be seen from Table 2 and Figure 2 that, all of the ten benchmarks have less remote accesses, diffs, and consequently message amounts in JIAsmp than in JIA and seven of them achieve noticeable speedups with the SMP protocol. Figure 2 shows that JIAsmp achieves significant speedup over JIA in Ocean (26.0%), MG (8.9%), 3DFFT (12.9%), and IAP18 (20.3%). The speedup of Ocean turns from negative in JIA to positive in JIAsmp . The common property of these four benchmarks is that their parallel speedup are not high. The eight-processor speedup of JIA is negative in Ocean, and is less than four in MG, 3DFFT, and IAP18. Therefore there is large room for performance improvement. Besides, the absolute number of remote accesses, diffs, and consequently message amounts reduced by the SMP protocol is great in Ocean, MG, and IAP18. Table 2 shows that JIAsmp causes 300MB less message amounts than JIA in Ocean
1140
Weiwu Hu, Fuxin Zhang, and Haiming Liu
and MG, and 1.6GB less message amounts than JIA in IAP18. As a result of the great reduction of page faults, both the SIGSEGV time and server time are significantly reduced, as can be observed in Figure 2. The synchronization time of Ocean, MG, and IAP18 are also reduced due to the reduction of the number of diffs (diffs are generated at synchronization points). In FFT, though the message amount reduced by the SMP protocol is not as significant as the above three benchmarks, JIAsmp still reduces SIGSEGV time significantly. It can also be seen from Figure 2 that a great part of performance gains of JIAsmp over JIA in FFT is caused by the reduction of computation time in JIAsmp . This fact implies that FFT has better utilization of processor pipeline, cache, and TLB in JIAsmp than in JIA. Figure 2 also shows that LU, SOR, and EM3D obtain moderate performance benefit (around 5%) from the SMP protocol. It can be seen from Table 2 that, though JIAsmp has only 43%, 44%, and 69% remote accesses of that required by JIA in SOR, EM3D, and LU, the absolute message amounts reduced are not significant. Besides, the performance of these three benchmarks is already high in JIA (the eight-processor speedup is 5.46 in LU, 6.4 in SOR, and 7.12 in EM3D) and the room for performance improvement is not large. Again, the execution time breakdown in Figure 2 shows that in LU, SOR, and EM3D, the performance gains of JIAsmp over JIA is mainly caused by the reduction of SIGSEGV time and server time as a result of remote accesses reduction. In EM3D, the synchronization time is also reduced in JIAsmp because JIAsmp generates much less diffs than JIA. In Water, Barnes, and TSP, though the number of remote accesses and diffs is reduced with the SMP protocol, both the reduction range and the absolute message amounts reduced are not large. Besides, in these three benchmarks, performance is already high in JIA (the eight-processor speedups is 7.28 in Water, 6.29 in Barnes, and 6.16 in TSP) and the bottleneck for further performance improvement does not lie in the communication. Rather, synchronization is the major obstacle for further performance improvement in these three benchmarks. It can be seen from Table 1 and Figure 2 that lock is the major synchronization mechanism and synchronization overhead constitutes the major overhead in these three benchmarks. This implies that the lock waiting time dominates the overhead and (with the high bandwidth Myrinet) the moderate remote accesses reduction caused by JIAsmp has little influence on the performance of Water, Barnes, and TSP.
5
Conclusion and Future Work
This paper introduces the SMP protocol for the JIAJIA software DSM system. In the protocol, processes in the same SMP node keep their separate caches but combine their home pages together so that each process in the node “sees” a larger home. Evaluation results show that compared to the JIAJIA without SMP optimization, the SMP protocol achieves significant speedups in seven out of ten
A New Home-Based Software DSM Protocol for SMP Clusters
1141
benchmarks. Quantitatively, the SMP protocol achieves a speedup of 4% − 5% in LU, SOR, and EM3D, about 10% in MG and 3DFFT, and more than 20% in Ocean and IAP18. Water, Barnes, and TSP do not benefit noticeably from the SMP protocol. It is expected that the SMP protocol of JIAJIA can perform better if there are more processors (e.g., four) within an SMP node. It can also be learned from the above performance evaluation that communication bandwidth is critical to the performance of software DSMs such as JIAJIA. In the evaluation, the high bandwidth Myrinet makes JIAJIA not sensitive to the moderate reduction of remote accesses in Water, Barnes, and TSP. Compared to our previous evaluation results[4,6] on clusters connected by 100Mbps Ethernet, higher speedups are obtained in the cluster connected by Myrinet. Our recent work about JIAJIA include further improving the coherence protocol of JIAJIA, supporting JIAJIA with faster communication mechanism such as special remote access hardware, implementing fault tolerance mechanism such as checkpoint in JIAJIA, building a pre-compiler to convert OpenMP programs to the API of JIAJIA, and porting more real applications on JIAJIA. Further information about JIAJIA is available at www.ict.ac.cn/chpc/index.html.
References 1. D. Bailey, J. Barton, T. Lasinski, and H. Simon, “The NAS Parallel Benchmarks”, Technical Report 103863, NASA, Jul. 1993. 2. S. Dwarkadas, N. Hardavellas, L. Kontothanassis, R. Nikhil, and R. Stets, “Cashmere-VLM: Remote Memory Paging for Software Distributed Shared Memory”, in Proc. of the 13th Int’l Parallel Processing Sym., pp. 153–159, April. 1999. 3. A. Erlichson, N. Nuckolls, G. Chesson, and J. Hennessy, “SoftFLASH: Analyzing the Performance of Clustered Distributed Virtual Shared Memory”, in Proc. of the 996 Int’l Conf. on Architectural Support for Programming Languages and Operating Systems, Oct. 1996. 4. W. Hu, W. Shi, and Z. Tang, “Reducing System Overhead in Home-Based Software DSMs”, in Proc. of the 13th Int’l Parallel Processing Symp., pp. 167–173, Apr. 1999. 5. W. Hu, F. Zhang, L. Ren, W. Shi, and Z. Tang, “Running Real Applications on Software DSMs”, in Proc. of the 2000 Int’l Conf. on High Performance Computing in the Asia-Pacific Region, May 2000. 6. W. Hu, “Reducing Message Overheads in Home-Based Software DSMs”, in Proc. of the 1st Workshop on Software Distributed Shared Memory, pp. 7–11. June 1999. 7. Y. Hu, H. Lu, A. Cox, and W. Zwaenepoel, “OpenMP for Networks of SMPs”, in Proc. of the 13th Int’l Parallel Processing Symp., pp. 302–310. Apr., 1999. 8. L. Iftode, “Home-based Shared Virtual Memory”, Ph. D. Thesis, Princeton University, Aug. 1998. 9. P. Keleher, S. Dwarkadas, A. Cox, and W. Zwaenepoel, “TreadMarks Distributed Shared Memory on Standard Workstations and Operating Systems”, in Proc. of the 1994 Winter Usenix Conf., pp. 115–131, Jan. 1994. 10. H. Lu, S. Dwarkadas, A. Cox, and W. Zwaenepoel, “Quantifying the Performance Differences Between PVM and TreadMarks”, Journal of Parallel and Distributed Computing, Vol. 43, No. 2, pp. 65–78, Jun. 1997.
1142
Weiwu Hu, Fuxin Zhang, and Haiming Liu
11. R. Samanta, A. Bilas, L. Iftode, and J. Singh, “Home-based SVM Protocols for SMP Clusters: Design and Performance”, in Proc. of the 4th Int’l Symp. on High Performance Computer Architecture, Feb., 1998. 12. D. Scales, K. Gharachorloo, and A. Aggarwal, “Fine-grain Software Distributed Shared Memory on SMP Clusters”, in Proc. of the 4th Int’l Symp. on High Performance Computer Architecture, pp. 125–136, Feb., 1998. 13. R. Stets et al., “Cashmere-2L: Software Coherent Shared Memory on a Clustered Remote-Write Network”, in Proc. of the 1997 ACM Symp. on Operating Systems Principles, pp. 170–183, Oct. 1997. 14. S. Woo, M. Ohara, E. Torrie, J. Singh, and A. Gupta, “The SPLASH-2 Programs: Characterization and Methodological Considerations”, in Proc. of ISCA’95, pp. 24–36, 1995.
Encouraging the Unexpected: Cluster Management for OS and Systems Research Ronan Cunniffe and Brian A. Coghlan Department of Computer Science Trinity College Dublin, Ireland {ronan.cunniffe, brian.coghlan} @cs.tcd.ie
Abstract. A framework for cluster management is proposed that enables a cluster to be more efficiently utilized within a research environment. It does so by removing cluster management to a management node, leaving the compute nodes as essentially bare machinery. Users may schedule access to one or more of the compute nodes via the management node. At the scheduled time, a previously-saved image of their research environment is loaded, and the session begun. At the end of the session the user may save a new image of the environment on the management node, to be reloaded at another time. Thus the user may work with a customized environment, which may even be a fledgling operating system, without fear of interference to other researchers. This enables the capital investment of a systems research cluster to be amortized over a greater number of researchers.
1. Introduction We assume clusters represent a significant capital investment, and that in order to maximise the utilization of that investment they are very closely managed, with jobs scheduled on the compute nodes by management software such as CCS [1], PBS [2,3], LSF[4] or LoadLeveller [5] integrated with the working environment. It is likely then, that any modifications to the working environment that might jeapordize stability will be very unwelcome. Hence OS or systems research, that is likely to jeopardize stability, requires either a private cluster or diplomatic negotiations over how far any researcher can modify the shared working environment. The former is a very inefficient use of funding, the latter a severe constraint on research targets. Here we propose a scheme that allows the researcher the illusion of a private cluster whilst in fact using a shared cluster.
2. The MultiOS Framework What we wish to describe here is the general philosophy rather than a specific implementation, since it is felt that any implementation may need to be adapted to sitespecific tools (schedulers, for instance) and the adaptability should be part of the A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1143-1147, 2000. Springer-Verlag Berlin Heidelberg 2000
1144
Ronan Cunniffe and Brian A. Coghlan
philosophy. However, describing the framework is probably best done in concrete terms, by describing the cluster for which it is being initially designed, how it will operate on that cluster, and what hardware/software tools are required. Our cluster is made up of sixteen PC compute nodes and two storage servers, linked by a switched fabric of SCI links. In addition, there are two NIS servers, a http server and a firewall server. The compute nodes are arranged in logical groups of 4. Each compute node has 256MB of DRAM and a 2GB local hard-disk. The storage servers are connected to a large RAID, and all machines are connected to the external network via 100Mb/s Ethernet. All normal access to the cluster is through this network connection and physical access to compute nodes and servers is minimised. The MultiOS server will execute either on an extra server node, or on one of the existing servers. Over time, it is hoped to increase the number of compute nodes. The central idea behind the MultiOS framework is that between two successive zero-management ‘research’ sessions there is a ‘management’ session where all access is suspended, and the environment installed on the local hard-disks can be changed by management software according to the schedule or via user interaction with the MultiOS server. Three requirements must be met to do this. The MultiOS server must be able to: 1) force a reboot of any compute node on demand (via a hardware mechanism) 2) gain control of a compute node during boot - before any environment starts. 3) install and run a management environment that does not use the local disk. 2.1 A Hardware Reset Mechanism There is no guarantee that a running environment will respond gracefully to a request to shut down, indeed the highly experimental work this framework is designed to support is quite likely to crash or lock up the hardware it is running on. Disruption of the schedule is not acceptable, nor is demanding human intervention. MultiOS must be able to regain control after a crash. In our case, this will either be implemented using a LonWorks network [6] with the module in each compute node wired to the reset pin, or a parallel switched 100Mbps Ethernet fabric with modified wake-on-LAN [7]. 2.2 Control Must Be Passed to the MultiOS Server During Boot This can be done by using the standard protocols for booting diskless workstations but in a slightly non-standard way. Most PC Ethernet cards have a socket for a ‘bootrom’, which can be recognised by the normal boot sequence and invoked before any other bootable device. This bootrom sends a BOOTP [8] request to find the machine’s own identity and the name of a file to download, and then uses TFTP [9] to download it. Once downloaded, that file is executed. Obviously, if two different files are downloaded on two successive boots, the compute node will boot differently. This is what MultiOS does, alternating between two executables. The first executable is the management environment; the second is a tiny program which simply passes control straight on to the boot-block of the local
Encouraging the Unexpected: Cluster Management for OS and Systems Research
1145
hard-disk, so that the compute node boots into whatever target environment has been installed, as though the network interrogation phase never occurred. 2.3 A Special Management Environment Which Does Not Use the Local Disk This special management environment could be a specialised program, or a suite of specialised programs. Many already exist [10, 11, 12], but hardcoded solutions are not flexible - it is difficult to take advantage of a hardware configuration or network topology unless the program is re-written for each specific cluster. It also does not easily exploit possible redundancy of transmission, where multiple compute nodes are to be loaded with near-identical images. Essentially this approach suffers from conception in a vacuum, where every new optimisation requires new tools or low-level modifications to existing ones. A much more flexible and powerful approach is to use a fully featured operating system, and to assemble custom solutions from its native toolkit. In our case this OS is Linux - the management environment downloaded via TFTP is actually a Linux kernel configured as for a diskless workstation - mounting some of its filesystems in memory, and the remainder over NFS (the OS images for instance). The high-level tools for moving OS images across the network are then built on the standard UNIX commandline tools, such as dd, gzip, diff, rcp, etc. The main costs of using Linux are the image storage (a 'minimal' Red Hat 6.0 install, slightly modified for diskless operation, is 20MB per node and 80MB shared by all), and the network bandwidth consumed by multiple parallel accesses to this resource.
3. The MultiOS Server We have described the low-level mechanism for transparently switching between research environments. A permanent high-level control mechanism - the MultiOS server - is also needed, to implement user commands and to provide status information. A further requirement is to allow users to reserve some or all of the cluster in advance. In line with our preferences for adaptability, it is proposed not to integrate a custom node reservation system or scheduler into the MultiOS server, but to use an external program. A basic implementation will be provided, but a modular approach allows it to be easily replaced. Likewise, we do not intend to integrate a user interface into this server, allowing for whatever variations on user access are desired. We intend to provide a web-based console, but this may not always be appropriate. The overall structure of the MultiOS ‘server’, therefore, is really a set of three elements: the user interface program, the reservation system, and the actual MultiOS server program that controls the low-level operation of the framework.
1146
Ronan Cunniffe and Brian A. Coghlan
4. Security Issues The MultiOS scheme introduces important security issues. Four primary areas of concern have been identified. Firstly, the BOOTP and TFTP protocols were designed to be small rather than secure, and are vulnerable to a variety of attacks, denial-of-service (DOS), IP and MAC address spoofing, etc. A successful attack results in the node running an OS of the attacker’s choice. Machines booting in this way have to be considered as totally unprotected, and therefore a weak point in whatever network(s) they belong to. The only effective workaround is to boot from a separate physically secure network. Secondly, the image server must allow external access to the environment images, both for users and for the MultiOS framework, and both read and write access must be subject to authorization. However, since a management cycle involves access from the compute nodes, the user's authorization must be transmitted explicitly to them. Transmissions can be encrypted, but transport layer security is useless if one of the parties is compromised. Again, using a switched private network eliminates this concern. Thirdly, it is ironic that the behaviour of a cluster being reset is indistinguishable from a well-synchronised distributed-denial-of-service (D-DOS) attack on the MultiOS server, mounted by a large number of high-bandwidth attackers. Services such as BOOTP, TFTP, and the network filesystems NFS and SMB are commonly configured to shut themselves down when attacked, rather than overload the server machine they run on. For the purposes of MultiOS, either the cluster-resetting must be done in staggered fashion to keep the load below the triggering threshold, or this mechanism must be disabled, and the cluster made physically secure against such attacks instead. Since the MultiOS server directly controls the compute node reset mechanism, the former is likely to be easier. The final area of security risk is the user interface. A web-based interface is favoured for customisability, platform independence and ease-of-use, but hard experience teaches that web servers can be crashed. This is a secondary argument for splitting the user-interface away from the MultiOS server. It can now reside on a separate machine, communicating through an encrypted channel. In this scenario, a webserver crash will shut down interactive access to MultiOS and prevent changes being made to the schedule, but will not affect the cluster itself.
5. Summary A framework for cluster management has been proposed that is specifically tailored to OS and systems software research. Users may schedule access to one or more of the compute nodes via a separate management node, the MultiOS server. At the appointed time, a previously-saved image of their research environment will be loaded from an image storage server, and the session begun. At any time during the session the user may save a new image of the environment, to be reloaded at another time. Hence a number of users can be accommodated, each within their own environment. This
Encouraging the Unexpected: Cluster Management for OS and Systems Research
1147
enables the capital investment of the cluster to be amortized over a greater number of researchers. Work on the framework, called MultiOS, began in October, 1999, and is still in progress. Our thanks to Prof. J. G. Byrne for his support.
References 1. Keller, A., Reinefeld, A., "CCS Resource Management in Networked HPC Systems", Proc. Heterogeneous Computing Workshop HCW'98, 1998 2. Portable Batch System Documentation, MRJ Ltd., 1998. http://pbs.mrj.com/docs/html 3. Henderson, R.L., "Job Scheduling under the Portable Batch System, In: Job Scheduling Strategies for Parallel Processing", Feitelson, D.G. and Rudolph, L. (eds), LNCS, pp.279294, Vol.949, Springer-Verlag, 1995. 4. Load Sharing Facility Suite 3.2 Documentation, Platform Computing Inc., 1998. 5. Prennis, A.jnr, "Loadleveller: workload management for parallel and distributed computing environments", Proc. Supercomputing Europe (SUPEUR’96), October 1996. 6. Foster, G.T., Glover, J.P.N., Warwick, K., "Flexible Distributed Control of Manufacturing Systems Using Local Operating Networks", Proc. LonUsers International Fall Conference, 1995 7. Magic Packet Technology White Paper, AMD Publication no.20213, Advanced Micro Devices Inc., November 1995 8. Wimer, W., "Clarifications and Extensions for the Bootstrap Protocol", IETF Request For Comments Document no.1542, October 1993. 9. Sollins, K., "The TFTP Protocol (Revision 2)", IETF Request For Comments Document no.1350, July 1992. 10. Rembo Technology, http://www.bpbatch.org 11. Free Software Foundation, "Grand Unified Bootloader" http://www.gnu.org/software/grub.en.html 12. Yap, K; Savoye, R; "Network Interface Loader" http://nilo.sourcefourge.net/
Flow Control in ServerNetR Clusters Vladimir Shurbanov1 , Dimiter Avresky1 , Pankaj Mehra2 , and William Watson3 1 2
Boston University, 8 Saint Mary’s St., Boston, MA 02215, USA, {vash,avresky}@bu.edu Compaq Tandem Labs, 19333 Vallco Parkway, Cupertino, CA 95014 [email protected], 3 Compaq Tandem Labs, 14231 Tandem Blvd., Austin, TX 77728 [email protected]
Abstract. This paper investigates the performance implications of several end-to-end flow-control schemes based on the ServerNetR systemarea network. The static window (SW), packet pair (PP), and the simplified packet pair (SPP) flow control schemes are studied. Additionally, the alternating static window (ASW) flow control is defined and evaluated. Previously, it has been proven that the packet-pair scheme is stable for store-and-forward networks based on Rate Allocation Servers. The applicability of a PP flow control to wormhole-routing networks is studied and evaluated through simulation. It is shown that if high throughput is desired, ASW is the best method for controlling the average latency. On the other hand, if low throughput is acceptable, SPP can be applied to maintain low latencies.
1
Introduction
The term flow control refers to the techniques that enable a data source to match its transmission rate to the currently available service rate in the network and at the receiver [9, 11]. Apart from this main goal a flow control mechanism also should adhere to the following requirements: be simple to implement, use a minimum of network resources (bandwidth, buffers, etc.), and operate effectively when used by multiple sources. Additionally, the principles of fairness should be observed for shared resources. And finally, the entire networked system should be stable, i.e., for a constant configuration the transmission rate of each source should converge to an equilibrium value. This paper considers two closed-loop flow control schemes - the static window and the packet pair flow control protocols. In the static window scheme [12] the source stops transmitting when in has sent a number of unacknowledged (outstanding) request equal to the size of the defined window. The main problem with this approach is that the optimal window size depends on many factors which vary over time and differ among connections. Therefore, choosing a single static window size that is suitable for all connections is impossible. In the packet pair scheme [8] the source estimates and predicts the network conditions based on the delay observed for a pair of consecutive packets and adjusts its transmission A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1148–1156, 2000. c Springer-Verlag Berlin Heidelberg 2000
Flow Control in ServerNetR Clusters
1149
rate accordingly. The scheme is proved [8] to result in a stable system for storeand-forward networks based on Rate Allocation Servers. This paper investigates the applicability of packet pair flow control to wormhole-routing networks that are not based on Rate Allocation Servers. Since the packet pair flow control does not limit the maximum number of outstanding requests, the static window protocol is employed in conjunction with it. Flow Control in the ServerNet SAN. The ServerNet system area network (SAN) is a wormhole-routed, packetswitched, point-to-point network with special attention paid to reducing latency and assuring reliability [4, 5]. It uses multiple high-speed, low-cost routers to rapidly switch data directly between multiple sources and destinations. ServerNet implements two levels of flow control - hop-by-hop flow control and end-to-end flow control. Hop-by-hop flow control is performed by the exchange of special flow control flits (busy and ready) between the two devices connected through the link. Busy flits signal that the receiver queue is full. When the transmitting device receives a busy flit it ceases sending data until it receives a ready flit. End-to-end flow control is performed through the static window protocol. In this scheme each request packet has to be acknowledged by a response packet. The size of the static window limits the number of unacknowledged (outstanding) requests that can be transmitted. When a source reaches this limit it ceases transmitting requests until it receives at least one response. Simulation Model. The simulation model is discrete-event and unit-time [7]. Each device enters a particular state during each time step. The devices are activated in an random order. All performance measures collected during the course of the simulation are averaged over a number of packets sufficient to achieve the desired level of data accuracy for a confidence level of 95%. Collection begins when the system enters a steady state. Steady state is determined by the method of moving averages presented in [7]. The statistical data produced by the simulator was validated using experimental data collected at Compaq Tandem Labs. Discrepancies between the simulation and experimental results were found to be less that 5%. Since the simulation operates at a data accuracy of 3% these discrepancy are insignificant.
2
Packet Pair Flow Control
The packet pair flow control [9] (PP) belongs to the class of rate-based flow control protocols. PP estimates the conditions in the network by observing the time interval between the receptions of the responses to a pair requests of requests (packet pair) transmitted back-to-back. Moreover, it predicts the future service rate in the network and adjusts incorrect past predictions. The PP flow control is subject to the following limitations: packets must always be transmitted in pairs; the service rate of non-bottleneck servers is assumed to be deterministic.
1150
Vladimir Shurbanov et al.
To circumvent these limitations the simplified packet pair (SPP) flow control defined in [6] is described below. Implementation of SPP. The simplified packet pair flow control (SPP) is implemented as follows: 1. The inter-request delay is determined by a variable, I, which is 0 initially. After a packet is transmitted the next packet may not be transmitted before a time period of I expires. 2. The difference between the RTTs of every pair of consecutive packets to/from the same destination is compared with a threshold parameter delta. If delta is greated a “win” is registered. 3. A history of the last h comparisons is kept. 4. If the number of wins, hW , are more than h2 , I is decremented by a value that depends on hW . The greater hW , the greater the decrement. 5. If hW < h2 , I is incremented by a certain value that depends on hW . The smaller hW , the greater the increment. Evaluation of SPP. Some statistics for the operation of SPP are shown in Table 1. They are based on the topology shown in Fig. 1-a with a Uniform traffic distribution and a generation rate of 200 requests/µs, which is selected to be past the saturation point of the network. Consider the statistics for the number of “wins” (Table 1a). A window of h = 8 comparisons is kept. When the number of “wins” is equal to 4, SPP does not modify the inter-request delay. It can be concluded based on the average number of “wins” and the low deviation that generation rate controlled by SPP converges to an equilibrium state, i.e., the system is stable. Average: 4.34 Std. Dev.: 1.16 Minimum: 0 Maximum: 8 (a) Number of Wins
Average: 25.96 Std. Dev.: 29.47 Minimum: 0 Maximum: 141 (b) Inter-Request Delay (µs)
Table 1. Statistics for SPP
The inter-request delay statistics (Table 1-b) show that the SPP algorithm introduces a significant inter-request delay - an average of approximately 26 µs. During this period requests are held at the source devices. By adding the RTT (5.94 µs is the average observed in this case), the average request-to-response time totals approximately 32 µs. This value is higher than the RTT in the absence of the SPP algorithm, where the request-to-response time is equivalent to the RTT of 30.2 µs. It becomes apparent that the RTT is reduced in SPP by introducing an approximately equivalent delay at the source device. Essentially, the SPP scheme changes the location (source device vs. network and destination
Flow Control in ServerNetR Clusters
1151
device) where delays are incurred, but does not reduce the total request-toresponse delay.
CPU CPU CPU CPU
CPU CPU CPU CPU
CPU CPU CPU CPU
8 X5
7
X4
X6
Throughput (flits/tick)
6
X8
X7
4 3
X2
1 1
CPU CPU CPU CPU
X1 2
3
X3
4
12
13
3
1
2
CPU CPU CPU CPU
11
4
2
6
5
5
14
CPU CPU CPU CPU
0
0
0.1
0.2
(a) 24-CPU Topology
0.3
0.4 0.5 Time (ms)
0.6
0.7
0.8
0.9
(b) Throughput snapshots
Fig. 1. Network Topology and Throughput Snapshots
8
0.035
SPP 0.1 microsec. SPP 1 microsec. SPP 8 microsec. SW 4 OR
7
0.025 Average RTT (ms)
Throughput (Flits/tick)
6
5
4 3 2
0.02 0.015 0.01 0.005
1 0
SPP 0.1 microsec. SPP 1 microsec. SPP 8 microsec. SW 4 OR
0.03
0
50 100 150 200 250 Request generation rate (64B-requests/ms)
(a) Throughput
300
0
0
250 200 150 100 50 Request generation rate (64B-requests/ms)
300
(b) Round Trip Time
Fig. 2. Performance of SPP, delta = 0.1, ..., 8µs The effect of the parameter delta on the operation of the SPP algorithm in the 24-CPU topology shown in Fig. 1-a is evaluated for delta = 0.1, 1, 8µs. The results are presented in Fig. 2 and Table 2. The window size is limited to 4 outstanding requests (OR). The data shows that as delta is increased both the throughput and the RTT increase, growing closer to the performance characteristics achieved with the static window (SW) protocol alone. This trend is also observed in the average ORs, shown in Table 2.
1152
Vladimir Shurbanov et al.
It can be concluded that the parameter delta essentially limits the generation rate of devices. As delta is decreased, SPP introduces higher inter-request delays, thus reducing the generation rate. For low values of delta, a less requests are transmitted into the network. This results in low congestion and hence low RTTs. However, the low generation rate also leads to low throughput. Conversely, increasing delta leads to increases in both the RTT and the throughput. SPP delta (µs) 0.1 1.0 8.0 Throughput (f lits/tick) 3.6 4.63 5.61 Avg. RTT (ms) 10.9 19.9 23.9 Avg. OR (packets) 1.07 1.21 3.38
SW ASW 5.93 5.53 26.4 21.9 3.5 2.83
Table 2. Flow Control Schemes: Throughput, Average Round Trip Time (RTT), and Average Outstanding Requests (OR)
3
Alternating Static Window Flow Control
Ideally, the flow control scheme should maintain a high number of ORs to maximize throughput by overlapping (pipe-lining) the request-propagation and the request-processing delays but at the same time it should limit the number of ORs to minimize queueing delays at the end devices. In the SW scheme the number of ORs is maintained at maximum regardless of the delays, which leads to high throughput and high delays. An alternative approach is to halt the generation of requests when the high window mark is reached and to resume generation when the low window mark is reached. Reaching the high window mark is taken as an indication that the RTT is large, i.e., the network is overloaded. While the low window mark indicates that a sufficient amount of requests have been processed and it can be assumed that the network load has decreased to an acceptable level. It can be expected that this scheme will maintain high throughput because it pipe lines the requests, however it should lead to reduced queueing delays since high delays are detected and generation is halted until the network load is relieved. To further support this conjecture we analyze the dynamic behavior of the network characteristics, based on the throughput and link usage snapshots shown in Figs. 1-b and 3. The link categories used in Fig. 3 are specified in Fig. 1-a. The following observations are made: • initially (0.004 ms) there is no stalling of the links and transmission is not at 100%; this occurs because the transfer of data from memory to the interface has a start-up delay and the interface is not fully utilized; • next (0.006 ms) there is more data available than the capacity of link 4 and stalling is observed;
Flow Control in ServerNetR Clusters
80.0
80.0
Utilization (%)
100.0
Utilization (%)
100.0
60.0 40.0
60.0 40.0 20.0
20.0 0.0
1
2
3
Category
0.0
4
Transmitting Stalled Idle
1
80.0
Utilization (%)
80.0
Utilization (%)
100.0
60.0 40.0 20.0
1
2
3
Category
0.0
4
Transmitting Stalled Idle
1
2
3
Category
4
Transmitting Stalled Idle
0.2 ms 80.0
Utilization (%)
80.0
Utilization (%)
Transmitting Stalled Idle
20.0
100.0
60.0 40.0
60.0 40.0 20.0
20.0
1
2
3
Category
0.0
4
Transmitting Stalled Idle
1
0.008 ms
2
3
Category
4
Transmitting Stalled Idle
0.3 ms 100.0
80.0
80.0
Utilization (%)
100.0
Utilization (%)
4
40.0
0.006 ms
60.0 40.0
60.0 40.0 20.0
20.0 0.0
3
Category
60.0
100.0
0.0
2
0.032 ms
0.004 ms 100.0
0.0
1153
1
2
3
Category
0.02 ms
4
Transmitting Stalled Idle
0.0
1
2
3
Category
4
Transmitting Stalled Idle
0.67 ms
Fig. 3. Link Usage Snapshots for Static Window Flow Control
1154
Vladimir Shurbanov et al.
8
0.035
SW 4 OR ASW 4 OR SPP
6 Throughput (Flits/tick)
SW 4 OR ASW 4 OR SPP
0.03 Average 2-way delivery (ms)
7
0.025
5
4
0.02
0.015
3 2
0.005
1 0
0.01
0
50 100 150 200 250 Request generation rate (64B-requests/ms)
(a) Throughput
300
0
0
250 200 150 100 50 Request generation rate (64B-requests/ms)
300
(b) Round Trip Time
Fig. 4. Performance of Flow Controls: (1) ASW, 4-0 OR; (2) SW, 4 OR; (3) SPP, delta = 8µs • the stalled time continues to increase until the static window limit (SWL) is reached and request transmission is halted, this causes idle time to appear at 0.01 ms and increase in proportion from there on; the idle time does not appear due to lack of data since it is constantly available; the increasing stalled time causes increasing delivery times; • the increasing delivery times cause the SWL to be reached more often and remain in effect for longer periods, thus causing increased idle periods, during which packet generation is halted. It is concluded that continuous transmission after the static window limit (SWL) is reached leads to a prolonged deterioration of the throughput, due to the increasing latencies displayed in the stalled time of link usage statistics. As shown in Fig. 1-b this deterioration continues for a period of time after which improvement is observed until the next period of deterioration commences. These trends alternate in a cyclic manner. It is desirable to maintain a controlled amplitude for this cyclic behavior, so that the average latency has a lower variance. One way to achieve a controlled amplitude is to halt transmission when the SWL is reached and to resume when the low window mark is reached. This would allow the network to recover from the large load and to transport the next burst of data more efficiently, with a lower latency. Such a scheme is implemented using the following rules. Definition 3.1 Alternating Static Window (ASW) flow control. 1. Transmission is allowed while the number of ORs is less than the high window mark (HWM); 2. once HWM is reached transmission is not allowed until enough acknowledgements are received to reduce the window size to the low window mark (LWM).
Flow Control in ServerNetR Clusters
1155
The performance of the network with ASW (HW M = 4, LW M = 0), is presented in Fig. 4 along with that of the 4-OR SW and the SPP algorithm with 4 OR and delta = 8µs. It can be seen that all three approaches achieve approximately equivalent throughput. However, ASW displays an average RTT approximately 25% lower that the other two approaches. This demonstrates that if high throughput is desired, ASW is the best method for controlling the average latency. On the other hand, if low throughput is acceptable, SPP can be used to provide very low latencies.
4
Summary
The simplified packet-pair (SPP) flow control is evaluated. It is shown that the operation SPP can be adjusted by varying the value of the threshold parameter delta. The alternating static window (ASW), is defined. It is demonstrated that ASW achieves a throughput equivalent to that of SW and SPP with a large delta. Additionally, ASW displays a significantly lower (approx. 25%) average two-way delivery time. It is concluded that SPP is a flexible mechanism which allows sources to maintain different generations rates for different destinations. The performance of a system with SPP can be adjusted ranging from high throughput and high latency to low throughput and low latency. On the other hand, ASW is a significantly simpler mechanism that provides high throughput and reduced latency in comparison with SW and SPP. Consequently, if high throughput is desired, ASW is the best method for controlling the average latency. On the other hand, if low throughput is acceptable, SPP can be used to provide extremely low latencies.
References [1] D. Avresky, V. Shurbanov, and R. Horst. The effect of the router arbitration policy on the scalability of ServerNetTM topologies. J. of Microprocessors and Microsystems, 21:545–561, 1997. Elsevier Science, The Netherlands. [2] D. Avresky, V. Shurbanov, and R. Horst. Optimizing router arbitration in pointto-point networks. J. of Comp. Comm., 22(5), April 1999. Elsevier Science, The Netherlands. [3] D. Avresky, V. Shurbanov, R. Horst, W. Watson, L. Young, and D. Jewett. Performance modeling of ServerNetTM topologies. The J. of Supercomputing, 14(1), August 1999. Kluwer Acad. Pub. [4] W. Baker, R. Horst, D. Sonnier, and W. Watson. A flexible ServerNet-based faulttolerant architecture. In Proc. of the 25th Int. Symp. Fault-Tolerant Computing, pages 2–11, Pasadena, CA, June 1995. [5] R. Horst. TNet: A reliable system area network. IEEE Micro, pages 37–45, Feb. 1995. [6] R. Horst and P. Mehra. ServerNet Rate Control. Tandem Labs Technical Memorandum TL.17.2, Dec. 1998. [7] R. Jain. The Art of Computer Systems Performance Analysis. John Wiley & Sons, Inc., 1991.
1156
Vladimir Shurbanov et al.
[8] S. Keshav. A control-theoretic approach to flow control. In Proc. of SIGCOMM’91 Conf., volume 21 of ACM Comp. Comm. Review, pages 3–15, Zurich, Switzerland, September 1991. [9] S. Keshav. An Engineering Approach to Computer Networks. Addison Wesley Longman, Inc., 1997. [10] J. Kim and D. Lilja. A network status predictor to support dynamic scheduling in network-based computing systems. In Proc. IEEE 13th Int. Par. Proc. Symp., pages 372–378, San Juan, PR, April 1999. [11] S. Low and D. Lapsley. Optimization flow control - i: Basic algorithm and convergence. IEEE Trans. on Networking, 7(6):861–874, December 1999. [12] C. Petitpierre and A. Zea. Implementing protocols with synchronous objects. In D. Avresky, editor, Dependable Network Computing, chapter 6, pages 109–140. Kluwer Acad. Pub., November 1999.
The WMPI Library Evolution: Experience with MPI Development for Windows Environments1 Hernâni Pedroso and João Gabriel Silva CISUC Universidade de Coimbra – Polo II 3030-397 Coimbra Portugal {hernani,jgabriel}@dei.uc.pt
Abstract. The usage of Windows based machines as a platform for parallel computing is rapidly increasing, mostly due to their excellent ratio cost/performance. The WMPI (Windows Message Passing Interface) was the first implementation of the MPI standard for Windows based machines. Originally based on the MPICH implementation, the library suffered several changes during the past years. This paper describes the evolution of the library since its first version. Recent changes have been introduced, which enable the dynamic creation of processes and the usage of simultaneous devices. This paper also describes the trends in the field that drove its evolution and the experience gathered through the development of the MPI standard for Windows clusters.
1 Introduction The performance of Personal Computers has considerably increased in the past years. Their level of performance rivals with the much more expensive Unix workstations [1], especially in integer computation. Most of the parallel computing users cannot afford an MPP machine or a large number of workstations, hence the option of using PCs became common. A cluster of PCs today is considered a high-performance platform with a low cost/performance ratio. The number of PCs available in the institutions is also a source of a considerable computational power that may be used for commodity supercomputing. WMPI (Windows Message Passing Interface) [2,3] was the first full implementation of the MPI standard [4] for Windows operating systems. MPICH [5] was the base of the first version, released in April 1996. WMPI used a Win32 port of the p4 [6] library to setup the environment and manage the communication between processes. By using the MPICH architecture it was also possible to maintain the compatibility between WMPI and the MPICH libraries in mixed clusters of Windows and Unix machines. 1
This work was partially supported by the Portuguese Ministry of Science and Technology and the European Commission through the R&D Unit 326/94 (CISUC) and the project PRAXIS XXI 2/2.1/TIT/1625/95 - ParQuantum.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1157-1164, 2000. Springer-Verlag Berlin Heidelberg 2000
1158
Hernâni Pedroso and João Gabriel Silva
Although the base architecture was never changed, we conducted several modifications during the WMPI lifetime to improve performance and usability. A recent study, which evaluated implementations of MPI for Windows NT environment [7], considered WMPI as the best freely available implementation. In addition, the study concluded that WMPI rivals with other commercial implementations in performance and functionality. However, the release of the MPI-2 standard [8] and requirements for more functionality led to the development of a new internal architecture for WMPI. A completely new library, WMPI version 1.5, was recently released. This new architecture is the base for WMPI 2.0, an MPI-2 compliant version of the WMPI library, which will be release in the near future. The MPICH architecture, which is the base for most of the existing libraries is presented. A section is dedicated to the evolution of the usage of PCs for high performance computing. The two WMPI architectures are presented as well as the major factors that determined the evolution of the library. Some lessons about Windows usage for MPI computing and implementation decisions are also presented.
2
Related Work
Several other implementations of MPI for Windows machines are available. There is a Windows version of MPICH developed by the Argonne National Laboratory (MPICH.NT) [9]. It uses shared memory for inter-process communication in the same machine and TCP/IP for remote processes. This implementation is still in its early steps. MP-MPICH [10] is a multi-platform MPI implementation based on the MPICH architecture developed by RTWH Aachen. This implementation is available for Unix and Windows systems. It supports SCI [11], TCP/IP and shared memory communication mediums. The low latency in the communication that this implementation provides [12] is achieved by using active wait in the devices. FM-MPI [13] is a MPI library built on top of the Fast Messages library [14]. It is based on the MPICH code. Relying on the Fast Messages implementation, FM-MPI has support for Myrinet and TCP/IP. All the above libraries are freely available, however there are also some commercial implementations of MPI for Windows environments. MPI/Pro [15], from MPI Software Technology, Inc and PaTENT MPI [16] commercialized by Genias Software GmbH. MPI/Pro is available for VIA, TCP/IP and shared memory intercommunication mediums. PaTENT MPI is provided with TCP/IP and shared memory support, and is actually an evolution of WMPI.
3
MPICH – The WMPI’s Base Architecture
The development of the MPICH library occurred along with the elaboration of the MPI standard. The objective of the Argonne National Laboratory/Missisipi State University development teams was to provide early feedback to the MPI Forum by creating a test bed to evaluate the correctness of the decisions. The library also aimed
The WMPI Library Evolution
1159
to enable the high performance community to experiment and use a standard implementation of MPI as early as possible. The aims of the architecture design were portability and efficiency. By freely providing the library, as well as the source code, the development team allowed the parallel computing community to rapidly use a reliable library that implemented the whole MPI interface (version 1 of the standard). Due to its portable characteristics, both the original development team other research institutions and software/hardware vendors developed several versions of the library for many different systems.
4 Windows Clusters Environment In 1996, when the first version of WMPI was released, dedicated clusters of PCs were practically unknown. The base idea of the library was to use the available PCs in the institutions that were being used for small interactive tasks (e.g. text editing and email reading). The PC’s owner wishes to keep the response time of the computer to its interactive actions. Hence, it was important that WMPI was as little intrusive as possible. For example, the use of polling was completely unacceptable because it would significantly slowdown the other processes, running on the same CPU, even when no calculation was occurring. A heterogeneous network of PCs and Unix workstations was very common. In the beginning, PCs were used to occasionally boost the computational power of the Unix workstations. PCs, due to their continued increase of performance, became more common for parallel computing. The use of common off-the-shelf components for creating clusters with a considerable computational power became normal. The computational power of each unit of the cluster had a considerable boost with the appearing of SMPs with two or four processors. The decision of purchasing PCs for high performance computing became normal, since it was a low investment with practically null risk. One of the drawbacks of constructing a PC cluster for high performance computing was the lack of interconnection networks that could present high bandwidth and low latency between nodes. Most of the PC clusters were using TCP over Ethernet and Fast Ethernet networks, which are not optimized for message exchange performance but to reliability and cost. As the computational power of the PCs grew, the network became the bottleneck of the cluster. Aware of this fact, the hardware vendors have started to create new technologies that improve the message passing performance between the computer nodes. VIA is the most recent effort, although SCI, Myrinet and Gigabit Ethernet are also available.
5 The First WMPI Architecture When releasing the first MPI library for parallel computing using Windows PCs, it was very important to allow it to cooperate with a MPI library running on Unix workstations as well. Although the idea of using PCs for high-performance computing was unusual, the possibility of using a heterogeneous environment seemed interesting
1160
Hernâni Pedroso and João Gabriel Silva
and promising for most of the parallel computing community. Hence, it was important that WMPI could cooperate with MPICH running on Unix workstations. WMPI was based on the existing MPICH implementation. For the sake of compatibility with Unix workstations, p4 was chosen as the communication subsystem that runs under the MPICH’s ADI. This decision was also the one that required a smaller development time, since p4 was very stable and a common communication subsystem for the MPICH library. Due to the excellent work of the MPICH developers, the upper layer was easy to port. P4 was the main concern, because it directly interacts with the operating system. P4 handles with two types of communication mediums: shared memory and TCP/IP. Processes running on the same machine use shared memory, while TCP/IP was used for exchanging information with remote processes. WMPI avoided any form of active wait. Any thread that needs to wait for some event to occur (typically waiting for a message) stops competing for the CPU and does not use its entire quantum. This was very important considering the environment that WMPI was addressing. During its lifetime, several problems were solved in the library. The development experience enabled to improve its performance. This resulted in a very stable and mature MPI library. Several thousands of users and institutions have already downloaded WMPI from our web site [3] and are actively using it.
6
Multiple Devices
New technologies have emerged to improve the message passing performance in clusters. VIA, SCI, Myrinet and Gigabit Ethernet are good examples of the recent efforts. The MPI library must be able to follow these improvements on the underlying systems. It is thus necessary to create specific devices for each technology in order to use the maximum performance that it can offer. The necessary resources to produce an efficient device for a new intercommunication media are considerable. It was urgent to create an architecture that enabled a rapid development of new WMPI devices for different communication media. In addition, it should be possible to create a device without knowledge of the library internals. This would enable third-party institutions (e.g. hardware vendors) to create devices for their technology, hence providing WMPI in a wide range of communication mediums. It is common to find more than one type of technology for communication in a cluster. The most common configuration is, probably, shared memory and TCP/IP. However, with the arrival of new technologies and with the falling of prices, other configurations can be found. Moreover, when upgrading an existing cluster, the new nodes can use a different communication medium than the older ones. The older nodes can be used in conjunction with the newer computers to increase the performance of the cluster to solve some important problem. Institutions may also desire to connect separated clusters, which might use different communication media. All these possibilities create a wide range of possible configurations for a MPI execution. The possibility of easily configuring the computation to use any number of specific devices, according to the cluster configuration, is presently considered very important.
The WMPI Library Evolution
1161
7 Dynamic Environment MPI was well accepted by the community and became a de facto standard for parallel computing. Eager for more functionality, both users and developers presented several requests to the MPI Forum to extend the MPI standard. Version 2.0 of the standard (MPI-2) includes significant new capabilities, from which dynamic process creation is a good example. The ability of using a dynamic process environment within MPI-2 improves the usability of the library for several types of applications. An application can decide in runtime the number of processes that are required to manage the data to be processed. Processes can be added or subtracted to the computation according to its needs. It can also adjust the number of processes according to the level of parallelism of each phase of the computation. Processing units or entire clusters can join at any time a client/server application, since MPI-2 allows for the joining of separately launched MPI computations. But the new chapters of MPI-2 require deep changes in existing MPI libraries. For instance, with the introduction of dynamic creation of processes, it is possible for processes not belong to the MPI_COMM_WORLD, that in MPI-1 was a global communicator. These processes are the result of a process spawn (creation) or the joining of two MPI computations. When a spawn is performed or two MPI computations join not all the processes of both computations have to be involved. This means that each process will have its own set of processes with which it can communicate. Although when the new processes join, the communication has to be performed through an inter-communicator, they can form an intra-communicator using the MPI_Intercomm_merge function. It is also possible to extract the remote group of processes in an inter-communicator and manipulate it as any other group. This implies that an inter-communicator can be closed (which should indicate that the two computations are independent) but some of the processes still communicate, through groups and intra-communicators that were built using the disconnected inter-communicator. This situation is an example of the extreme freedom that the users have to manipulate the processes and create the configuration of interconnections and communicators. The usage of ranks in MPI_COMM_WORLD to globally identify the processes is impracticable in such environment. It is necessary to create another form of global identification, which must be valid whether the process starts within the same computation or joins in runtime.
8
The Second WMPI Architecture
The new MPI-2 features lead us to design a completely new architecture for WMPI, to avoid having to do successive patches, which are more prone to errors and render the evolution difficult. Based on our experience with cluster computing and considering the trends on the field, several other reasons were identified that strongly required a new architecture. The design aimed to create an architecture that is able to: • manage a dynamic environment, where processes could be created and destroyed, join and leave the MPI computation;
1162
Hernâni Pedroso and João Gabriel Silva
• work with several devices simultaneously; • easily support new communication technologies by reducing the complexity, hence the development time, of the communication dependent code; • be completely thread safe; • diminish the communication latency; • increase communication throughput. By creating our own architecture, the compatibility to the MPICH library is lost, but WMPI gains the freedom to pursue its own goals without relying on the others pace. The new architecture does not try to be portable across other platforms. Since the WMPI library is used in Windows environments, we decided to use the features that the operating system provides. This reduces the complexity and increases the performance of the library. Thread safeness and performance were a constant concern during the design and implementation of the library’s architecture. Every structure and all the functions that manipulate its data were studied to verify if they have or not to be thread safe. The synchronization points were reduced to the minimum to improve the general performance of the library. The user is responsible for defining how processes communicate during the WMPI computation through a cluster configuration file. Devices are now independent DLLs that are loaded at the startup of each process according to the cluster configuration. WMPI has the capability to interact with any number of different devices simultaneously. Within each process, WMPI associates a device to each machine of the cluster, according to the cluster configuration. When one MPI process needs to interact with another process, it chooses the correct device and performs the necessary action. During the design of the new architecture, we identified the necessary operations that the devices should perform. It was important to reduce the expected functionality of the devices to a minimum, since it should be feasible to produce a device for every possible technology. Moreover, simple devices are easier to optimize and guarantee the independence of the library core. Each process contains a structure to represent each process with which it is connected. This structure has one record per process, containing the set of addresses of the process that it represents. Each record also contains a reference to the device that must be used to communicate with the process. This set of records represents all the processes in the computation with which the process that owns the structure can communicate, because it has at least one group in common with each of them. This new structure is already implemented and available in the WMPI 1.5 version, which can be downloaded from the WMPI web site [3].
9 Lessons Learned During the development and design of the WMPI library, we came to some conclusions about cluster computing, implementation and design decisions. This section describes some of the most important ones:
The WMPI Library Evolution
1163
• Windows Clusters: Dedicated clusters are still a small minority in the universe of Windows PCs used for parallel computing. WMPI is commonly used in clusters shared by several users, without any major management of the resources. Moreover, many of the applications run in a set of computers that are also used for interactive tasks. Hence, it is very important to make the MPI processes use fairly the available resources, since they are shared with other processes running in the same machine (MPI processes or others). In this environments the usage of polling is self-defeating, since the processes start to stall each other and the whole systems runs slower. • Resources Access: In a Windows environment, resources are shared among all the processes in the system. Moreover, each process may have more than one thread. These threads compete to access to the system resources. The operating system automatically manages the access to this resources. We have verified that in most of the occasions is more effective that the processes control the access and allow, through synchronization, only one thread to use the resource at a time. The results were more perceptible in SMPs, where more synchronization is necessary from the operating system. • Windows Synchronization API: The Windows API provides several synchronization functions. Through exhaustive tests, we concluded that considerable latency differences exist between them. It is thus important to carefully choose the one that best fits each synchronization requirements. • Synchronization Methods: The synchronization is responsible for most of the latency in the communication. The penalty of having a wrong synchronization design is enormous. The WMPI 1.5 version suffered several changes in the architecture to diminish the communication latency and an improvement of 50% was achieved when compared with the first tests.
10 Conclusions Due to the increase of computational power of PCs, their use became common for high-performance computing. Clusters of PCs, due to their low price, allowed small companies and universities to have access to a parallel computing platform to speedup their applications. The intercommunication technology became the bottleneck of the clusters, since the computational power of the PCs increases rapidly. New intercommunication technologies have emerged in the past years and are now becoming common as their price gets lower. Today it is possible to build a cluster with an enormous performance at a quite low price. WMPI was the first MPI implementation for Windows based machines, its first release having been, in 1996. Originally based on the MPICH implementation, it evolved through the years and became a very stable MPI implementation with a good performance. The number of users has increased along with the usage of PCs for high performance computing. The release of the MPI-2 standard brought new features that, together with other important demands, like thread safeness and the ability to work with multiple devices simultaneously, required a complete change of the internal architecture of the library, leaving behind the MPICH architecture that proved not be capable of withstanding
1164
Hernâni Pedroso and João Gabriel Silva
those changes. This new architecture will, hopefully, form the basis of a high performance stable MPI-2 library for Windows based clusters.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
CPU Info Center – http://bwrc.eecs.berkeley.edu/CIC/. Marinho, J. and Silva, J.G.: WMPI – Message Passing Interface for Win32 Clusters. Proc. of 5th European PVM/MPI User’s Group Meeting, pp.113-120 (September 1998). WMPI Homepage – http://dsg.dei.uc.pt/wmpi Message Passing Interface Forum: MPI: A message-passing interface standard. International Journal of Supercomputer Applications, 8(3/4):165-414 (1994). Gropp, W., Lusk, E., Doss, N. and Skejellum, A.: A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard. Parallel Computing Vol. 22, No. 6, (September 1996). Butler, R. and Lusk, E.: Monitors, messages and clusters: The p4 parallel programming system. Parallel Computing, 20:547-564 (April 1994). Baker, M: MPI on NT: The Current Status and Performance of the Available Environments. NHSE Review, Volume 4, No 1 (September 1999). Message Passing Interface Forum: MPI-2: Extensions to the Message-Passing Interface. (June 1997), available at http://www.mpi-forum.org. MPICH for WindowsNT Download Page - http://www-unix.mcs.anl.gov/~ashton/ mpich.nt.html. MP-MPICH: Multiple Platform MPICH - http://www.lfbs.rwth-aachen.de/~joachim/MPMPICH.html. Hellwagner, H., Reinefeld, A.: SCI: Scalable Coherent Interface – Architecture and Software for High-Performance Compute Clusters. Lecture Notes in Computer Science Vol. 1734, Springer, ISBN 3-540-66696-6 (1999). Performance of NT-MPICH - http://www.lfbs.rwth-aachen.de/~karsten/projects/ntmpich/performance.html. Lauria, M., Chien, A.: MPI-FM: High Performance MPI on Workstation Clusters. Journal of Parallel and Distributed Computing, Vol. 40, No. 1, pp. 4-18 (January 1997). Pakin, S., Karamcheti, V. and Chien, A.: Fast Messages (FM): Efficient, Portable Communication for Workstation Clusters and Massively-Parallel Processors. IEEE Concurrency, vol. 5, no. 2, April-June 1997, pp. 60-73 (1997). MPI Software Technology, Inc – http://www.mpi-softtech.com. Genias Software GmbH, PaTENT – Parallel Tools Environment on NT, http://www.genias.de/products/patent/index.html. The Beowulf Project – http://www.beowulf.org.
Implementing Explicit and Implicit Coscheduling in a PVM Environment andez2 , and Emilio Luque2 Francesc Solsona1 , Francesc Gin´e1 , Porfidio Hern´ 1
Departamento de Inform´ atica e Ingenier´ıa Industrial, Universitat de Lleida, Spain. {francesc,sisco}@eup.udl.es 2 Departamento de Inform´ atica, Universitat Aut` onoma de Barcelona, Spain. {p.hernandez,e.luque}@cc.uab.es
Abstract. Our efforts are directed towards the understanding of the coscheduling mechanism in a NOW system when a parallel job is executed with local workloads, balancing parallel efficiency against the local interactive response. Explicit and implicit coscheduling techniques in a PVM-Linux NOW (or cluster) has been implemented. Their performance and overheads executing local tasks and representative distributed benchmarks have been analyzed and compared.
1
Introduction
Over the years, researchers have been developing time-shared distributed schedulers using coscheduling techniques, trying to adapt them to the new situation of mixing local and parallel workloads [1], [2], [3], [4] and [5]. Explicit coscheduling, all processes in a parallel application are scheduled simultaneously, with coordinated time-slicing between them. Generally, this yields good parallel program performance and this is widely used to schedule parallel processes involving frequent communication [1]. Coscheduling will ensure that no process will wait for a non-scheduled process for synchronization/communication and will minimize the waiting time at the synchronization points. Two-phase spin-block synchronization primitives used for dynamic coscheduling, named implicit coscheduling in [2], [3] and [4], only requires processes to block awaiting messages arrivals for coscheduling to happen. With two-phase spin-blocking, the waiting process spins for a determined time and if the response is received before the time expires then continues executing else the requesting process blocks and another one is scheduled. Algorithms for implementing new explicit and implicit coscheduling environments are presented in this paper. Extensive performance analysis, as well as studies of the parameters and overheads involved in the implementation, demonstrated the applicability of the proposed algorithms in these new environments.
This work was supported by the CICYT under contract TIC98-0433
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1165–1170, 2000. c Springer-Verlag Berlin Heidelberg 2000
1166
2
Francesc Solsona et al.
Coscheduling
In this section, the methods and metrics to measure their cost for explicit and implicit coscheduling distributed tasks in a PVM-Linux NOW are described. 2.1
Explicit Coscheduling
The aim of explicit coscheduling is to schedule all the distributed tasks in the cluster at the same time and let them execute during a period of time. From one global controller process running in one node named master, control messages are sent (in a broadcast form) to every explicit process (named dts) running in the composing workstations of the cluster, which are responsible for implementing explicit coscheduling. One of these control messages (init ) informs all the dts processes to start delivering STOP and CONTINUE signals to their local highpriority distributed processes at regular intervals (see also [5]). The time spent in starting (Tstart ) all the distributed tasks is: Tstart = Ws (local) + Ww (dts) + Ssig (CON T ) + Ww (dis) + Ws (dts),
(1)
where Ww /Ws is the elapsed time in waking up/suspending dts, a local task (local ) or a distributed task (dis). Ssig (CON T ) is the maximum elapsed time in sending a CONTINUE signal to all the distributed tasks in the node. The time spent in stopping (Tstop ) all the distributed tasks is: Tstop = Ws (dis) + Ww (dts) + Ssig (ST OP ) + Ww (local) + Ws (dts),
(2)
where Ssig (ST OP ) is the maximum elapsed time in sending a STOP signal to all the distributed tasks in the node. Because the time in delivering a signal to a group of processes does not depend on the signal to deliver, we consider that Ssig (ST OP ) = Ssig (CON T ) = Ssig . Similarly, the values Ww = Ws = W are considered to be equal. In consequence 1 and 2 can be reformulated as: Tex = Tstart = Tstop = 4W + Ssig . 2.2
(3)
Implicit Coscheduling
The implicit coscheduling aim is to schedule only communicating distributed tasks at the same time. We are interested in only spinning the tasks during at most a context-switch period and not in spinning during the deliver of a roundtrip message as in [2,3,4], as distributed tasks can follow many types of communication patterns and the messages can arrive asynchronously to distributed tasks, at any time. The metric Tim is used to compute the maximum overhead added in spinning, which also gives us a first reference to choose the spin interval (sp): Tim = Ws (dis) + Ww (local)
(4)
Implementing Explicit and Implicit Coscheduling in a PVM Environment
1167
Algorithm 1 ImCoscheduling. Implements the implicit coscheduling. Initialize input time, execution time, sp while (no new fragment) and (execution time ¡ sp) and (execution time ¡ timeout) do execution time = current time - input time if (no new fragment) if (timeout) then block (timeout - execution time) else block (indefinitely)
Algorithm 2 OneFragment. Reads the fragment in only one phase. call pvm receive and wait until the fragment arrives read the whole fragment ( header + body )
3
Algorithms
The implemented algorithms detailed in this section show how new distributed environments were created. Function ImCoscheduling (Algorithm 1) implements implicit coscheduling by realizing a spin-block while the fragments (unit of PVM transmissions) composing a message are read. Algorithm 2, called OneFragment, reads each fragment in only one phase, instead of twice, as PVM does. Both algorithms were implemented in the pvm recv ( ) PVM routine. Algorithm 3, called Priority was implemented outside the PVM, in a process named Priority, a copy of which is in each node of the cluster. It is responsible for assigning a high priority to distributed tasks. To do this only is necessary to assign a high priority (one unit level less than Priority) to pvmd in its creation.
4
Experimentation
The experimentation has been performed in a Now made up of an interconnection network of 100 Mbps Fast Ethernet and four PVM-Linux PCs with the same characteristics: 350Mhz Pentium II processor, 128 MB of RAM, 512 KB of cache. A distributed application, sintree was implemented to measure performance of the implemented environments. It attends for a communication pattern of one to vary, and vary to one. sintree accepts two arguments: number of processes (M ) and number of iterations (N ). By default M = 4 and N = 30.000. Also, two kernel benchmarks (class A) from the NAS parallel benchmarks suite [6] were used: is and mg. In all the benchmarks, the communications between remote tasks was done through RouteDirect PVM mode. 4.1
Implemented Environments
The next distributed environments were created. The algorithm(s) used to implement each model is in parenthesis. PVM: original PVM. SPIN (1): the spin-block
1168
Francesc Solsona et al.
400
250
700
200 150
600 500 400 300
100 50
MXISPIN PRIOSPIN SPIN MXI PVM EXPLICIT PRIO
800
TIME (S)
300 TIME (S)
900
MXISPIN PRIOSPIN SPIN MXI PVM EXPLICIT PRIO
350
200 0
0.5
1 1.5 2 LOCAL TASKS
2.5
3
100
0
0.5
1 1.5 2 LOCAL TASKS
2.5
3
Fig. 1. sintree execution. (left) N = 30000. (right) N = 70000.
is only performed in the reading of the data fragment. MXI (2). MXISPIN (1, 2). PRIO (3). PRIOSPIN (1,3). EXPLICIT: periodically, after 90000 µs the dts daemon in each node delivers a STOP signal to all the local distributed processes and then, elapsed 10000 µs, dts delivers a CONTINUE signal to reawaken them. The measured Tim 10 µs, so in the spin models an sp of 10 µs was chosen.
Algorithm 3 Priority. Assigns a high priority to distributed tasks. fork&exec ( pvmd ) set priority ( pvmd = max priority - 1)
4.2
Results
Distributed Tasks Performance Fig. 1 shows the sintree execution times executing in the seven above cited modes while the local workload in each node (simulated by compiling applications) is varied from 0 to 3. As was expected, optimal execution of the PRIO case can be observed. EXPLICIT without tasks is the worst mode, by increasing the workload, its performance scarcely decreases due to Tex does not vary. MXI and SPIN modes scale fine and their performance is always between the PVM and PRIO. SPIN is faster than PVM because avoids a lot of times the blocking overhead in receiving messages. The PRIOSPIN case gives worse results than PRIO, as the unnecessary spin-block phase added in the first mode, this only adds an unnecessary overhead in the reading of the fragment. MXISPIN works worse than MXI, as in this case penalties when time-slice expires are more than ones in context switching. Fig. 2 shows the results obtained from executing is and mg in the different models. The behavior of mg is similar to the sintree one. On the other hand, is does not work as fine as mg and sintree in the SPIN cases.
Implementing Explicit and Implicit Coscheduling in a PVM Environment 550
MXISPIN 500 PRIOSPIN SPIN 450 MXI PVM 400 EXPLICIT PRIO 350
TIME (S)
TIME (S)
550
300 250
MXISPIN 500 PRIOSPIN SPIN 450 MXI PVM 400 EXPLICIT 350 PRIO 300 250
200
200
150
150
100
0
0.5
1169
1 1.5 2 LOCAL TASKS
2.5
3
100
0
0.5
1 1.5 2 LOCAL TASKS
2.5
3
Fig. 2. Execution of the NAS parallel benchmarks (left) mg and (right) is. Table 1. slowdown of a compiling local task. slowdown PRIO PRIOSPIN EXPLICIT SPIN MXI MXISPIN sintree 1.4 1.4 1.4 3.6 1.4 3.6 is 2.8 2.8 4.2 2.1 0 2.1 mg 90 92 42 8 1.6 8
Local Tasks Performance The influence of the models in the local tasks was based on measuring the slowdown calculated as follows: sdM OD =
T M ODEL − T P V M 100, TPV M
where TMODEL (TPVM) is the execution time of a local task (a compiling application) when it was executed in such model (original PVM). See Table 1. As might have been expected, when intensive message-passing distributed applications are executed (sintree and is), no effect in the local task is produced. On the other hand, a high slowdown is introduced if intensive CPU distributed tasks are executed (mg). As was to be expected, the explicit model has a great impact on the local task and even more in the PRIO and PRIOSPIN cases.
5
Conclusions and Future Work
In a PVM environment made up of a NOW of Linux nodes, we have implemented and discussed different coscheduling techniques and compared their performance. Also, we have discussed their main advantages and drawbacks. We are interested in developing a dynamical model and new coscheduling techniques for that environment.
References 1. Ousterhout, J.K.: Scheduling Techniques for Concurrent Systems. In Third International Conference on Distributed Computing Systems. (1982) 22–30.
1170
Francesc Solsona et al.
2. Arpaci, R.H., Dusseau, A.C., Vahdat, A.M., Liu, L.T., Anderson, T.E., Patterson, D.A.: The interaction of Parallel and Sequential Workloads on a Network of Workstations. SIGMETRICS’95. (1995). 267–278. 3. Arpaci, R.H., Dusseau, A.C., Culler, D.E., Mainwaring, A.M.: Scheduling with Implicit Information in Distributed Systems. SIGMETRICS’98. (1998). 4. Dusseau, A.C., Arpaci, R. H., Culler, D. E.: Effective Distributed Scheduling of Parallel Workloads. SIGMETRICS’96 . (1996). 5. Solsona, F., Gin´e, F., Hern´ andez, P., Luque, E.: Synchronization methods in distributed processing. IASTED AI’99. (1999) 471–473. 6. Bailey, D. et al.: The NAS parallel benchmarks. International Journal of Supercomputer Applications. vol. 5 no. 3 (1991) 63–73.
A Jini-Based Prototype Metacomputing Framework Zoltan Juhasz1,2 and Laszlo Kesmarki1 1
2
University of Veszprem, Department of Information Systems, Veszprem, Hungary, [email protected] University of Exeter, Department of Computer Science, United Kingdom
Abstract. This paper investigates the possible use of Jini technology for building Java-based metacomputing environments. The paper presents the structure of a prototype Jini metacomputing system. The overall working mechanism and some details of the implementation are described. It is shown that Jini is a suitable technology that deserves the interest of the metacomputing research community.
1
Introduction
The past few years generated tremendous interest in metacomputing. Several research groups world wide are working on how to provide seamless access to the vast number of computers connected by the Internet, to create better and more useful computing services, and to improve the performance of metacomputing systems. Different approaches, languages and paradigms have been used [3][4][5][6][7][8]. Jini technology [2] is a new Java-based technology to create autonomous, ad hoc networks of digital devices. It is a service-based system featuring automatic discovery of the network and its available services, fault tolerance, transactions and distributed event mechanism. The purpose of this paper is to report on our ongoing work whose aim is to investigate the suitability of Jini as a metacomputing technology, and outline the structure and underlying working mechnism of one possible Jini metacomputing system.
2
Metacomputing Systems
The term metacomputing system [9] now commonly refers to systems composed of potentially thousands of personal computers, workstations or supercomputers to solve compute-intensive problems or provide better solutions in geographically distributed work environments. Compared to parallel systems, metacomputing environments open up a set of new problems and issues that must be dealt with. Successful systems must cope with partial failure, geographical distance, large network latencies, different A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1171–1174, 2000. c Springer-Verlag Berlin Heidelberg 2000
1172
Zoltan Juhasz and Laszlo Kesmarki
language environments, hardware platforms and operating sytems, and must provide solutions for security problems, fault tolerance and portability [1]. Most metacomputing systems are service based. Services provide distinct groups of functionality and are generally used for resource management (allocating/scheduling tasks, load-balancing, process management), communication (eg message passing), security and authentication, system status (monitoring the availability and system load of components), and accessing remote data. These are nicely represented by the service set of the Globus [3] project, such as the Resource management, Communication, Security, Information, Health and Status, Remote Data Access, and Executable Management services.
3
A Minimal Metacomputing System
Our current plan is to create minimal system (a testbed), which we can use to experiment with various features of Jini and consequently, evaluate its general suitability for building metacomputing systems. We expect the prototype system to be able to (i) monitor the current state of the computing resources, (ii) identify which resources it should use for parallel tasks and (iii) map and execute the tasks on the selected resources. We use the following three participants, Client, Broker and Host. The Client represents the parallel application to be run on the system. A client first has to find available brokers then request the execution of the program from one or more. A Broker is responsible for two main actions. It must be able to (i) monitor resources and (ii) select a subset of them for executing the client’s tasks. Monitoring is performed through the Host services that provide crucial state information and are able to execute tasks allocated to them by the broker service. 3.1
The Operation of the System
The system is depicted in Fig 1. Each processor runs a host service that implements a generic compute engine and contains methods that manage processes. The host service also contains static attributes describing the type of computer it represents. The service is lease based, and the attributes are updated on each lease renewal. Brokers discover potential computing resources by finding Host services in the registrars and —after downloading the service— extract information from the host services. In the simplest form of implementation, the Host service will run entirely in the broker’s virtual machine. Note that it is possible to create host services that provide dynamic, time-varying information (e.g. change in load level). In this case, the downloaded host service cannot run exclusively in the VM of the broker, as it has to obtain information from the originator host. Thus, in this case, the host service will only provide a proxy for the service that stays on the host.
A Jini-Based Prototype Metacomputing Framework
1173
Service registrar
Resource Broker
Resource Broker
Service registrar
Client (program to run)
Registers servic e Uses service service object
Host service Machine A
Host service
Host service
Machine B
Machine C
Fig. 1. Movement of service and application objects during the operation of the system
4
Implementation of the Prototype System
Each service registers with the registrar through the registrar object’s register() method. This has two input parameters, a ServiceItem object describing the service and a lease period given in milliseconds. A ServiceItem object comprises a service ID, the actual service object representing the service, and an attribute set describing the service for clients during the template-matching based service lookup operation. 4.1
The Host Service
The Host service must implement a generic compute engine. In addition we store static system information such as host URL, the number and types of CPUs, the number and types of network interfaces of the host, network latency and bandwidth, topology, memory size and location. The Host service implements a public Object execute(Task t) method to execute client task t of type Task interface. System and state information are represented as Jini attributes. We have abstracted out attributes such as Processor, Network and Memory for our ptototype. 4.2
The Broker Service
The responsibility of the Broker service is to take an execution request from the Client, find suitable machines for execution, then allocate and run tasks on these machines. Allocation is based on requirements received with the execution request (e.g. required performance, maximum number of tasks, maximum allowed cost of execution to be payed for, etc) as well as on machine information extracted from Host services. In our prototype the priorities in task scheduling is
1174
Zoltan Juhasz and Laszlo Kesmarki
to minimize the number of processors and maintain geographical locality of the selected machines. This can only be violated if such a parallel supercomputer is available that can handle the execution of the complete problem itself. During execution, the broker first looks for available Host services, then registers with the lookup service to become available to clients. The main methods of the Broker are findMachines() and execute(). The method findMachines() takes a list of execution requirements from the Client and returns a set of machine list–cost pairs. If the set is not empty, the Client can invoke the execute() method either on specific hosts or on arbitrary ones selected by the broker by specifying allocation priorities such as minimum cost, maximum speed.
5
Conclusions
This paper presented early results of the Jini metacomputing project. It outlined the structure of a possible Jini service-based metacomputing framework and described some details of the Host and Broker service implementation. Our short term plan is to complete the implementation of the framework and experiment with it. There are several issues that will be have to be addressed in the future. Scalability of the system requires that increasing the size of the system does not create bottlenecks in service access and use. We will explore possibilities of creating a network of registrars as well as an array of brokers to avoid the system relying on a single or small number of registrars and brokers. Automatic execution mode will require negotiation between client and broker and among brokers, therefore we will investigate the potential of using multiagent techniques in coordinating program execution.
References 1. M. Baker and G. Fox, Metacomputing: Harnessing Informal Supercomputers in R. Buyya (ed.) High Performance Cluster Computing Vol. 1, Prentic Hall, 1999. 2. W. Keith Edwards, Core JINI, Prentice Hall 1999. 3. I. Foster and C. Kesselman, The Globus project: a status report,Future Generation Computer Systems 15 (1999) pp 607-621. 4. A. Grimshaw, W. Wulf et al. The Legion Vision of a Worldwide Virtual Computer. Communications of the ACM, vol. (40)1, January 1997. 5. T. Haupt, E. Akarsu, G. Fox and W. Furmanski, Web based metacomputing, Future Generation Computer Systems 15 (1999) pp 735-743. 6. M. Migliardi, V. Sunderam, The Harness Metacomputing Framework, in Proc. of the Ninth SIAM Conference on Parallel Processing for Scientific Computing, S. Antonio (TX), USA, March 22-24, 1999. 7. M.O. Neary, B.O.Christiansen, P. Capello and K.E.Schauser, Javelin: Parallel Computing on the Internet, Future Generation Computer Systems 15 (1999) pp 659-674. 8. L.F.G Sarmenta and S. Hirano, Bayanihan: building and studying a web-based volunteer computing systems using Java, Future Generation Computer Systems 15 (1999) pp 675-686. 9. L. Smarr and C. E. Catlett, Metacomputing, Communications of the ACM, Vol. 35, No. 6, 1992, pp 44-52.
SKElib: Parallel Programming with Skeletons in C Marco Danelutto and Massimiliano Stigliani Department Computer Science – University of Pisa Corso Italia, 40 – I-56125 Pisa – Italy [email protected], [email protected]
Abstract. We implemented a skeleton library allowing the C programmer to write parallel programs using skeleton abstractions to structure and exploit parallelism. The library exploits a SPMD execution model in order to achieve the correct, parallel execution of the skeleton programs (which are not SPMD) onto workstation cluster architectures. Plain TCP/IP sockets have been used as the process communication mechanism. Experimental results are discussed that demonstrate the effectiveness of our skeleton library.1 Keywords: skeletons, parallel libraries, SPMD.
1
Introduction
Recent works demonstrated that efficient parallel applications can be easily and rapidly developed by exploiting skeleton based parallel programming models [5, 16, 15, 3]. With these programming models, the programmer of a parallel application is required to expose the parallel structure of the application by using a proper nesting of skeletons. Such skeletons are nothing but a known, efficient way of exploiting particular parallel computation patterns via language constructs or library calls [9]. The skeleton implementation provided by the support software takes care of all the implementation details involved in parallelism exploitation (e.g. parallel process network setup, scheduling and placement, communication handling, load balancing). Therefore the programmer may concentrate his efforts on the qualitative aspects of parallelism exploitation rather than on the cumbersome, error prone, implementation details mentioned above [14]. Currently available skeleton programming environments requires considerable programming activity in order to implement the skeleton based programming languages [3, 5]. Skeleton programs are compiled by generating the code of a network of cooperating processes out of the programmer skeleton code, and providing all the code necessary to place, schedule and run the processes on the processing elements of the target architecture. When compiling skeleton programs, the knowledge derived from the skeleton structure of the user program is exploited via heuristics and proper algorithms. Eventually, efficient code is obtained, exploiting the peculiar features of the target architecture at hand [14]. 1
This work has been partially funded by the Italian MURST Mosaico project.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1175–1184, 2000. c Springer-Verlag Berlin Heidelberg 2000
1176
Marco Danelutto and Massimiliano Stigliani
Capitalizing on the experience we gained in the design of such environments [2, 8], we designed a library (SKElib) that allows the programmer to declare skeletons out of plain C functions, to compose such skeletons and to command their evaluation by a simple C library call statement. The library allows the programmer to structure parallel computations whose pattern do not correspond to a skeleton by using standard Unix mechanisms. The library can be used on any cluster of workstations (COW) running Unix OS. This kind of architecture is commonly available and delivers very high performance at a price which is a fraction of the price of other, specialized, (massively) parallel architectures [4]. In this paper we first describe the choices taken in the library design (Sec. 2), then we discuss the implementation details of SKElib (Sec. 3) and eventually we present the performance results we achieved when running skeleton programs written with SKElib on a workstation cluster (Sec. 4).
2
Library Design
The skeleton set provided to the parallel application programmer by SKElib includes a small number of assessed skeletons [1, 16, 15]: farm a task farm skeleton, exploiting parallelism in the computation of a set of independent tasks appearing onto the input stream. pipe a skeleton exploiting the well known pipeline parallelism. map a data parallel skeleton, exploiting simple data parallel computations and, particularly, computations where a result can be obtained combining a set of partial results. These results, in turn, are computed by applying a given function on all the data items obtained by partitioning an input data item into a set of independent data partitions. while a skeleton modeling iterative computations. seq a skeleton embedding sequential code in such a way that the code can be used as a parameter of other skeletons. The skeletons we provided in SKElib can be nested, e.g. a pipe may have a farm stage that, in turn, computes tasks by exploiting pipeline parallelism. Such a skeleton set allows significant parallel applications to be developed and, in the meanwhile, is simple enough to be reasonably handled2 . All these skeletons are provided to the programmer by including functions in the library that can be called to declare both sequential code to be used as a skeleton parameter (i.e. as a pipeline stage, via the SKE_SEQ function) and pipe, farm, map and while skeletons having other skeletons as parameters (SKE_PIPE, SKE_FARM, SKE_MAP and SKE_WHILE functions). A different function is implemented in the library to ask the evaluation of a skeleton expression (SKE_CALL). This function takes parameters denoting where the input stream (the sequence of input data sets) and the output stream (the 2
one of our main aims was to demonstrate the feasibility of the library approach to skeleton implementation
SKElib: Parallel Programming with Skeletons in C task farm parallelism degree
"emitter" process: schedule tasks on worker processes (on demand)
E input stream
w
compute task farm worker function
w
w
1177
"collector" process: gathers results from worker processes (reordering)
C
output stream processes
communication channels
Fig. 1. Implementation template of the farm skeleton
sequence of the data results) have to be taken/placed. Such parameters, formally denoting the skeleton input and output streams, are plain Unix files. The programmer can ask the library to evaluate a skeleton expression taking input data from a file and delivering output data to another file. But he can also ask SKElib to evaluate the same expression by taking input data from the output of an execed Unix process whose output has been redirected to a named pipe, just by using the named pipe as the skeleton input stream. Further parameters must be supplied to SKE_CALL denoting the processing elements (PE) that can be used in the parallel skeleton computation. We choose to implement skeletons by exploiting implementation template technology [2, 14]. Following this approach, each skeleton is implemented by setting up a network of communicating sequential processes arranged as described in a template library. This library holds an entry for each skeleton supported. The entry completely describes the structure of an efficient process network that can be used to implement the skeleton on the target architecture (i.e. the entry describes how many processes must be used, where the processes have to be placed, how they have to be scheduled, etc.). As an example, Figure 1 shows the process network relative to the implementation template of the farm skeleton used in SKElib. This template (as the other templates used in SKElib) does not maintain the input/output ordering of tasks. Results relative to input task i may appear on the output stream after the results relative to input task i+k (k > 0). SKElib just preserves ordering between application input and output streams: results are delivered (using a proper buffering algorithm) on the output stream in the same order as the corresponding input data appeared onto the input stream. SKElib uses plain TCP/IP Unix sockets to implement interprocess communications. We could have used any one of the communication libraries available in the Unix environment (e.g. MPI [13]), but the usage of such kind of libraries can prevent the user from using some of the typical Unix mechanisms and one of our goals was to provide a seamless skeleton framework. The overall goal we achieved with SKElib has been to make available to the Unix programmer a new paradigm to structure common parallelism exploitation patterns while leaving the programmer the ability to “hand code” those parts of the parallel application that do not fit any of the skeletons in the library. It’s worth pointing out that in classical skeleton frameworks the programmer is com-
1178
Marco Danelutto and Massimiliano Stigliani user functions: read first parameter write the second one
library types & function prototypes
sequential skeleton declaration
function input data size
#include "ske.h"
farm decl.
pipe decl.
extern void f(F_IN * in, F_OUT * out); extern void g(F_OUT * in, G_OUT * out);
function output data size
balancing policy int main(int argc, char * argv[]) { SKELETON * seqf, * seqg, * farmf, * the_pipe; number of parallel worker int n_workers = atoi(argv[1]); processes in the farm ... seqf = SKE_SEQ((FUN *)f, sizeof(F_IN), sizeof(F_OUT)); seqg = SKE_SEQ((FUN *)g, sizeof(F_OUT), sizeof(G_OUT)); number of stage farmf = SKE_FARM(n_workers, seqf, BALANCING); in the pipeline the_pipe = SKE_PIPE(2,farmf,seqg); ... n_hosts = atoi(argv[2]); SKE_CALL(the_pipe,OPTIMIZE,"input.dat","output.dat", n_hosts,"alpha1","alpha2","alpha3","alpha4","alpha5"); ... return(0); skeleton } expression process graph input output call hosts used to optimizations stream stream (evaluation) run templates file file options
Fig. 2. Sketch of sample application code
pletely and consistently assisted in the implementation of parallel applications whose parallel structure fits some particular (nesting of) skeleton(s), but has no way to implement some parallel exploitation pattern which is not captured by the skeletons provided by using the other typical mechanisms supported by the hardware and software architecture at hand.
3
SKElib Implementation
Figure 2 sketches a simple parallel application written using SKElib. The application computes a stream of results out of a stream of input data. Each input data is used to compute a partial result via the function f and this partial result is used to compute the final result via the function g. This computation structure is naturally modeled by the pipe skeleton. By taking into account that the computation of function f is much more expensive than computation of function g, the programmer has inserted a farm in the first pipeline stage. Therefore, the program will read – in a cycle – an input data set from the input stream (file input.dat, actually), deliver it to a worker process of the farm that computes an intermediate result applying the f function, and eventually deliver such intermediate result to the process sequentially computing the g function. The process leading to the execution of this skeleton program is outlined in Figure 3. First of all the program is compiled and linked to SKElib code. Then the user runs the program on a node of the target COW. When the control flow reaches the request for skeleton code evaluation (the SKE CALL library call), the library code analyzes the skeleton structure declared by the programmer and
SKElib: Parallel Programming with Skeletons in C
source code
PE
gcc -lSKE ... obj code
SKE lib
PE
run user program SKE_CALL(...)
1179
PE
f f E
derive template structure & run template processes
Cg
f
Fig. 3. Parallel application execution schema
“execs” on the processing elements of the COW all the processes necessary to implement the process network derived from the skeleton code. Functions representing the sequential code executed by skeletons must be supplied as void C functions with two parameters: a pointer to the input data and a pointer to the output data. This restriction on the sequential code format has been introduced to avoid a number of unnecessary buffer copy operations in the library code, while requiring a moderate “extra” programming effort to the user. The processes in the implementation templates of skeletons must be able to execute user supplied code and, in particular, the code embedded in the seq skeletons and all the related code. In order to allow the template code to execute such user code without charging the programmer with the task of following particular formats in the source code, we decided to exploit an SPMD execution model. SKElib wrappes the user main. Therefore, when the user program is run the library main function is executed in place of the user main. If the first command line argument is not the special (reserved) SKEslave string, the SKElib main just calls the user main. Otherwise SKElib takes the complete control of the execution and proper process templates are run instead of the user main, depending on the other command line parameters. Therefore, when the user runs the skeleton program supplying his own command line parameters, his main code is executed. When the user code reaches a SKE_CALL statement, the skeleton program is remotely run on the PEs in the COW. The command line parameters passed to the remote exec command are such that the SKElib main takes definitely the control and calls those parts of the SKElib code that implement the processes that must be run on the remote nodes to implement the skeleton templates. Which template code has to be executed on the nodes is derived by the SKE_CALL code by looking at the information gathered with skeleton declarations. This leads to an SPMD execution model of the skeleton program. The same program is eventually run on all the nodes of the target architecture. The behavior of the program on each node depends on the command
1180
Marco Danelutto and Massimiliano Stigliani 10 ideal scalability
9
Pipeline
8
Farm
Scalability
7
Map
6 5 4 3 2 1 0
1
2
3
4
5 6 Processes (on 10 PEs)
7
8
9
10
Fig. 4. Scalability results of the library line parameters used to invoke the code. Parameters passed to the code executed on the remote nodes are both symbolic and pointer parameters. The symbolic parameters tell SKElib main which process templates are to be executed. The pointer parameters tell the library which user code must be eventually be called by the process templates. Pointer parameters (virtual addresses, actually) can be used because all the nodes eventually will run the same code – the one derived from the compilation of the user code linked with the library code. This mechanism allows any user function to be called on the processing elements of the target architecture without requiring the user to supply the code embedded in the seq skeletons in a particular file/library. In the normal SPMD model, all the processes running on the different processing elements and executing the same code are started at the same time. In our case, one process is started by the user, whereas the other processes are started in the SKE CALL code, via rsh calls. This approach has a drawback: all the user code is actually replied on all the processing elements participating in the computation. In case the user code is large, this may lead to virtual memory waste on the target architecture PEs. The programmer may ask SKElib to optimize the object process graph by specifying a proper parameter in the SKE CALL (in the example of Figure 2 an OPTIMIZE parameter is passed to this purpose). The optimizations currently performed by SKElib mainly concern process grouping on the same processing element. As an example, in a two stage pipeline with both stages exploiting farm parallelism, the collector process of the first farm and the emitter process of the second one are actually merged when the OPTIMIZE parameter is specified in the SKE CALL. The resulting process network shows less communication overhead than the original one. We are considering further process graph optimizations. As an example, when the processing elements of the target architecture have to be multiprogrammed (due to the large number of processes derived from the user
SKElib: Parallel Programming with Skeletons in C
1181
code) communicating processes will be mapped to the same processing element, in such a way that a lower network overhead is paid. Overall, SKElib has been implemented on a Linux COW using standard Unix mechanisms, namely rsh to run processes on remote processing elements and TCP/IP BSD sockets to perform inter-process communications. As a consequence, the library can be used on any workstation network supporting these mechanisms. The only thing to do in order to port the library across different COWs is to compile the library code. Due to the SPMD execution model adopted, however, all the processing elements in the COW must have the same processor (i.e. the same instruction set) as well as the same operating system. This implies that the library is not to guaranteed to work on a COW with processing elements running different (versions of the) operating system.
4
Experimental Results
We run some experiments to test SKElib performance. The experiments concerned both the evaluation of the absolute performance achieved by SKElib and a comparison with the performance values achieved by using Anacleto, the P3L compiler developed at our Department [8]. P3L is the skeleton based parallel programming language developed in Pisa since 1992 [2]. Anacleto compiles P3L programs generating C + MPI [13] code that can be run on a variety of parallel machines and workstation clusters. All the experiments have been performed using Backus as the target architecture. Backus is a Beowulf class workstation cluster with eleven PC based workstations. Each PC sports a 233Mhz Pentium II with 128Mbyte of main memory, 2Gbyte of user disk space and a 100Mbit Fast Ethernet network interface card. The PCs run Linux (kernel 2.0.35) and are interconnected by a 3Com SuperstackII 100Mbit Ethernet switch. All the PCs are “dedicated” in that no other computations where performed on the machines during the experiments but our skeleton processes and the usual system processes. When executing skeleton code on the COW, we achieved effective speedups with respect to the execution of the corresponding sequential code. Concerning scalability, the typical results we achieved are summarized in Figure 4. The Figure plots the scalability3 s achieved in the execution of medium to coarse grain skeleton programs written using our library and exploiting parallelism by using a single skeleton. The x-axis is relative to the number of processes used to implement the program. The pipeline program used a 10 stage pipe. The farm and map have been compiled to use a number of processes ranging from 1 to 20. SKElib is able to schedule more than a single template process on a single processing element (currently the scheduling is performed with a round-robin algorithm) in such a way that communication and computation times relative to different processes can be overlapped and therefore a certain degree of communication 3
s= PEs
T (1) , T (n)
where T (i) is the time spent in computing the parallel program using i
1182
Marco Danelutto and Massimiliano Stigliani 9 SKElib map SKElib farm
8
P3L/Anacleto map
Scalability
7
P3L/Anacleto farm
6
5
4
3
5
6
7 8 Processes (on 10 PEs)
9
10
Fig. 5. Performance: SKElib vs. Anacleto latency hiding is achieved. As a consequence, the runs requiring more than 10 processes perform node multiprogramming on the COW PEs. From Fig. 4 it is clear that the skeletons implemented in the library achieve quite a good efficiency. Using a number of processes that do not exceed the number of processing elements available, we achieve an efficiency which is more that 80%. However, as the library has been designed to take advantage of the node multiprogramming facilities offered by Unix, we also run skeleton programs implemented with a higher number of processes than the actual number of processing elements in the machine. In this case we achieved scalability values larger that 9.8 on 10 PEs. Such results where achieved running map and farm skeletons using up to 20 processes. It’s worth pointing out that in order to increase the parallelism degree of a skeleton – and, as a consequence, the number of processes used to implement the skeleton – the only thing the programmer must do is to specify the proper parameter value in the declaration of the skeleton. In the code of Figure 2, the programmer specifies that the farm must have a parallelism degree of n workers just passing this number as the first parameter of the SKE CALL function call. Figure 5 shows the results achieved by our library and the results achieved running similar skeleton programs written in P3L and compiled using Anacleto. The performance Figures are similar, but our library sports slightly better results. This is in part due to the fact that Anacleto generates C + MPI code and therefore communications are performed via MPI primitives that, in turn, are implemented on top of the same sockets we directly used to implement SKElib communications. However the result is quite satisfactory. SKElib presents some advantages with respect to the use of P3L/Anacleto, in particular concerning the possibility to use mixed skeleton/standard parallelism exploitation mechanisms, and still delivers a performance which is slightly better than the one delivered by the Anacleto runs.
SKElib: Parallel Programming with Skeletons in C
1183
All the results discussed above have been achieved by using either simple programs using a single parallelism exploitation pattern/skeleton (just to test the different skeleton implementation features) or simple applications templates (image processing, simple numerical algorithms such as matrix multiplication or mandelbrot set computation). In this way we have been able to validate the SKElib design and to draw complete performance figures. Currently we are developing full applications in order to better evaluate both the expressive power provided to the programmer by SKElib and the performance achieved when exploiting complex parallelism exploitation patters using significant skeleton compositions.
5
Related Work & Conclusions
Many efforts are being made to implement skeleton based parallel programming environments. Our research group in Pisa is active in the development of SkIE [16, 3]. Serot is currently developing a skeleton framework for image processing [15]. Darlington’s group at the Imperial College in London also gave substantial contributes to skeleton research [11] as well as the group of Burkhart [7]. [5] discusses a skeleton framework mainly focused on data parallelism exploitation. [6] studied skeletons in a functional programming framework and we also provided a “skeleton embedding” within Ocaml, which is the ML dialect of INRIA [10]. All these projects (but the one described in [10], that actually supports skeletons in Ocaml via a library) either provide skeletons as new programming languages and/or language extensions or they rely on a heavy compile process in order to compile skeleton programs to parallel object code. There is no previous attempt to provide a skeleton programming environment via a plain C library, to our knowledge. Anyway, we exploited experiences performed in all these projects in the design of the template system used to implement the skeletons in SKElib (in particular the approach used to implement skeletons in SKElib is almost completely derived from the projects discussed in [2, 11]). Different project exists, not directly involved with the skeleton research track, aimed at making available libraries providing the programmer with suitable tools to structure parallelism exploitation within applications. As an example, [12] discusses a library allowing data parallel computations to be expressed using the classes provided within a C++ library. We owe these projects, as well as the project summarized in [10], for some of the ideas used to implement data parallel skeleton templates and the general library structure. In this work, we discussed the design and implementation of SKElib, a library providing the C Unix programmer with a simple way to implement parallel applications using skeletons. SKElib has been developed on a Linux COW. Using SKElib, parallel applications can be written that exploit parallelism both by using skeletons and by using standard Unix concurrency mechanisms. Skeletons can be used to program those application parts that match standard parallelism exploitation patterns. Standard Unix mechanisms can be used to program those application parts that do not match the skeleton provided. We showed exper-
1184
Marco Danelutto and Massimiliano Stigliani
imental results demonstrating that the approach is feasible in terms of performance. We also showed that the performance results achieved by SKElib are slightly better than the results obtained by using existing skeleton compilers.
References [1] P. Au, J. Darlington, M. Ghanem, Y. Guo, H.W. To, and J. Yang. Co-ordinating heterogeneous parallel computation. In L. Bouge, P. Fraigniaud, A. Mignotte, and Y. Robert, editors, Europar ’96, pages 601–614. Springer-Verlag, 1996. [2] B. Bacci, M. Danelutto, S. Orlando, S. Pelagatti, and M. Vanneschi. P3 L: A Structured High level programming language and its structured support. Concurrency Practice and Experience, 7(3):225–255, May 1995. [3] B. Bacci, M. Danelutto, S. Pelagatti, and M. Vanneschi. SkIE: a heterogeneous environment for HPC applications. Parallel Computing, 25:1827–1852, December 1999. [4] M. Baker and R. Buyya. Cluster Computing at a Glance. In Rajkumar Buyya, editor, High Performance Cluster Computing, pages 3–47. Prentice Hall, 1999. [5] George Horatiu Botorog and Herbert Kuchen. Efficient high-level parallel programming. Theoretical Computer Science, 196(1–2):71–107, April 1998. [6] T. Bratvold. Skeleton-Based Parallelisation of Functional Programs. PhD thesis, Heriot-Watt University, 1994. [7] H. Burkhart and S. Gutzwiller. Steps Towards Reusability and Portability in Parallel Programming. In K. M. Decker and R. M. Rehmann, editors, Programming Environments for Massively Parallel Distributed Systems, pages 147–157. Birkhauser, April 1994. [8] S. Ciarpaglini, M. Danelutto, L. Folchi, C. Manconi, and S. Pelagatti. ANACLETO: a template-based P3L compiler. In Proceedings of the PCW’97, 1997. Camberra, Australia. [9] M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computations. Research Monographs in Parallel and Distributed Computing. Pitman, 1989. [10] M. Danelutto, R. Di Cosmo, X. Leroy, and S. Pelagatti. Parallel Functional Programming with Skeletons: the OCAMLP3L experiment. In ACM Sigplan Workshop on ML, pages 31–39, 1998. [11] Darlington, J. Guo, Y.K, H.W. To, and Y. Jing. Functional skeletons for parallel coordination. In Proceedings of Europar 95, 1995. [12] E. Johnson, D. Gannon, and P. Beckman. HPC++: Experiments with the Parallel Standard Template Library. In Proceedings of the 1997 International Conference on Supercomputing, pages 7–11, July 1997. [13] M.P.I.Forum. Document for a standard message-passing interface. Technical Report CS-93-214, University of Tennessee, November 1993. [14] S. Pelagatti. Structured Development of Parallel Programs. Taylor & Francis, 1998. [15] J. Serot, D. Ginhac, and J.P. Derutin. SKiPPER: A Skeleton-Based Parallel Programming Environment for Real-Time Image Processing Applications. In Proceedings of the 5th International Parallel Computing Technologies Conference (PaCT-99), September 1999. [16] M. Vanneschi. PQE2000: HPC tools for industrial applications. IEEE Concurrency, 6(4):68–73, 1998.
Token-Based Read/Write-Locks for Distributed Mutual Exclusion Claus Wagner1 and Frank Mueller2 2
1 Computer Science Department, TECHNION, Haifa, Israel 32000 Humboldt University Berlin, Institut f. Informatik, 10099 Berlin, Germany phone: (+49) (30) 2093-3011, fax: -3010 [email protected]
Abstract. The contributions of this paper are twofold. First, a protocol for distributed mutual exclusion is introduced using a token-based decentralized approach, which allows either multiple concurrent readers or a single writer to enter their critical sections. This protocol utilizes a dynamic structure incorporating path compression to keep the messages overhead low resulting in an average complexity of O(log n) messages per request. Second, this protocol is evaluated in comparison with another protocol that uses a static structure instead of dynamic path compression. The measurements show that although concurrent readers may require at most one additional message per entry, the concurrent execution of critical sections results in faster responses of up to 30% for short critical sections. For longer critical sections, savings in the overall execution time increase with the fraction of readers to up to 50%. In particular applications with large fractions of readers, e.g., database queries, may exploit these benefits. The results further indicate that problems with fine-grained parallelism are more suitable for the dynamic protocol proposed here while the static protocol used for comparison performs equally well for coarse-grained parallelism. Overall, reader/writer distinction provides promising benefits in both cases.
1
Introduction
Common resources in a distributed environment may require that they are used in mutual exclusion. This problem is similar to mutual exclusion in a sharedmemory environment. However, while the shared memory architectures generally provide atomic instructions (e.g., test-and-set) to implement mutual exclusion, such provisions do not exist in a distributed environment. Furthermore, commonly known mutual exclusion protocols for shared-memory environments that do not rely on hardware support still require access to shared variables. In distributed environments, mutual exclusion is provided via a series of messages passed between nodes that are interested in a certain resource. Several protocols to solve mutual exclusion for distributed systems have been developed [1]. They can be distinguished by their approaches as token-based and non-tokenbased. The former ones may be based on broadcast protocols or they may use logical structures with point-to-point communication. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1185–1195, 2000. c Springer-Verlag Berlin Heidelberg 2000
1186
Claus Wagner and Frank Mueller
In this work, we assume a fully connected network (complete graph). Network topologies that are not fully connected can still use our protocol but will have to pay additional overhead when messages from A to B have to be relayed via intermediate nodes. Second, we assume reliable message passing (no loss, duplications, or modifications of messages) but we do allow out-of-order message delivery with respect to a pair of nodes, i.e., if two messages are sent from node A to node B, then they may arrive at B in a different order than they were sent. Our assumption is that local operations are several orders of a magnitude faster than message delivery since this is the case in today’s networks and the gap between processor speed and network speed still seems to widen. Thus, our main concern is to reduce the amount of messages at the expense of local data structures and local operations. We introduce a new protocol to provide mutual exclusion in a distributed environment that distinguishes read and write requests. The protocol is based on some ideas by Naimi et al. [10], i.e., a decentralized protocol that did not distinguish readers and writers. We explain the design of our approach, illustrate it by examples and provide the pseudocode for it. Measurements in a simulation testbed are obtained to evaluate the performance of our protocol.
2
Related Work
A number of protocols exist to solve the problem of mutual exclusion in a distributed environment. Chang [1] and Johnson [4] give a more detailed overview and compare the performance of such protocols. Non-token based protocols establish consensus between nodes before entry to a critical section is granted and often employ global logical clocks and timestamps [6]. Token-based protocols link permission of entry to token ownership. The broadcast-based subclass of protocols does not require specific communication paths since all nodes always receive a message. Token-based protocols with logical structures constrain messages to certain paths, which may be static or they may change dynamically. Our work focuses on this last class of protocols since their message complexity of O(log n) is lower than that of most of the above approaches, which require at least O(n) messages on the average for a request in a network of n nodes. Exceptions are √ Maekawa with O( n) realized by an evenly distributed set of nodes over their quorum (subset of nodes that have to grant access to a requester) [8] and Kumar with O(n0.63 ) building on the former approach [5]. Raymond proposed an O(n) quorum protocol [12], which was improved by Srimani and Reddy [14]. However, quorum protocols with k concurrent entries to the critical section do not provide the distinction between an arbitrary number of readers and writers of our work. Another protocol by Raymond [13] utilizes a static tree for request propagation and utilizes local queues for requests. This work was extended by Neilsen and Mizuno [11] to replace these local queues by a distributed queue. In contrast, Chang, Singhal and Liu [2] use a dynamic tree similar to Naimi et al. [10]. The protocol makes use of path compression similar to independent work on memory consistency protocols [7]. Our work was influenced by the approaches
Token-Based Read/Write-Locks for Distributed Mutual Exclusion
1187
of Raymond and Naimi et al.. Our protocols are based on the same assumptions and use similar data structures for write requests, which also ensures that the complexity of the original protocols is preserved in our solution in the absence of read requests. The additional support of concurrent readers in our protocols then only adds a constant overhead of one message for a read chain, if this chains build up. Hence, the order of complexity of our protocols in terms of the number of messages remains the same as within the original protocols.
3
Dynamic Reader/Writer Protocol for Mutual Exclusion
The dynamic protocol utilizes a directed tree-like structure. The edges form a chain leading new requests from node R to the last requester L (or the token holder if no requests were pending). While a request is in transit, intermediate nodes set their edges to point to the new requester R, thereby providing path compression, i.e., future requests will propagate directly to R from any of the intermediate nodes similar to Naimi et al. [10]. An example is depicted in Fig. 1 where a (read) request from A is sent via B to T . The intermediate nodes B and T have edges directed at A afterwards. If T is still engaged in the critical section with a write lock, the read request cannot be served yet. Instead, T creates a next pointer (dashed edge) to A indicating the next recipient of the token. Once T exits its critical section, it sends the token to A and removes the next edge. A proceeds with its critical section under read protection. At this time, C issues a read request that is sent to A via B resulting in new edges from A and B to C. A responds by sending the token to C even before exiting its critical section because both A and C may execute concurrently in their critical sections under read protection. In addition, A registers C as the next reader (dotted edge) and C notes the fact that it is the last reader in a chain of readers. Node C will hold on to the token until (1) a request from another reader arrives next or (2) C and A exit their critical sections. The former case allows more concurrent readers to be served if they arrive ahead of writers. The latter case ensures that a writer will only be served after all reads have completed. Once A exits its critical section, it sends an acknowledgement to C, which must be received by C before the token may be forwarded to a writer. Read−Req from A
Token to A
T B A
T D
C
Read−Req from C
A
B
T D
C
B A
Fig. 1. Simple Example
T D
C
A
B
D C
1188
Claus Wagner and Frank Mueller
The example illustrates that the protocol requires directed edges that requests travel along. It includes a distributed next queue of pending requests whose source is the token holder while the sink is the last requester. A distributed read queue links concurrent readers starting at the earliest requester still engaged in a critical section via consecutive requesters to the sink of the reader chain. Consecutive members of the read chain are either in the critical section or have not received an acknowledgement from their predecessor in the read chain yet. When a node issues multiple read requests in a row, the read chain may be circular imposing the necessity to log pending recipients of acknowledgements in a local FIFO queue. Requests are always handled in the order that they arrive at the tail of the request queue, i.e., a write request always terminates a chain of readers and subsequent read requests are served after the write. This ensures fairness and avoids starvation. The following notation is used in order to reflect these requirements when describing the protocol: Token is true if nobody is engaged within a critical section. Dir represents an edge that points towards the last requester or, in the absence of requests, to the token holder, creating a distributed tree (solid edges). Next points to the next requester of the token. The next chain represents a distributed queue of pending requests (dashed edges). Next−readers represents a FIFO list of readers with outstanding acknowledgements. The next readers over all nodes form a chain of concurrent readers that will be reduced successively by acknowledgements (dotted edges). Pending−Acks counts the number of acknowledgements a node still expects. When positive, a node may only forward the token, if it possesses it, to a reader since other readers are still active. Only when the value because zero, may the token be forwarded to a writer. Token−mode is write if a writer is engaged in a critical section. The value is read if the last reader (of the reader chain) is in a critical section or multiread if the token has already been sent to a concurrent reader node. It is undef in any other case. Next−mode is read/write if next points to the next R/W requester, respectively. The pseudocode of the protocol is depicted in Fig. 2. Write requests are handled similar to the protocol by Naimi et al. [10]. Next, the relation between the abstract operational model and the actual implementation shall be discussed. At initialization, all nodes point via dir to the start node, which is the token holder. Notice that the edge of the token holder to himself is omitted in the example since it represents a special case beyond the tree structure. A lock request for a locally unused token can be served right away. All other requests propagate along the dir edges while the requester clears his dir pointer since he is the last requester waiting for the token to arrive. Once the token arrives, the flag to indicate if the node should “expect an acknowledgement” is set according to the value piggybacked with the token message and the number of pending acknowledgements may be incremented accordingly. In case of concurrent reads, the token is forwarded to the next reader and the mode is set to multiread.
Token-Based Read/Write-Locks for Distributed Mutual Exclusion
1189
token = (self == start); // TRUE if token locally available dir = start; // edge with changing destination for requests propagation next = NULL; // next receiver of token (⇒ dist. Q of unserved requesters) // FIFO list of next readers (⇒ dist. Q of conc. readers) next−readers = φ; // number of outstanding acknowledgements pending−acks = 0; // protection of critical section (undef/read/multiread/write) token−mode = undef; // protection for next receiver (undef/read/write) next−mode = undef; PROC lock(mode) IS IF ¬ token THEN PROC receive−request(sender, mode) IS SEND request(self, mode) to dir; IF dir = self THEN dir = self; SEND request(sender, mode) to dir; AWAIT(token(&expect−ack)); ELSE IF expect−ack THEN concurrent−read = (token−mode == pending−acks += 1; read AND mode == read); ENDIF; IF token AND pending−acks == 0 concurrent−read = (mode == read OR concurrent−read THEN AND next−mode == read); SEND token(¬ token) to sender; IF next = NULL AND token = FALSE; concurrent−read THEN IF concurrent−read THEN SEND token(TRUE) to next; token−mode = multiread; append(next−readers, next); append(next−readers, sender); next = NULL; ENDIF; token−mode = multiread; ELSE ELSE next = sender; token−mode = mode; token = FALSE; ENDIF; next−mode = mode; ELSE ENDIF; token−mode = mode; ENDIF; ENDIF; dir = sender; token = FALSE; END receive−request; END lock; PROC receive−ack IS PROC unlock IS pending−acks -= 1; IF token−mode == multiread AND IF pending−acks > 0 OR pending−acks == 0 THEN ¬ empty(next−readers) AND SEND ack to head(next−readers); token−mode == undef THEN delete−head(next−readers); send ack to head(next−readers); ENDIF; delete−head(next−readers); IF next = NULL AND ELSE IF token AND pending−acks == 0 pending−acks == 0 THEN AND next = NULL THEN SEND token(FALSE) to next; send token(FALSE) to next; next = NULL; next = NULL; ELSE IF token−mode = multiread THEN token = FALSE; token = TRUE; ENDIF; ENDIF; END receive−ack; token−mode = undef; END unlock;
Fig. 2. Pseudocode of Read/Write-Lock Protocol
1190
Claus Wagner and Frank Mueller
Otherwise, the request mode is stored before entering the critical section. The token flag is also cleared during the critical section indicating that it is in use. Upon the end of the critical section (unlock), several cases are distinguished. Readers in the chain (except for the tail of the chain) send an acknowledgement to the next member of the chain, thereby reducing the chain if they have already received an acknowledgement from their predecessor. If a next requester exists and all read requests have completed (no pending acknowledgements), the token is sent to the next requester. Otherwise, the token is marked as locally available unless it was already sent to a concurrent reader at an earlier time. A receiver of a request from some sender also has to distinguish certain cases before changing its dir edge to the sender. If the receiver is not the last requester, then he simply forwards the request with the sender’s id along the directed edges. Otherwise, one of two cases may apply. If the token is available and all readers have completed (no acks pending) or if the request is for a concurrent read, the receiver sends the token to the sender, clears the token status and records the next concurrent reader if necessary. The piggybacked flag is true if the new request represent a concurrent read. If neither the token was available nor was the request for a concurrent read, the sender is logged as the next requester. Upon reception of an acknowledgement, the number of pending acknowledgments is decremented and an acknowledgement is sent to the next reader if this had not already been done in the unlock operation. Otherwise, the token is forwarded to the next requester if all acknowledgements have been sent and the read chain has collapsed. Notice that the next requester was either a writer or a reader at the head of another read chain so that the piggybacked value FALSE requires no checks for acknowledgments by this requester (similar as in unlock). The protocol ensures mutual exclusion since either (a) a single writer may own the token or (b) the tail of the read chain owns the token (and predecessors in the read chain may execute their critical read section concurrently). Deadlocks are avoided since requests are forwarded via a tree-like structure to the tail of the requester queue. These requests are then served in the order that they were queued according to the mutual exclusion property, i.e., circular dependencies cannot occur. Starvation is avoided by the FIFO policy of queuing requests, i.e., a new request will be served exactly after all its predecessors in the FIFO queue have been served (again subject to mutual exclusion properties for readers/writers). The protocol has a complexity of O(log n) messages since it uses the same tree structure and messages for requests as Naimi et al. [10] although a constant of one message per reader in a chain is added.
4
Experimental Platform
The experiments were conducted in a simulation environment where distributed nodes were mapped onto processes and communication was performed via asynchronous TCP operations on a uniprocessor. Each process is itself multi-threaded to allow asynchronous communication. Two components implemented as threads require further explanation: A user thread may execute application code includ-
Token-Based Read/Write-Locks for Distributed Mutual Exclusion
1191
ing lock and unlock operations. The communication server receives requests, tokens and acknowledgements. Requests are serviced right away while the other messages are registered. If a user thread awaits such a message, he is activated. Otherwise, the message is stored and when the user thread performs the next await operation, this operation will not block. These operations are implemented using a POSIX mutex and condition variables. In addition, each operation of the protocol depicted in Fig. 2 is executed within mutual exclusion, i.e., the protocol ensures a monitor semantics to keep its internal state consistent, which arbitrates the threads on a node. The monitor is realized by the POSIX mutex also used in conjunction with the condition variables. In prior experiments, the testbed has proved its adequacy to evaluate different mutual exclusion protocols yielding results comparable to prior work [3]. The main experiment consists of a synthetic benchmark as a user thread along the lines of prior work [3, 9]. A loop of i iterations consists of a critical section followed by non-critical code. Durations of the critical section ξ and of non-critical code Γ can be varied to simulate different request rates. Requests are randomized by ±10% around ξ and Γ providing a framework of contention where requests are clustered around a certain point in time but arrive in random orders, as previously shown in [15]. The number of distributed nodes N can be varied and the fraction of readers may be specified to evaluate the protocol. This setup allows the measurement of various aspects. First, the number of messages exchanged per entry (NME) of a critical section can be determined. This allows a comparison between different methods and parameters to relate timing results to message overhead. Second, the average response time for a series of requests is calculated. This provides the means to determine savings per critical section entry. Third, the overall execution time of a test run may be measured. This indicates the overall savings that may be obtained in an application. Varying the proportion of readers and writers gives an insight on how a protocol performs for a varying number of read requests. But it also provides a comparison with the base protocol without distinction between readers and writers, namely when the fraction of read locks is set to 0. The testbed includes the implementation of the dynamic tree-structured protocol for token-based mutual exclusion in a distributed environment, called DVARW in the following. In addition, a token protocol based on a static tree structure has also been implemented. The base protocol by Neilsen and Mizuno [11] has been extended to distinguish read and write locks [15] and is referred to as SVARW in the following. Neilsen and Mizuno allow requests only to travel along the edges of the static tree. An edge may reverse its direction. Thereby, edges form a chain pointing to the last requester. In contrast, Naimi’s approach adopted in this work modifies the sink of edges, not just their direction, as requests propagate along paths. This results in path compression intended to reduce the number of hops for future requests. The modifications to Neilsen and Mizuno’s algorithm to distinguish readers and writers are similar to the protocol described in this paper including token forwarding for concurrent reads before completing a critical section and sending acknowledgements along the chain of
Claus Wagner and Frank Mueller
5
12000
4
10000
response time [µsec]
number of messages
1192
3 2 1 0
0
N=2
20
40 60 80 (a) % of Readlocks N=4
N=8
100
8000 6000 4000 2000 0
0
20
40 (b)
N = 16
N = 32
60
80
100
% of Readlocks N = 64
Fig. 3. Messages and Response Time of the Dynamic Tree Protocol (DVARW)
concurrent readers upon completion of the critical section. The details of the static reader/writer protocol are beyond the scope of this paper (see [15]).
5
Measurements
The measurements were obtained on a 133MHz Pentium under Linux 2.1 executing the benchmark described in the aforementioned testbed. The number of iterations was i = 30. The duration of a critical section was chosen as ξ = 200µsec while the non-critical part was Γ = 2000µsec. This choice reflects that critical sections tend to be short while non-critical code takes up more of the overall execution time. The choice of values was also influenced by prior simulation studies on the performance of distributed mutual exclusion protocols [3, 9]. Fig. 3 depicts the results for the dynamic tree protocol under varying number of nodes (N = 2..64) and varying fractions of read requests. Fig. 3(a) shows that the number of messages depends on the total number of nodes in the system, which is caused by an increasing path length to propagate requests. For a given number of nodes, the number of messages remains constant up to a certain point. This indicates that the dynamic protocol with read/write distinction performs just as well as the base protocol without this distinction, which underlines the qualities of our approach. At some point, read requests dominate (>75%) and the number of messages increases by almost 1 at 100%. This effect is caused by acknowledgement messages that are only sent when read chains start to build up at high reader frequencies. Fig. 3(b) indicates that the response times increases linearly with the number of nodes. Furthermore, with an increasing portion of read requests, the response time is reduced up to 30% for N = 64. The actual reduction (the gradient of each curve) increases with the number of nodes. This demonstrates that although concurrent readers may require an additional message, the concurrent execution of critical sections results in faster responses. In other words, applications can benefit from read concurrency with this protocol (quantified more precisely in Fig. 5).
Token-Based Read/Write-Locks for Distributed Mutual Exclusion
1193
5 4.5 4 3.5 3 2.5 2 1.5 1 0.5
14000 response time [µsec]
number of messages
In contrast, Fig. 4 depicts the results for the static tree protocol. The number of messages remains unchanged or increases slightly, depending on the number of nodes, since messages have to propagate along longer paths within the static tree. The response time also increases linearly for the number of nodes while it decreases with higher reader frequencies. While the response time of the static variant is higher than in the dynamic counterpart in the absence of readers, the results are about the same in the absence of writers. In the latter case, the number of messages is also about the same, which explains this behavior. Fig. 5 summarizes the impact of a read/write distinction for an entire execution of the benchmark. Here, the number of iterations was i = 20, there were N = 16 nodes, the critical section was ξ = 5000µsec and ξ = 20.000µsec while non-critical code took Γ = 2000µsec. This simulates a case where complex operations in critical sections dominate non-critical code. Both the dynamic protocol in Fig. 5(a) and the static protocol in Fig. 5(b) show execution time savings of around 50% when all requests are for read access compared to only write requests. The savings are slightly higher for longer critical sections than for shorter ones. While some savings already materialize around 50% readers, the savings increase even more rapidly for a higher proportions of readers. Typically, applications experience far more read than write requests, e.g., database queries. Hence, these savings have an impact on the overall performance. The last experiment was conducted to determine the impact of different request frequencies on the protocol. For this purpose, Γ was varied for N = 32 nodes, i = 20 iterations and ξ = 200µsec per critical section. Figure 6 shows that for high request frequencies (small Γ ) the dynamic protocol results in fewer messages (about one messaged saved) and shorter response times (by about 10%) compared to the static case. For low request frequencies, the differences between the protocols diminishes. This is caused by the properties of path compression in the dynamic protocol where the structure of the dynamic tree degenerates into a list under these circumstances. This shows that the dynamic protocol exhibits its full advantage mostly for applications which use synchronization heavily throughout its execution. In short, problems with fine-grained
0
20 (a)
N=2
40 60 80 % of Readlocks
N=4
N=8
100
12000 10000 8000 6000 4000 2000 0
0
20
40 (b)
N = 16
N = 32
60
80
100
% of Readlocks N = 64
Fig. 4. Messages and Response Time of the Static Tree Protocol (SVARW)
Claus Wagner and Frank Mueller
execution time [microsec]
1e+07
1e+07
xi = 20000 xi = 5000
9e+06
execution time [µsec]
1194
8e+06 7e+06 6e+06 5e+06 4e+06 3e+06
0
20
40
60
80
8e+06 7e+06 6e+06 5e+06 4e+06 3e+06
100
xi = 20000 xi = 5000
9e+06
(a) DVARW [% of Readlocks]
0
20 40 60 80 (b) SVARW [% of Readlocks]
100
Fig. 5. Overall Execution Time (Dynamic vs. Static Tree) parallelism are more suitable for the dynamic protocol while the static protocol performs equally well for coarse-grained parallelism. Overall, reader/writer distinction provides promising benefits in both cases.
6
Conclusions
We developed a protocol for distributed mutual exclusion that distinguishes reader and writer requests utilizing a dynamic tree structure to reduce message overhead resulting in an average complexity of O(log n) messages per request. This protocol was evaluated in comparison with a protocol using a static structure. The measurements show that although concurrent readers may require at most one additional message per entry, the concurrent execution of critical sections results in faster responses of up to 30% for 64 nodes when critical sections are short. For longer critical sections, savings in the overall execution time increase with the fraction of readers to up to 50%. In particular applications with large fractions of readers, e.g., database queries, may exploit these benefits. The results further indicate that problems with fine-grained parallelism
4.6 response time [µsec]
number of messages
4.4 4.2 4 3.8 3.6 3.4 3.2
SVARW DVARW 0
1000 2000 3000 4000 5000 (a) Request Interval Γ [µsec]
6000
5600 5500 5400 5300 5200 5100 5000 4900 4800 4700 4600
SVARW DVARW 0
1000 2000 3000 4000 5000 6000 (b) Request Interval Γ [µsec]
Fig. 6. Messages and Response Time for Varying Request Frequencies
Token-Based Read/Write-Locks for Distributed Mutual Exclusion
1195
are more suitable for the dynamic protocol proposed here while the static protocol used for comparison performs equally well for coarse-grained parallelism. Overall, reader/writer distinction provides promising benefits in both cases.
References [1] Y. Chang. A simulation study on distributed mutual exclusion. J. Parallel Distrib. Comput., 33(2):107–121, March 1996. [2] Y. Chang, M. Singhal, and M. Liu. An improved O(log(n)) mutual exclusion algorithm for distributed processing. In Int. Conference on Parallel Processing, volume 3, pages 295–302, 1990. [3] S. Fu, N. Tzeng, and Z. Li. Empirical evaluation of distributed mutual exclusion algorithms. In International Parallel Processing Symposium, pages 255–259, 1997. [4] T. Johnson. A performance comparison of fast distributed mutual exclusion algorithms. In Proc. 1995 Int. Conf. on Parallel Processing, pages 258–264, 1995. [5] A. Kumar. Hierachical quorum consensus: A new algorithm for managing replicated data. IEEE Trans. On Computers, 40(9):994–1004, 1991. [6] L. Lamport. Time, clocks and ordering of events in distributed systems. Comm. ACM, 21(7):558–565, June 1978. [7] K. Li and P. Hudak. Memory coherence in shared virtual memory systems. ACM Trans. Comput. Systems, 7(4):321–359, November 1989. [8] M. Maekawa. A sqrt(n) algorithm for mutual exclusion in decentralized systems. ACM Trans. on Computer Systems, 3(2):145–159, 1985. [9] F. Mueller. Prioritized token-based mutual exclusion for distributed systems. In International Parallel Processing Symposium, pages 791–795, 1998. [10] M. Naimi, M. Trehel, and A. Arnold. A log(N) distributed mutual exclusion algorithm based on path reversal. J. Parallel Distrib. Comput., 34(1):1–13, April 1996. [11] M. L. Neilsen and M. Mizuno. A dag-based algorithm for mutual exclusion. In Distributed Computer Systems, pages 354–360, 1991. [12] K. Raymond. A distributed algorithm for multiple entries to a critical section. Information Processing Letters, 30(4):189–193, February 1989. [13] K. Raymond. A tree-based algorithm for distributed mutual exclusion. ACM Trans. Comput. Systems, 7(1):61–77, February 1989. [14] P. Srimani and R. Reddy. Another distributed algorithm for multiple entries to a critical section. Information Processing Letters, 41(1):51–57, January 1992. [15] C. Wagner. Algorithmen zum gegenseiten Ausschluß in verteilten Systemen. Master’s thesis, Humboldt University Berlin, Germany, December 1999.
On Solving a Problem in Algebraic Geometry by Cluster Computing Wolfgang Schreiner, Christian Mittermaier, and Franz Winkler Research Institute for Symbolic Computation (RISC-Linz) Johannes Kepler University, Linz, Austria FirstName.LastName @risc.uni-linz.ac.at http://www.risc.uni-linz.ac.at
Abstract. We describe a parallel solution to the problem of reliably plotting a plane algebraic curve. The sequential program is implemented in the software library CASA on top of the computer algebra system Maple. The parallel version is based on Distributed Maple, a parallel programming extension written in Java. We evaluate its performance on a cluster of workstations and PCs, on a massively parallel multiprocessor, and on a cluster that couples workstations and multiprocessor.
1
Introduction
We describe a parallel solution to the problem of reliably plotting a plane algebraic curve. Our starting point is the software library CASA (computer algebra software for constructive algebraic geometry) which has been developed on top of the computer algebra system Maple under the direction of the third author [2]. The basic objects of CASA are algebraic sets represented e.g. as systems of polynomial equations. Algebraic sets represented by bivariate polynomials model plane curves; an important problem is the reliable plotting of such curves. Numerical methods often yield qualitatively wrong solutions, i.e., plots where some “critical points” (e.g. singularities) are missing. For instance, the left diagram in Figure 1 shows a plot generated by the Maple function implicitplot. The numerical approximation fails to capture two singularities; even if we refine the resolution of the underlying grid, only one of the missing singularities emerges. On the contrary, the CASA function pacPlot produces the diagram shown to the right. This plot of an algebraic curve a is generated by a hybrid combination of exact symbolic algorithms for the computation of all critical points and of fast numerical methods for the interpolation between them [3] (see Figure 1): 1. Compute the critical points of a in the y-direction and sort them according to their y-coordinates. 2. Intersect a with the horizontal lines that lie in the middle of the stripes determined by the y-coordinates of the critical points. 3. Trace a from each intersection point in both directions towards the border points of the stripe.
Supported by the FWF projects P11160-TEC (HySaX) und SFB F013/F1304.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1196–1200, 2000. c Springer-Verlag Berlin Heidelberg 2000
On Solving a Problem in Algebraic Geometry by Cluster Computing
2
2
1.5
1.5
y1
y1
0.5
0.5
1.5 1 0.5
0 0.5x 1 1.5
1.5 1
0
1197
0.5 x 1 1.5
Simple Branch
Critical Points
Fig. 1. Plotting an Algebraic Curve pacPlot spends virtually all computation time in Step 1, a symbolic preprocessing phase that applies exact arithmetic to compute those points that determine the topological structure of a, essentially the singularities and the extrema in one coordinate direction. Figure 2 sketches the structure of the algorithm that computes a set of intervals that isolate these critical points. The computation time of the critical point computation dominates the total plotting time and even on fast workstations only curves with total degree up to 8 or so can be plotted in a time that is acceptable for interactive usage. In the diploma thesis of the second author [1], this function has been parallelized.
2
The Parallelization Approaches
Investigating the dynamical behavior, it turns out that it does not pay off to parallelize the outermost loop which iterates over all systems in S: typically there are not more than two systems and only one system is not immediately recognized as trivial. Therefore we apply parallelism on several other levels: 1. 2. 3. 4.
parallel parallel parallel parallel
resultant computation, real root isolation, solution test, interval refinement.
1198
Wolfgang Schreiner, Christian Mittermaier, and Franz Winkler
:= criticalPoints(a(x; y )): P := ∅ S := {(p(y), q(x, y)) | ), p(y) ∈ factor(p (y))} ∃p (y) : (p (y), q(x, y)) ∈ triangulize(a(x, y), ∂a(x,y) ∂x for (p(y), q(x, y)) ∈ S do r(x) := resultantx (p(y), q(x, y)) X := realroot(r(x)) Y := realroot(p(y)) for x ∈ X do q (y) := squarefree(q(x, y), x.0, p(y)) q (y) := squarefree(q(x, y), x.1, p(y)) for y ∈ Y do if test(q (y), q (y), y, p(y)) then P := P ∪ {(x, y)} end end end end
P
Fig. 2. Computation of Critical Points
Parallel Resultant Computation A single resultant computation may take a good deal of the computation time of the whole algorithm. Our parallelization approach applies a modular method to compute the resultant in multiple homomorphic images of the domain and to combine the homomorphic results to the result in the original domain. We thus get a divide and conquer structure where both the divide phase (the modular resultant computation) and the conquer phase (the “Chinese Remaindering” computation) can be parallelized [4]. Parallel Real Root Isolation The isolation of the real roots of the resultant by Uspensky’s method, a recursive divide and conquer search algorithm, may also take a significant computation time. A naive parallelization of this algorithm typically yields poor speedups due to the narrowness of the highly unbalanced search tree. A parallelization of the arithmetic in each recursive invocation step is only feasible for tightly coupled multiprocessors. Therefore our approach is to broaden and to flatten the search tree (to a certain extent) by applying speculative parallelism [1]. Parallel Solution Test and Interval Refinement The tests which of the candidates (x, y) are indeed solutions of the given system can be performed in parallel in a straight-forward fashion. Likewise, we can apparently refine all isolating intervals in parallel to the desired accuracy [1]. We have implemented all four ideas on the basis of Distributed Maple.
On Solving a Problem in Algebraic Geometry by Cluster Computing
3
1199
Distributed Maple
Distributed Maple is an environment for writing parallel programs on the basis of the computer algebra system Maple developed by the first author [4]. It allows to create tasks and to execute them by Maple kernels running on various machines of a network. Each node of a session comprises two components (see Figure 3): user interface
Maple dist.maple
dist.Scheduler
dist.systems
..... dist.Scheduler
dist.Scheduler
dist.maple
dist.maple
Maple
Maple
Fig. 3. Distributed Maple Architecture Scheduler The Java program dist.Scheduler coordinates the node interaction. The scheduler process attached to the frontend kernel starts instances of the scheduler on other machines and communicates with them via sockets. Maple Interface The program dist.maple running on every Maple kernel implements the interface between kernel and scheduler. Both components use pipes to exchange messages (which may embed any Maple objects). The user interacts with Distributed Maple via the Maple frontend by a number of programming commands, in particular: dist[start](f, a, ...) creates a task evaluating f(a, . . . ) and returns a task reference t. Tasks may create other tasks and arbitrary Maple objects (including task references) may be passed as arguments and returned as results. dist[wait](t ) blocks the execution of the current task until the task represented by t has terminated and returns its result. Multiple tasks may independently wait for and retrieve the result of the same task t. This parallel programming model is based on para-functional principles which is sufficient for many kinds of computer algebra algorithms. In addition, the environment supports a non-deterministic form of task synchronization for speculative parallelism and shared data objects which allow tasks to communicate by side effects on a global store.
1200
4
Wolfgang Schreiner, Christian Mittermaier, and Franz Winkler
Experimental Results
We have benchmarked the parallel critical point computation in three environments with four random algebraic curves for which the sequential program takes 6870s, 470s, 155s, and 11748s (measured on a PIII@450MHz PC): – The cluster of our institute which comprises 4 Silicon Graphics Octanes (2 R10000@250Mhz each) and 16 Linux PCs (various Pentium processors), – a Silicon Graphics Origin multiprocessor (64 R12000@300 Mhz, 24 processors used) located at the university campus, – a mixed configuration consisting of our 4 dual-processor Octanes and 16 processors of the Origin multiprocessor connected via an ATM line. The raw computing power of each environment is 18.3, 17.1, and 18.7 (multiplication factors compared with a single PIII@450MHz processor); the benchmark results are as follows: Execution Times (s) Example Environment 1 2 4 1 (6870s) Cluster 14992 8035 2732 Origin 7290 4217 1789 Mixed 14992 8035 2732 2 (470s) Cluster 810 648 328 Origin 667 541 297 Mixed 810 648 328 3 (155s) Cluster 267 191 112 Origin 196 147 90 Mixed 267 191 112 4 (11748s) Cluster 25178 15559 6820 Origin 13397 8223 4009 Mixed 25178 15559 6820
8 1186 872 1368 173 166 210 67 63 74 3562 2281 4042
16 552 446 597 95 112 116 46 54 58 1915 1726 2004
24 488 513 519 108 116 127 45 54 56 1563 1420 1599
A detailed analysis of the results and our conclusions are given in a longer version of this paper at http://www.risc.uni-linz.ac.at/software/distmaple.
References [1] C. Mittermaier. Parallel Algorithms in Constructive Algebraic Geometry. Master’s thesis, Johannes Kepler University, Linz, Austria, 2000. [2] M. Mnuk and F. Winkler. CASA - A System for Computer Aided Constructive Algebraic Geometry. In DISCO’96 — International Symposium on the Design and Implementation of Symbolic Computation Systems, volume 1128 of LNCS, pages 297–307, Karsruhe, Germany, 1996. Springer, Berlin. [3] T. Q. Nam. Extended Newton’s Method for Finding the Roots of an Arbitrary Systemof Equations and its Applications. In IASTED’94: 12th International Conference on Applied Informatics, Annecey, France, 1994. [4] W. Schreiner. Developing a Distributed System for Algebraic Geometry. In B. H. Topping, editor, EURO-CM-PAR’99 Third Euro-conference on Parallel and Distributed Computing for Computational Mechanics, pages 137–146, Weimar, Germany, March 20-25, 1999. Civil-Comp Press, Edinburgh.
PCI-DDC Application Programming Interface: Performance in User-Level Messaging Eric Renault, Pierre David, and Paul Feautrier Laboratoire PRiSM Universit´e de Versailles - Saint-Quentin-en-Yvelines 78035 Versailles Cedex, France {Eric.Renault, Pierre.David, Paul.Feautrier}@prism.uvsq.fr
Abstract. Many programming interfaces which deal with peripherals (especially network peripherals) need to access the critical resources of the operating system, and system calls (or drivers) are generally used. Unfortunately, the time spent on such system calls is often greater than those requested for the operations themselves. In the case of the MPC machine, most of these operations do not need an intervention from the system and resources may be available from user applications. Our programming interface provides different levels of integration in the kernel depending on the security and the performance expected by the administrator. This article presents PAPI user-level performance.
1
Introduction
The MPC project started in 1995 as a collaborative endeavour between the LIP6 and PRiSM laboratories (France). Its goal is the development of both hardware and software layers to use the new HSL technology, a high speed network with an affordable price. An MPC machine is composed of nodes, each one with one or more processors and 1-Gbits/s links (HSL links). On each node, a FastHSL card is the interface between the network and the computer. An HSL link (IEEE 1355 [1]) is a bidirectional serial link which delivers a throughput of 1 Gbit/s. As 4 bits of control (generated by the crossbar to perform various controls) are transmitted with every byte, the effective bandwidth is limited to 80 MBytes/s. The RCube (for Rapid Reconfigurable Router) component [2] is a 8 × 8 crossbar with a high bandwidth of 640 MBytes/s and a low latency of 150 ns per hop. Routing in this chip is wormhole and adaptative and it is possible to define different configuration tables for each of the 8 bidirectional links. The PCI-DDC (for PCI-Direct Deposit Component) component [3] is a network interface for message-passing multiprocessor architectures using HSL links and RCube routers. It is connected to the PCI bus (where it can act as a master) and sends/receives messages to/from the RCube component without processor intervention, using Direct Memory Access. A Fast-HSL card is a 32bit 33-MHz PCI card which includes the PCI-DDC and RCube components. It provides seven HSL ports, each one connected to a RCube entry; the last entry of the RCube chip is connected to the PCI-DDC component. In this article, we A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1201–1205, 2000. c Springer-Verlag Berlin Heidelberg 2000
1202
Eric Renault, Pierre David, and Paul Feautrier
present a short description of the PCI-DDC component and a fast presentation of PAPI. Then, our user-level results are compared to those of Active Messages, BIP and Fast Messages.
2
Programming the PCI-DDC Component
The aim of the PCI-DDC component is to exchange messages with other nodes via the HSL network and the RCube components. Interface with the CPU uses two circular lists in main memory and some registers to store pointers in these lists. The two lists are the LPE (List of Pages to Emit) where CPU describes messages to send and the LMI (List of Message Identifiers) where the PCI-DDC component writes the description of each received message. A message identifier is a 24-bit integer which identifies a message; a single node does not permit two incoming messages with the same identifier at the same time. The component allows the application to send two different message types: normal messages are using the “remote-write” protocol [4], i.e. both local and remote physical addresses must be specified by the sender; short messages do not need any physical address, i.e. instead of specifying local and remote physical addresses, the content of the message is placed directly in these fields. Short messages are particularly important during the initialization phase of an application, for example, since they are the only way to exchange physical addresses needed for normal messages. The size of these short messages is limited to 8 bytes, the room available in LPE and LMI entries. PAPI [5], which stands for PCI-DDC Application Programming Interface, is a modular interface. It allows the system administrator to choose the security level of the system depending on the performance desired and the users. In the current version, three levels of integration are available: – NONE: no security is provided. All of the module code is placed in user mode except some system calls (used only at initialization), used to access information from the kernel; – ACCESS: traditional system security is provided, i.e. code is placed in kernel space and a system call is performed each time the application calls functions from our API; – HIDE: the total security configuration, i.e. the code is placed in kernel space and no system call is provided to access it. The only way to run a function of a hidden module is to have another module with an ACCESS configuration, and system calls to access the functions of the hidden module.
3
Performance
The program used to determine the performance times of the MPC machine (see below) is the classical ping-pong program. It sends and receives one million messages between two nodes (one message can be sent if the message from the remote node has been received). Once this is done, the elapsed time is divided
PCI-DDC Application Programming Interface
1203
by two millions in order to determine the total average time to send one message from one node to another one (i.e. one-way latency). The effective user bandwidth is computed by dividing the size of messages by the latency (throughout this section, we will refer to the effective user bandwidth as bandwidth). At the PRiSM laboratory, each node of our MPC machine is composed of two 233-MHz Pentium II with 32 MB of memory. Another network (a 10-Mbit/s Ethernet TCP/IP network) is used for control operations, such as program launching. The operating system is FreeBSD 3.2. We will discuss our results by comparing them with Myrinet [6] message-passing libraries, especially Active Messages [7], BIP [8] and Fast Messages [9]. These support an architecture similar to that of the MPC project, i.e. a cluster of PCs, and are some of the fastest available. Performance results from these other message-passing libraries are taken from the literature [10][11][12], and not measured directly. Figure 1 shows the time required to send a message from one node to another for different message-passing libraries. It shows that a latency for message sizes less than or equal to 64 bytes is quite similar for BIP, Fast Messages and PAPI. However, PAPI has a far better latency for messages larger than 64 bytes.
120
Active Messages BIP Fast Messages PAPI
One-way latency (us)
100 80 60 40 20 0
4
8
16
32
64
128 256 512 1024 2048
Message size (bytes)
Fig. 1. One-way latency for different message-passing libraries
The anomaly in the BIP curve above 128 bytes (see Fig. 2) is due to the short message concept of BIP. The boundary is software-dependent, and reflects the threshold where messages are sent directly to the host memory rather than copied in the adapter memory. The PCI-DDC concept of short message is quite different: short message data are copied in the LMI of the receiving node, and are restricted to 8 bytes, the room available in LPE and LMI entries. Where is the latency time spent? Figure 3 analyses the details of the transmission of a short message. On the sender node, the application calls our PAPI SSend function, which fills an entry in the LPE in memory, performs a write in a PCIDDC register which is located in PCI configuration space, and returns. Once
1204
Eric Renault, Pierre David, and Paul Feautrier 60
Active Messages BIP Fast Messages PAPI
Bandwidth (MBytes/s)
50 40 30 20 10 0
4
8
16
32
64
128
256
512 1024 2048
Message size (bytes)
Fig. 2. Bandwidth for different message-passing libraries
informed by the write that a new message must be sent, the PCI-DDC component starts to work. Hardware specifications [3][2] tell us that the components and the network spend 1.7 µs (hardware latency) to bring the message to the other node memory. On the receiver node, where the PAPI Receive function is called by the application, the LMI list is monitored. As soon as it is updated by the receiving PCI-DDC component, PAPI Receive returns with information about the incoming message. 0
1000
2000
3000
4000
Time (ns) LPE entry fill (400 ns) PCI access (1850 ns) function return (150 ns) PAPI_SSend
hardware latency (1700 ns)
Hardware component and network PAPI_Receive polling function return (400 ns)
Fig. 3. Decomposition of the transmission time for a short message
This figure highlights another important point: during the transmission of the message, only a little part (0.8 µs over 4.39 µs, i.e. less than 20%) depends upon the CPU speed. The rest of the time depends upon the PCI bus speed and the hardware latency. The greatest part (more than 80%) of the latency time is
PCI-DDC Application Programming Interface
1205
thus independent of the CPU. So this means that PAPI is close to the minimum latency for the Fast-HSL card.
4
Conclusion
In this article we have presented PAPI, an application programming interface for the MPC machine. We have shown that, even for messages of less than one system page (4 kB), our programming interface takes advantage of the HSL link bandwidth. The latency is very close to the minimum latency one can expect from the Fast-HSL cards. Our study shows that a large part of the latency time is spent on a PCI access through a configuration space register. Given the same HSL network and router (RCube), improvements could be made only with modified access to the PCIDDC registers, through a memory-mapped access. However, this would require a modification of the PCI-DDC hardware component.
References [1] C. Whitby-Strevens and al. IEEE Draft Std P1355 — Standard for Heterogeneous Interconnect — Low Cost Low Latency Scalable Serial Interconnect for Parallel System Construction, 1993. [2] V. Reibaldi. RCube Specifications. Laboratoire Informatique de Paris VI, February 1997. [3] J.J. Lecler, F. Potter, A. Greiner, J.L. Desbarbieux, and F. Wajsburt. PCI-DDC Specifications. Laboratoire Informatique de Paris VI, 1996. [4] F. Potter. Conception et r´ealisation d’un r´eseau d’interconnexion a ` faible latence et haut d´ebit pour machines multiprocesseurs. PhD thesis, Universit´e Pierre et Marie Curie, 1996. [5] E. Renault. Pci-ddc Application Programming Interface User Manual. Research report, Laboratoire PRiSM, Universit´e de Versailles – Saint-Quentin, France, May 2000. [6] N.J. Boden, D. Cohen, R.E. Flederman, A.E. Kulawik, C.L. Seitz, J.N. Selzovic, and W.-K. Su. Myrinet – A Gigabit-per-Second Local-Area Network. In IEEEMicro, volume 15, pages 29–36, 1995. [7] T. von Eicken, D.E. Culler, S.C. Goldstein, and K.E. Schauser. Active Mesages: a Mechanism for Integrated Communication and Computation. In 19th International Symposium on Computer Architecture, 1992. [8] L. Prylli. BIP Messages User Manual for BIP 0.94, June 1998. [9] S. Pakin, V. Karamcheti, and A.A. Chien. Fast Message (FM): Efficient, Portable Communication for Workstation Clusters and Massively-Parallel Processors. In IEEE Concurrency, volume 5, pages 60–63, 1997. [10] S. Araki, A. Bilas, C. Dubnicki, J. Edler, K. Konishi, and J. Philbin. UserSpace Communication: A Quantitative Study. In The SuperComputing, Orlando, Florida, 1998. [11] Concurrent Systems Architecture Group. Fast Messages Performance. Web page, 1999. http://www-csag.ucsd.edu/projects/comm/fm-perf.html. [12] High Speed Networks and Multimedia Application Support. BIP Performance Curves. Web page, 1997. http://lhpca.univ-lyon1.fr/Resultats/bipres.html.
A Clustering Approach for Improving Network Performance in Heterogeneous Systems Vicente Arnau, Juan M. Ordu˜na, Salvador Moreno, Rodrigo Valero, and Aurelio Ruiz Departamento de Inform´atica. Universidad de Valencia.SPAIN [email protected]
Abstract. A lot of research has focused on solving the problem of computationaware task scheduling on heterogeneous systems. In this paper, we propose a clustering algorithm that, given a network topology, provides a network partition adapted to the communication requirements of the applications running on the machine. Also, we propose a criterion to measure the quality of each one of the possible mappings of processes to processors based on that network partition. Evaluation results show that these proposals can greatly improve network performance, providing a basis of a communication-aware scheduling technique.
1 Introduction Networks of Workstations (NOW’s) are used nowadays as parallel computers, forming heterogeneous systems. A lot of research has focused on solving the NP-complete problem of efficiently scheduling diverse groups of tasks to the machines that form these systems from the computational point of view. However, the increasing computational power of new processors and the growing communication requirements of the applications may cause the interconnection network in these heterogeneous systems to become the system performance bottleneck. In a previous paper, we proposed a new model of communication cost between nodes, the table of equivalent distances [1]. This model consist of a table of N × N distances, where N is the number of nodes in the network. In this table, the element Tij represents the communication cost for messages going from node i to node j. In this paper, we propose a clustering algorithm based on the table of distances that provides a network partition, and a criterion to measure the suitability of each allocation of network resources to the applications that the provided network partition may generate. Evaluation results show that the use of this proposals significantly improve network performance by reducing communication bottlenecks. Furthermore, this clustering technique is applicable to both regular and irregular topologies, providing a general basis for communication-based process mapping.
2 A New Clustering Approach From a general point of view, we can consider that each application belongs to a different user. Therefore, we can assume that the processes belonging to the same application
Supported by the Spanish CICYT under Grant TIC97–0897–C04–01
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1206–1209, 2000. c Springer-Verlag Berlin Heidelberg 2000
A Clustering Approach for Improving Network Performance
1207
may intensively communicate between them, but they will not communicate at all with processes from other applications. Therefore, we can group the processes running on the machine, forming a set of logical clusters of processes, where each cluster is formed by the processes belonging to each application. The proposed algorithm intends to provide a network partition adapted to any existing set of logical clusters. The first step in this method is to find an Euclidean metric space in which we can represent our N nodes, in such a way that the resulting distances between them are as close as possible to the values in the table of distances (the latter one does not define a metric space). We have computed a least squares linear adjustment using the steepest gradient method [2]. The solution consists of N points in an Euclidean space whose coordinates generate a table of Euclidean distances with the least quadratic error with regard to the table of distances. It is worth mention that the table of Euclidean distances does not contain repeated values (except zero values in the diagonal). Once a table of Euclidean distances has been computed, the furthest-neighbor algorithm is used to compute the optimal dendogram [4]. This algorithm uses a similarity measure. In each step the algorithm merges two of the existing clusters into a new one, choosing the two clusters that result in the lowest similarity measure when the step is applied. The similarity measure usually used in this algorithm is the intracluster distance, and therefore it is called the furthest-neighbor algorithm. However, we considered as the similarity measures f to be maximized the inverse of the Euclidean distance, defined as fa = D1ij , where Dij is the distance from cluster i to cluster j in the Euclidean table of distances. The initial network partition consists of the N nodes located at the coordinates given by the computed table of Euclidean distances. In each step a new partition is formed, decreasing the number of clusters by one. When merging two clusters, they are replaced by a new cluster, and the Euclidean table of distances must be computed again in each step. The result of the above clustering approach is a dendogram, but not a mapping of processes to processors. The cardinal of the set of logical clusters can be used to determine when to stop the clustering algorithm, obtaining a network partition with a number of network clusters equal to the number of existing logical clusters of processes. Nevertheless, the number of nodes (switches) in each network cluster may significantly differ from the number of processes in each logical cluster of processes. Therefore, new changes in this partition are still needed in order to map all the existing processes according to the communication requirements. We have performed manually this clustering adjustment, obtaining different possible process mappings from each network partition. However, it is necessary to define a metric of the communication bandwidth achieved by each one of the possible mappings, in order to select the one who provides the best network performance. We have defined two distinct and complementary global quality functions, the similarity function FG and the dissimilarity function DG . FG measures the intracluster distances, and DG measures the intercluster distances. The cluster quality function FAi for cluster Ai is defined as the quadratic sum of all intracluster distances. It must be noticed that for these functions we are considering the distances in the table of distances. Although the partition provided by the Euclidean approach is based on an Euclidean table of distances, the quality function must be based on the table of distances, since it mea-
1208
Vicente Arnau et al.
sures the actual network distances. The similarity global function for the final partition FG is computed as the sum of all the FAi values divided by the total number of intracluster distances existing in partition P and normalized by the quadratic average value of all of the distances between the network nodes. For the dissimilarity global function we define the cluster dissimilarity function DAi for a cluster Ai as the quadratic sum of all intercluster distances from nodes in cluster Ai to all the nodes in the rest of the clusters. The dissimilarity global function DG is defined as the sum of all the DAi values divided by the total number of existing intercluster distances in partition P and normalized by the quadratic average value of all of the distances between the network nodes. FG and DG provide a measurement of the intracluster and intercluster communication costs, respectively. Thus, the quotient DG / FG provides the relationship between the intracluster and intercluster bandwidth achieved with each process mapping. We will denote this relationship as the clustering coef ci ent Cc . This clustering coefficient can be used to measure the quality of each process mapping.
3 Performance Evaluation We have evaluated the improvement in network performance that the proposed clustering approach can provide, as well as the correlation between the clustering coefficient and network performance. This study assumes that all the communication between processors is intracluster communication and all the processors transmit the same amount of information. We have evaluated the performance of several irregular networks by simulation. The evaluation methodology used is based on the one proposed in [3]. The most important performance measures are latency and throughput. The network is composed of a set of switches. The network topology is irregular and has been generated randomly. We assumed 8-port switches. Each switch has 4 ports available to connect to other switches. From these 4 ports, three of them are used in each switch when the topology is generated. The remaining port is left open. We have evaluated networks with a size ranging from 16 switches (64 nodes) to 24 switches (96 nodes). Several distinct topologies have been analyzed. For the sake of simplicity, we have assumed a fixed pool of N processes grouped into 4 clusters with N 4 processes, where N is the number of network nodes. Each process is assumed to send all of the generated messages to processes in the same logical cluster of processes. For each network, we have computed the clustering algorithm until it has provided a 4-cluster partition of the network, and then we have chosen several possible mappings based on this partition. Additionally, we have computed several random mappings for each considered network. Figure 1 shows network performance for some of the mappings based on the partition provided by the Euclidean approach (Ei labels) for a 16–switch network, compared with the network performance obtained by several randomly generated mappings (Ri labels). The clustering coefficient Cc obtained for each mapping is shown on the right side of each plot label. The network throughput obtained with any of the mappings based on the proposed approach is about a 55% higher than the network throughput obtained with any of the randomly generated mappings, while the network latency is less
A Clustering Approach for Improving Network Performance
1209
Fig. 1. Simulation results for a 16-switch network than 63%. On the other hand, the value of Cc is clearly lower for randomly generated mappings, showing that this function is directly related to network performance. We have also studied the correlation of the clustering coefficient Cc with network performance. We computed the correlation index between the clustering coefficient and the network performance obtained for all the mappings in all of the considered networks. In any case this index resulted higher than 80% for simulation points of both low network load and network saturation. These results validates the clustering coefficient as an “a priori” measure of relative network performance.
4 Conclusions Network throughput and network latency are greatly improved when using the mappings based on the proposed approach, showing that it can be used as the basis for a communication-aware mapping technique. We have also studied the correlation between the proposed clustering coefficient and network performance. The results show that when only exists intracluster communication then this coefficient is highly correlated with network performance. For further information, please see technical report TR-AR99 on http://www.gap.upv.es
References 1. V. Arnau et al., “On the Characterization of Interconnection Networks with Irregular Topology: A New Model of Communication Cost”, in PDCS 99, November 1999. 2. M. S. Bazaraa et al., Nonlinear Programming: Theory and Algorithms, J. Wiley, 1993. 3. J. Duato, “A new theory of deadlock-free adaptive routing in wormhole networks,” IEEE Trans. Parallel and Distributed Systems, vol. 4, no. 12, pp. 1320–1331, December 1993. 4. R. Duda, P. Hart, Pattern Classi cat ion and Scene Analysis, J. Wiley, 1973.
Topic 19 Metacomputing Alexander Reinefeld, Geoffrey Fox, Domenico Laforenza, and Edward Seidel Topic Chairmen
The basic idea about metacomputing is to utilize a variety of geographically dispersed resources, such as computers, storage systems, data sources and special devices, which are seen by the user as a single unified resource. This new computing paradigm was born with the growing success of the Internet which made it possible to link remote high-performance computers for the collaborative use. A metacomputer provides a variety of capabilities that can be orchestrated to execute multiple tasks with varied computational requirements. Applications in these environments achieve their performance by properly mapping the tasks onto the best suited platforms while considering the overhead of inter-task communication and the coordination of distinct data sources and administrative domains. Ideally, the distributed nature of a metacomputer environment is transparent to the user, that is, the user only needs to describe the constraints connected with the job while the system selects the most suitable machine for the execution of his job. This selection process is subject to a large variety of different constraints including access restrictions, user priorities, machine workload, job characteristics and user preferences. In addition the metacomputer structure may change due to maintenance shut downs or temporary failures of sub-systems. It is the task of the management software to handle those problems, that is, to provide a transparent access to the users while considering any special requests from users and owners. This metacomputer management software is the core topic of the research described in the following papers. The paper by Arnold, Bachmann, and Dongarra presents a general technique to reduce network traffic when executing several requests to a grid computing system. This technique, called “request sequencing”, uses DAGs for building task sequences which allows to minimize the data transfer by grouping requests. Request sequencing can thereby affect scheduling policies and enable more expedient resource allocation methods. The paper by Kindermann and Fink describes an approach for a componentbased meta-applications design based on formal architecture description of the gross organization of an application. Simple architectural styles developed to support data-flow and control-flow driven meta-application design on top of the Amica metacomputing infrastructure are presented. The paper by Kamachi, Priol and Ren´e describes the use of distributed objects (i.e., parallel CORBA objects) as a modern approach for programming computational grids. In particular, the authors focus on the problems related to A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1211–1212, 2000. c Springer-Verlag Berlin Heidelberg 2000
1212
Alexander Reinefeld et al.
the handling of distributed data within parallel objects, showing some interesting performance results on a practical distributed computing platform. The paper by Neary, Phipps, Richman, and Cappello focuses on Java-based parallel computing in the Internet. It describes enhancements to the well-known Javelin system. Javelin aims at freeing the application developer from concerns about processor interconnect issues, thereby allowing to focus on the application issues.
Request Sequencing: Optimizing Communication for the Grid Dorian C. Arnold1 , Dieter Bachmann2 , and Jack Dongarra1 1
Department of Computer Science, University of Tennessee, Knoxville, TN 37996 [darnold, dongarra]@cs.utk.edu 2 Computer Graphics and Vision, Graz University of Technology, Inffeldg. 16/E/2, A-8010 Graz Austria [email protected]
Abstract. As we research to make the use of Computational Grids seamless, the allocation of resources in these dynamic environments is proving to be very unwieldy. In this paper, we introduce, describe and evaluate a technique we call request sequencing. Request sequencing groups together requests for Grid services to exploit some common characteristics of these requests and minimize network traffic. The purpose of this work is to develop and validate this approach. We show how request sequencing can affect scheduling policies and enable more expedient resource allocation methods. We also discuss some of the reasons for our design, offer the initial results and discuss issues that remain outstanding for future research.
1
Introduction
The vBNS[1] and Myrinet[2] represent two technologies that connect computational resources at high speeds in settings from local to global area networks. However, with the speed of processors increasing at a rate much greater than that of networking infrastructure, data transfer continues to bear large overhead for many applications of high-performance computing. Yet, straightforward ways for increasing application performance by optimizing communication often go overlooked. Our research on call sequencing for Grid middleware aims to take a significant step in this direction. Computer applications generally exhibit two common characteristics: large input data sets and data dependency amongst computational cycles. The goal of this effort is to employ simple, yet highly effective, strategies for exploiting these characteristics. We believe that not enough attention has been given to examining ways to effectively distribute application data amongst the different components of a Grid. We qualify this last statement by saying that data partitioning has been well researched for parallel programming, but in cases where
This work supported in part by Raytheon Systems subcontract #AA23, PACI subaward #790 under prime NSF Cooperative Agreement #ACI-9619019 and NSF Grant #ACI-9876895.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1213–1222, 2000. c Springer-Verlag Berlin Heidelberg 2000
1214
Dorian C. Arnold, Dieter Bachmann, and Jack Dongarra
computational modules execute concurrently with no data exchange, often the same data is unnecessarily transported multiple times between the same components. 1.1
Positioning Our Work
This paper explores the design, implementation and initial results of what we call request sequencing. This term encompasses both an interface to group a series of requests and a scheduling technique, viz. one that uses data persistence and a Direct Acyclic Graph (DAG) representation of computational modules. Our motivation was to allow users to take advantage of data redundancies within a sequence of requests and optimize data communications. We developed and tested our ideas using the NetSolve system described in Sect. 2. Below, we offer the relationship of this research to other research that has been or is being done. Our scheduling work is reminiscent of techniques utilized in schedulers that execute “batches” of processes on parallel machines. We create task graphs or DAGs that represent execution dependencies and schedule them for execution [3]. J. Dennis researched data flow scheduling techniques for Supercomputers[4]; our work presents a similar idea for Grid environments. Ninf [5] is a functional metacomputing environment that shares many similarities with NetSolve. The project has implemented a strategy to increase parallelism amongst the computational services. Similar to this work, they group together requests and execute the modules simultaneously, when possible. Their main focus is on parallel module execution, and it is not stated whether redundant messages are sent or not. Our focus is on minimizing network traffic; our design ensures that no unnecessary data transfer takes place. Condor [6] is a high-throughput computing system that manages very large collections of distributively owned workstations. The Directed Acyclic Graph Manager (DAGMan) is a meta-scheduler for Condor jobs. Users can submit batch jobs to the Condor system and use DAGMan to pre-define execution order. Once again, however, the main focus is on parallel execution and not data transfer. As an extra burden, the data dependency analysis is left to the user. The contribution put forth by this paper is a thorough understanding of an approach to optimize data transfer in Grid settings and the empirical data to justify using this approach. We also offer discussion of scheduling in this environment. Section 2 of this paper presents details about NetSolve, which is our deployment environment. Section 3 describes the design and implementation of the sequencing interface, the server data persistence and execution scheduling. Section 4 contains the experimental test cases and the results that validate this strategy. Finally, Sect. 5 summarizes the work and discusses future research goals.
2
An Overview of NetSolve
The NetSolve [7] project is being developed at the University of Tennessee. It provides remote access to computational resources, both hardware and software.
Request Sequencing: Optimizing Communication for the Grid
1215
Built upon standard Internet protocols, like TCP/IP sockets, it supports popular variants of UNIX, and the client is available for Microsoft Windows ’95, ’98, NT and ’00. Figure 1 shows the infrastructure of the NetSolve system and its relation to the applications that use it. NetSolve and similar systems are referred to as Grid Middleware; this figure explains this terminology. The shaded areas represent the NetSolve system; it can be seen that NetSolve acts as a glue layer that brings the application or user together with the hardware and/or software it needs to complete useful tasks.
NS
Applications Client Library
Users
NS Agent
Resource Discovery Load Balancing Resource Allocation Fault Tolerance NS Server
NS Server
NS Server
Fig. 1. Architectural Overview of the NetSolve System
At the top tier, the NetSolve client library is linked in with the user’s application. The application then makes calls to NetSolve’s application programming interface (API) for specific services. Through the API, NetSolve client-users gain access to aggregated resources without the users needing to know anything about computer networking or distributed computing. The NetSolve agent maintains a database of NetSolve servers along with their capabilities (hardware performance and allocated software) and dynamic usage statistics. It uses this information to allocate server resources for client requests. The agent finds servers that will service requests the quickest, balances the load amongst its servers and keeps track of failed ones. The NetSolve server is a daemon process that awaits client requests. The server can run on single workstations, clusters of workstations, symmetric multiprocessors or machines with massively parallel processors. A key component of the NetSolve server is a source code generator which parses a NetSolve problem description file (PDF). This PDF contains information that allows the NetSolve system to create new modules and incorporate new functionalities.
1216
3
Dorian C. Arnold, Dieter Bachmann, and Jack Dongarra
Sequencing Design and Implementation
As stated in Sect. 1, our aim in request sequencing is to decrease network traffic and overall request response time. Our design needs to ensure that i) no unnecessary data is transmitted and ii) all necessary data is transferred. We also need to cut execution time by executing modules simultaneously when possible. We do this by performing a detailed analysis of the input and output parameters of every request in the sequence to produce a DAG that represents the tasks and their execution dependencies. This DAG is then sent to a server in the system where it is scheduled for execution. 3.1
The DAG Model
Kwok et al. [3] offers a very good description of the DAG: The DAG is a generic model of a parallel program consisting of a set of processes (nodes) among which there are dependencies. A node in the DAG represents a task which in turn is a set of instructions that must be executed sequentially without preemption on the same processor. A node has one or more inputs. When all inputs are available, the node is triggered to execute. The graph also has directed edges representing a partial order among the tasks. The partial order introduces a precedenceconstrained directed acyclic graph and implies that if ni → nj , then nj is a child which cannot start until its parent ni finishes and sends its data to nj . 3.2
Data Analysis and the DAG
In order to build the DAG or task graph, we need to analyze every input and output in the sequence of requests. We evaluate two parameters as the same if they share the same reference. We use the size fields and reference pointer of the input parameters to calculate when inputs overlap in the memory space. NetSolve supports many object1 types, including matrices, vectors and scalars; only matrices and vectors are checked for reoccurences on the premise that these are the only objects that tend to be large enough for the overhead of the analysis to pay dividends. This analysis yields a DAG. The graph is acyclic because looping control structures are not allowed within the sequence, and therefore, a node can never be its own descendant. 3.3
The Interface
In addition to the original function used for request submittal, two functions are implemented; their purpose is to mark the beginning and end of a sequence of 1
We use the term object to refer to a composition of native data types, as in a matrix object of native integers
Request Sequencing: Optimizing Communication for the Grid
1217
requests. begin sequence() takes no arguments and returns nothing; it notifies the system to begin the data analysis. end sequence() marks the end of the sequence; at this point, the sequence of collected requests is sent to a server(s) to be scheduled for execution. As an enhancement, this function also takes a variable number of arguments describing which output parameters NOT to return. This means that if the intermediate results are not necessary for any local computations, they need not be returned. This is a part of the API as it is the user who should determine which results are mandatory and which are useless. Figure 2 illustrates what a sequencing call might look like. Two points to note in this example: i)for all requests, only the last parameter is an output, and ii)the user is instructing the system not to return the intermediate results of command1 and command2.
... begin_sequence(); submit_request("command1", A, B, C); submit_request("command2", A, C, D); submit_request("command3", D, E, F); end_sequence(C, D); ...
Fig. 2. Sample C Code Using Request Sequencing Constructs
For the system to be well-behaved, we must impose certain software restrictions upon the user. Our first restriction is that no control structure that may change the execution path is allowed within a sequence. We impose this restriction because the conditional clause of this control structure may be dependent upon the result of a prior request in the sequence, and since the requests are not scheduled for execution until the end of the sequence, the results will likely not be what the programmer expects. The other restriction is that statements that would change the value of any input parameter of any component of the sequence are forbidden within the sequence (with the exception of calls to the API itself that the system can track.) This is because during the data analysis, only references to the data are stored. So if changed, the data transferred at the end of the sequence will not be the same as the data that was present when the request was originally made. We contemplated saving the entire data, rather than just the references, but this directly conflicts with one of our premises – that the data sets are large; multiple copies of these data are not desirable. 3.4
Execution Scheduling at the Server
Once the entire DAG is constructed, it is transferred to a NetSolve computational server. [3] offers a taxonomy of graph scheduling algorithms in multi-processor
1218
Dorian C. Arnold, Dieter Bachmann, and Jack Dongarra
environments. These algorithms take into account both node-computation and inter-node communication costs. In this first version of request sequencing, the NetSolve agent uses a larger granularity and decides which server should execute the entire sequence. We execute a node if all its inputs are available and there are no conflicts with its output parameters. The reason for this is that currently the only mode of execution we support is on a single NetSolve server – though, that server may be a symmetric multi-processor (SMP). We discuss our plans for expanding this model in Sect. 5. For data partitioning, we transfer the union of the input parameter sets to the selected server host. This makes input for all nodes, except those which are intermediate output from prior nodes, available for the execution of the sequence. When we move to a multi-server execution mode for the sequence, we must enhance our data staging technique, and this is also discussed in Sect. 5. Since all execution is on a single server, our execution scheduling algorithm is reasonably uncomplicated. In essence, we execute all nodes with no dependencies, updating the dependency list as nodes complete, and then check for further nodes to execute. We do this until all nodes have executed: while(problems left to execute) execute all problems with no dependencies; wait for at least one problem to finish; update dependencies; } 3.5
Discussion
Figures 3 and 4 show the reduced network activity between client and server during execution of the sequence in Fig. 2. In the first case, input A is sent to the server twice, and output C and D are unnecessarily sent back to the client as intermediate output and then to the server once again as input. In the latter case, these unnecessary transfers are removed. (These diagrams show three potentially different servers, but our current implementation sees this as three instances of the same server.) Our hypothesis is that this reduction in data traffic will yield enough performance improvements to make sequencing worthwhile.
4
Applications and Initial Results
In this section, we discuss the applications that we used to test our request sequencing infrastructure. They are from the remote sensing/image processing domain as it was the nature of some of these applications that led us to investigate request sequencing. The size of images that are analyzed can become very large and easily extend into the gigabyte range. It is also common in many image processing applications to execute a series of operations on an image, usually one transformation after another.
Request Sequencing: Optimizing Communication for the Grid sequence(A, B, E)
command1(A, B) Client
Server1
Client
Server1
result C
intermediate result C + input A
command1(A, C) Client
Server2
Server2 result D
intermediate result D + input E
command1(D, E) Client
1219
Server3 result F
Fig. 3. Client-Server Data Flow Without Request Sequencing
Server3
Client result F
Fig. 4. Client-Server Data Flow With Request Sequencing
Experiments were executed from NetSolve clients connected to a switched 10/100Mbit ethernet and crossing a 155Mbit ATM switch that is directly connected to the NetSolve servers. The NetSolve server was an SGI Power Challenge with eight R10000 processors. Graphed results are the averages of four independent sets of runs. For the experiments, we varied the network bandwidth by using the NistNet [8] interface on a Linux router. A network performance testing tool, TTCP, which is able to generate TCP traffic on IP based networks, was used to obtain a correction curve for the values set by NistNet.
4.1
Linear Sequence: Principle Component Analysis
Multispectral or multidimensional remote sensing data can be represented by constructing a vector space using one axis per dimension. By calculating the covariance matrix , the axes are transformed into an uncorrelated system. This transformation is called Principle Component Analysis (PCA) [9].
open Image
open image convert format
cluster
cluster
pca
convert format
combine
Fig. 5. Principal Component Analysis
Fig. 6. Multimodal Image Clustering
1220
Dorian C. Arnold, Dieter Bachmann, and Jack Dongarra
In remote sensing, the PCA is used to reduce the number of channels for input images by moving the information towards the first bands. The application used for testing a linear sequence opens a 10MB image, performs a PCA and stores the transformed result. The structure is shown in Fig. 5. Fig. 7 shows how the total response time (from request initiation to availability of results) varies with the bandwidth. It confirms our beliefs: with sequencing in place, there is a significant reduction in execution time of the PCA application. Similar results are to be expected for applications that exhibit similar levels of parameter sharing, and in fact, our examples are not contrived, but represent realistic applications that scientists have used to support their research.
30:00
no sequencing sequencing
25:00
Time in minutes
20:00
15:00
10:00
05:00
00:00
0
100
200
300 400 Bandwidth in kByte/s
500
600
700
Fig. 7. PCA Sequence Executed on an SGI workstation
4.2
Parallel Sequence: Clustering
To handle multisource/multimodal satellite images or to improve clustering accuracy, several classification steps are performed, and their results are combined by a fusion module. Such a module can consist of either a simple pixel selection approach based on severity ratings or a knowledge based combination module [9]. For our tests a pixel based approach has been chosen using an image size on the order of 1MB. This process is illustrated in Fig. 6. Fig. 8 graphs the variation of response time with bandwidth for this parallel sequence. The shape is similar to that of the PCA application. Again, request sequencing yields decreases in execution time.
Request Sequencing: Optimizing Communication for the Grid 10:00
1221
no sequencing sequencing
Time in minutes
08:00
06:00
04:00
02:00
00:00
0
100
200
300
400
500
600
700
Bandwidth in KByte/s
Fig. 8. Clustering Sequence Executed on Two Processors of an SGI workstation. These preliminary results encourage our investigation of sequencing, hinting that a multi-server mode can be a very useful methodology in Grid Computing.
5
Conclusion and Future Work
We have presented a general technique to reduce network traffic when executing requests in Grid environments. Our approach is to build a DAG whose structure expresses the dependencies amongst the requests; the DAG is then scheduled for execution. Our initial experiments are promising and show that sequencing significantly reduces the execution time of our client application. As a final thought, we offer Fig. 9 which shows that even at its worst, request sequencing improves execution time by a factor of about 1.5. Though we have not proven this result, it is our belief that in most cases, sequencing should never decrease performance. Section 3.5 mentions that the sequences are currently restricted to execution on a single server. The next logical progression is to allow different components of the sequence to execute on different hosts. The implications are that no single server needs to possess all the software capabilities for the sequence. This also means that the modules will truly be able to execute in parallel even when no parallel machine is present. Scheduling techniques as discussed by [3] will be evaluated, and we will incorporate factors like computational and communication costs to better approximate optimal solutions. It makes little sense to execute the components of a sequence on various servers without taking data locality into account. Tools like the Internet Backplane Protocol[10] will be leveraged to provide all servers with convenient access to the necessary data.
1222
Dorian C. Arnold, Dieter Bachmann, and Jack Dongarra 5
Clustering speedup PCA speedup
4
Speedup
3
2
1
0
0
100
200
300 400 Bandwidth in kByte/s
500
600
700
Fig. 9. Reduction in Execution Time due to Request Sequencing
References [1] J. Jamison and R. Wilder. vBNS: The Internet Fast Lane for Research and Education. IEEE Communications Magazine, 35(1):60–63, January 1997. [2] N. Boden, D. Cohen, R. Felderman, A. Kulawik, C. Seitz, J. Seizovic, and W. Su. Myrinet: A Gigabit per Second Local Area Network. IEEE-Micro, 15:29–36, February 1995. [3] Y. Kwok and I. Ahmad. Benchmarking and Comparison of the Task Graph Scheduling Algorithms. Journal of Parallel and Distributed Computing, 59(3):381– 422, December 1999. [4] J. Dennis. Data Flow Supercomputers. IEEE Computer, 13(11):48–56, November 1980. [5] S. Sekiguchi, M. Sato, H. Nakada, S. Matsuoka, and U. Nagashima. Ninf : Network based Information Library for Globally High Performance Computing. In Proc. of Parallel Object-Oriented Methods and Applications (POOMA), Santa Fe, CA, 1996. [6] M. Litzkow, M. Livny, and M. Mutka. Condor - A Hunter of Idle Workstations. In Proc. of the 8th International Conference of Distributed Computing Systems, San Jose, CA, pages 104–111, June 1988. [7] H. Casanova and J. Dongarra. NetSolve’s Network Enabled Server: Examples and Applications. IEEE Computational Science & Engineering, 5(3):57–67, September 1998. [8] S. Parker and C. Schmechel. RFC 2398: Some testing tools for TCP implementors, August 1998. [9] J. A. Richards. Remote Sensing Digital Image Analysis. Springer, 2nd edition, 1993. [10] J. Plank, M. Beck, W. Elwasif, , T. Moore, M. Swany, and R. Wolski. IBP – The Internet Backplane Protocol: Storage in the Network. In NetStore ’99: Network Storage Symposium, Seatle, WA, October 1999.
An Architectural Meta-application Model for Coarse Grained Metacomputing Stephan Kindermann1 and Torsten Fink2 1
University of Erlangen-Nuremberg, Germany [email protected] 2 Free University of Berlin, Germany [email protected]
Abstract The emerging infrastructures supporting transparent use of heterogeneous distributed resources enable the design of a new class of applications. These meta-applications are composed of distributed software components. In this paper we describe a new model for component based meta-application design based on a formal architectural description of the gross organization of an application. This structural description is enriched by a formal process algebraic characterization of component behavior. Using this behavioral model we can formally check meta-applications in an early development phase. We present simple architectural styles developed to support data-flow and control-flow driven meta-application design on top of the Amica metacomputing infrastructure.
1
Introduction
There is a growing interest in defining and constructing an infrastructure which gives users the illusion that distributed, heterogeneous (computing and storage) resources constitute one giant transparent environment, a metacomputer [8]. It enables the development of a new class of applications: meta-applications. They are composed of multiple (partly reusable) components. Looking at the current practice of meta-application development there is no agreement on a common programming model for component based application design. In this paper we describe an abstract, formally concise, and extendable programming model defined to develop meta-applications on top of our metacomputing environment Amica1 [2]. The overall organization of meta-applications are given in a formal architecture description language (ADL) (see e.g. [6]). Different architectural styles [13] are defined to build up a basic description vocabulary which can be refined to define domain specific extensions. For every element of the vocabulary its behavior is defined by a process algebra term. The overall behavioral model of an application is automatically derived through the appropriate composition of the behavioral description of its components. This model can then be checked against a set of formal requirements (e.g. liveness 1
Abstract Metacomputing Infrastructure for Coarse Grained Applications
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1223–1230, 2000. c Springer-Verlag Berlin Heidelberg 2000
1224
Stephan Kindermann and Torsten Fink
and progress properties). This allows formal correctness checks on Amica metaapplications in an early development phase. The remainder of this article is as follows. In Sect. 2 we introduce shortly the basic concepts and services of Amica. In Sect. 3 we introduce our new programming model. In Sect. 4 we describe firstly how applications given in our programming model are executed using Amica and secondly how a behavioral model of the application is generated automatically which can be used as input to formal analysis tools. In Sect. 5 we give an overview of related work and discuss the advantages of our approach. Finally, an outlook to future work is given.
2
The Amica Metacomputing Infrastructure
Amica has been designed as a prototypical middleware foundation to investigate the composition of metacomputing applications from reusable components (e.g. legacy systems). Amica provides abstraction of heterogeneous distributed data storage by data objects and of computing facilities by metabricks. Additional application specific code can be integrated using so called user bricks. Data objects (possibly replicated) are related dynamically to real storage resources (data store objects) interconnected by link objects which provide abstraction of the networking infrastructure of the metacomputer. Metabricks are related to the basic computational services (bricks) based on a broker mechanism which looks for appropriate brick factories. The concise meaning of ’appropriate’ is given by a cost function which takes into account the current load of computing resources provided by special objects named computation units. Essentially, Amica provides an infrastructure to carry out the instantiation of abstract data storage and computing service requests transparently taking into account the current load within the distributed system. The implementation relies on the standard middleware foundation CORBA and uses in the current version an interpreter based instantiation approach. For a more detailed description of the Amica infrastructure see [2].
3
The Amica Programming Model
A metacomputing application on top of Amica is composed of data storage and computation components which are directly related to the abstract data objects and metabricks of Amica. User-bricks allow additional application specific code to be integrated. A basic set of control flow and data flow connectors is used to specify and control component activation and interaction. In the following we formally describe the organization of meta-applications as a hierarchical collection of interacting components with well defined properties and interfaces. This structural description is combined with a formal characterization of component behavior based on a process algebra. On this combination we build up a basic vocabulary for meta-application description.
An Architectural Meta-application Model
3.1
1225
Architecture Description
Despite the variety of existing software architecture description languages (ADLs) there is a considerable agreement about the role of structure in architecture description. Our description of metacomputing applications is based on ACME [5] which emerged from a joint effort of the architecture research community to provide a common intermediate representation for different ADLs. Meta-applications are given as collections of components interconnected by connectors in a meta-application graph. The structural description is enriched by a behavioral characterization of components and connectors using the process algebraic description language Lotos [10]. Definition 1. A meta-application graph (MG) is given as a bipartite graph, which is characterized by a quintuple G = (N odes, I, E, Beh). – The N odes of the graph consist of a set of components Comp and a disjoint set of connectors Conn: N odes = Comp ∪ Conn, Comp ∩ Conn = {}. – The function I : N odes → 2IP associates each node with its set of interconnection points (∈ IP ). Interconnection points are subdivided into ports (for components) and roles (for connectors). (IP = P orts ∪ Roles). – The interconnection of components and connectors via their ports and roles is given by the relation E ⊆ (Comp × P orts) × (Conn × Roles). – The function Beh : N odes → LotosT erm associates each node to its behavioral description in form of a Lotos process algebra term. For a node c in a meta-application graph with an interconnection point p1 and an associated set of actions {a1 , .., an } (e.g. services needed or provided or events emitted at the interconnection point) the associated process term Beh(c) defines a Lotos process c with parameter list [p1 a1 , .., p1 an ]. This list is extended accordingly if multiple interconnection points are defined for a node. Interaction with other (node-) processes is exclusively possible via synchronization with these externally oberservable actions. 3.2
An Architectural Style for Amica Meta-applications
Each node and each interconnection point in a meta-application graph is an instance of a type from a set of predefined type definitions. These types are used to build up a basic vocabulary Voc to describe the architecture of a metaapplication. This vocabulary along with a set of constraints is often called an architecture style [13]. Definition 2. A meta-application vocabulary V oc to build up (behavioral) metaapplication graphs is given as a quintuple (N T, IP T, SC, BC) where N T is a set of node (component or connector) types, IP T is a set of interconnection point (port or role) types, SC is a set of structural constraints, and BC is a set of behavioral constraints.
1226
Stephan Kindermann and Torsten Fink 0..1
DReadPT DobPT
DWritePT
CreatePT 1
0..1
DeletePT 1 0..1
CinPT
CoutPT
◆ DataObjectCpT ◆ DataObjectType type
1 DFlowCnT ◆
1
◆ CFlowCpT ◆ WorkerCpT
MetaBrickCpT ServiceT service
DobRT
FarmCnT
0..1
DAccessPT * String para
DAccessRT
1..* CFlowCnT ◆
UserBrickCpT
CinRT
◆ CStartCnT
1..* CoutRT
◆ CEndCnT
Fig.1. Basic meta-application vocabulary
The description of metacomputing applications on top of Amica is currently based on a simple architectural vocabulary which is illustrated in Fig. 1. It contains component and connector types to characterize the flow of control and the flow of data, to describe farm parallelism, and to provide access to data storage and computing resources. For brevity we omit the structural and semantic constraints. In general, structural constraints are given by the associations in the class diagram and first order logic predicates. Semantic constraints currently are given in an action based temporal logic. Data is stored in instances of the type DataObjectCpT. Its definition includes two port types for read-write access and two port types to create and delete object instances. The access to data object components is done via connectors of the type DFlowCnT. It provides roles for connection to data objects and to objects needing data access. Each component which is wired in control flow is an instance of type CFlowCpT. This type defines two interconnection points of type CinPT,CoutPT, characterizing incoming and outgoing control flow. Control flow components are connected to control flow connectors (type CFlowCnT). Different subtypes characterize connectors which split and combine control flow.
ci:CinRT cconn co:CoutRT ci:CinPT ccomp co:CoutPT ci
τ
co
candi1
i1 i2
cor i2 i2 i1
τ
w_rep d: DeletePT
w_ind
c
w: DWritePT rep d ind r_ind rep ind r_rep r: DReadPT
o
v
[x = 0] -> o1
ci ci
o
o1 o2
v?x
[x =|= 0] -> o2
DO:DataObjectCpT c: CreatePT
split choice
o
dflow:DFlowCnT do: DobRT ind rep
c_conf
c_req do_ind do_rep
c:DAccessRT req conf
Fig.2. Basic data and control flow components and connectors
o1 o2
An Architectural Meta-application Model
1227
Examples of basic data and control flow components are given in Fig. 2. Their basic behavior is illustrated in the form of labeled transition systems. Thus control flow is simply propagated and data flow is based on a simple request– confirm protocol. Instances of the special connector type FarmCnT are used to express and control ”bag of task” like parallel computations. Farm connectors are used only in a well defined cooperation with data object and worker components, see Fig. 3. After started the farm reads in a bag containing the tasks to be distributed (using bi:DobRT) and starts a number of workers over ws:CoutRT (this number is given as a property value of the connector). Then the tasks are distributed to the workers using role do:DAccessRT. Thereafter the results are collected over di:DAccessRT and stored in a result bag data object attached to role bo. This description of the behavior corresponds directly to the Lotos description given as skeleton in Fig. 3.
:WorkerCpT
:DataObjectCpT :DWritePT :DReadPT
:DataObjectCpT :DWritePT :DReadPT
:FarmCnT bi:DobRT
ws:CoutRT we:CinRT
ci:CinRT workers:Int
bo:DobRT co:CoutRT
do:DAccessRT di:DAccessRT
ci:CinPT
co do_conf
co:CoutPT di:DAccessPT
do_req τ
ci di_req di_conf
do:DAccessPT
Beh(:FarmCnT) = process farm[ci,co,ws,we,bi_ind,bi_rep,bo_ind,bo_rep,di_req,di_conf,do_req,do_conf]:exit := ci; bi_ind;bi_rep?n:Nat; ( WorkerStart[ws](workers) >> ( DistributeTasks[do_req,do_conf](n) ||| CollectResults[di_req,di_conf](n) ) >> WorkerEnd[we](workers) >> (bo_ind!n;bo_rep;co;exit) ) Data Flow |[ws,we,do_req,do_conf,di_req,di_conf]| WorkerGroup[ws,we,do_req,do_conf,di_req,di_conf] endproc
Control Flow
where
process WorkerGroup[ci,co,di_req,di_conf,do_req,do_conf]:exit := (worker[ci,co,di_req,di_conf,do_req,do_conf] |||...||| worker[ci,co,di_req,di_conf,do_req,do_conf] ) endproc
Fig.3. Bag of task like parallelism based on farm connector and worker component
3.3
A Small Example
A simple exemplary composition based on our vocabulary defining a ray tracing meta-application is given in Fig. 4. Two data objects are involved, scen stores the three dimensional scenario and pic stores the generated picture. In init these data objects are created and initialised. Then, the remote computation is started by a metabrick. In parallel the user can monitor the current state of the picture by a specialized user brick. Using the graphical
1228
Stephan Kindermann and Torsten Fink
init:UserBrickCpT
:CinPT
:CStartCnT :CoutRT
:DAccessPT
:DAccessPT
para="picture"
para="scenario"
:CoutPT
:DAccessRT
pic :DataObjectCpT
:CreatePT
:DobRT
:DeletePT
:ViewPicUBCpT
:DobRT
:DFlowCnT
:DAccessRT
:DAccessRT
:CEndCnT :CinRT
type = Text
:DobRT
:DobRT
:DFlowCnT
:DataObjectCpT
:DWritePT :DReadPT
:DAccessPT para="picture" :CoutPT
:DeletePT
:DobRT
:CinRT :CoutRT
:DReadPT
:CinPT
:CreatePT scen
:DFlowCnT :CFlowCnT
:DWritePT
type = Picture
:DAccessRT
:DFlowCnT
:CinPT
:DFlowCnT
:MetaBrickCpT
:DAccessRT
service = PoVRay :DAccessPT :DAccessPT para="output" para="scenario" :CoutPT
Fig.4. An exemplary application
front-end for ACME this application can be intuitively defined by drag-anddrop nodes out of our meta-application vocabulary. Another simple application to parallel simulation of mobile communication systems is described in [2].
4
Meta-application Execution and Formal Analysis
A meta-application graph is dynamically interpreted and mapped to the Amica metacomputer. Also automatically a global behavioral model of the meta-application is generated in a compositional way. This model can be analyzed and checked for design errors (e.g. resulting from component composition mismatches). These two steps, namely interpretation and behavioral model generation are discussed in the following section in more detail. A simple meta-application interpreter is used to map our basic component and connector vocabulary to the services provided by Amica: Components of type DataObjectCpT and MetaBrickCpT are directly correlated to the data object and metabrick abstractions of basic data storage and computing services provided by Amica. The data store network and the broker mechanism of Amica relate these dynamically to the distributed data objects and bricks (created by brick factories). Data flow connectors and control flow connectors are interpreted to control component interaction and activation. In the special case of a farm connector (type FarmCnT) a specified number of workers is instantiated, the tasks contained in a bag are distributed and the results are collected. The structure information given in the meta-application graph is used to compose the individual node behaviors to the combined overall system behavior. Composition is based on appropriate synchronization of component and connector actions. To correlate these actions we have to define a renaming operator
An Architectural Meta-application Model
1229
\{(old1 \new1 ), .., (oldn \newn )} which replaces all actions oldi act over interconnection point names oldi to to newi act. Given a behavioral meta-application graph G = (N odes, I, E, Beh) with N odes = Comp ∪ Conn = {CP1 , .., CPN } ∪ {CN1 , ..CNM } the associated overall system behavior is given by the parallel (full interleaving |||) composition of all component instances and also all connector instances. These two groups synchronize over all actions of their interconnection points (||). Thus the general structure of the overall system behavior description is given as the following process term scheme: (N CP1 ||| ... ||| N CPN ) || (N CN1 ||| ... ||| N CNM ) where N CPi = Beh(CPi )\{(old\new) | old ∈ I(CPi ) ∧ new = CNj newp ∧ attachedCN (CPi , old) = (CNj , newp)} N CNi = Beh(CNi )\{(old\new) | old ∈ I(CNi ) ∧ new = CNi old} The function attachedCN used above gives exactly one attached (connector, role) pair for a given component and port, that is attachments of multiple roles to a port are disallowed. In the case of multiple attachments of ports to a role, the above scheme implies an or semantics (the role-action synchronizes with an action associated with one of the attached ports). We have extended this to generally allow n-port to one role attachments with different semantics (e.g. and) given by the type of the role. This generation scheme was implemented in Java and we apply powerful formal analysis tools (the CADP tool set [4] and the model checker XTL [11]) for abstract meta-application property checks.
5
Related Work and Conclusion
Different approaches to build component based meta-applications are used in literature, ranging from simple data-flow models to general component frameworks. In WebFlow [9] (restricted) data flow graph models are proposed for component composition. In [14] a distributed component architecture toolkit is described for meta-application design on top of the Globus metacomputing infrastructure [3]. Also scripting languages can be used for meta-application description, see e.g. [12]. In contrast to these approaches having no or only a very restricted semantic foundation we use a more flexible and general formal process algebraic model combined with an architectural description of meta-applications using a well defined (and extendable) set of components and connectors. Other approaches exist which are promoting the general idea of combining an architectural description with a formal behavioral model in the more general context of distributed software design. The ADL Darwin is combined with labeled transition systems in [7] to facilitate a compositional reachability analysis and in [1] CSP is used within the ADL Wright. To conclude, we have described a formal model supporting component based meta-application development. A formal architecture description language was
1230
Stephan Kindermann and Torsten Fink
combined with a process algebraic behavioral description to define a basic extendable vocabulary of component types for meta-application development. This vocabulary has been implemented on top of the Amica metacomputing infrastructure. Component compositions can automatically be checked based on well known state space analysis methods (e.g. model checking). Next steps include an extension of our vocabulary (including e.g. event propagation and handling) and an application to more complex problems. As we want to check not only functional but also performance properties of meta-applications we plan to use a stochastic process algebra description of component behavior. Also a behavioral model of the Amica infrastructure itself will be integrated in the future.
References [1] R. Allen and D. Garlan. A formal basis for architectural connection. ACM TOSEM, 6(3):213–249, July 1997. [2] T. Fink and S. Kindermann. First steps in metacomputing with Amica. In Euromicro-PDP 2000, pages 197–204. IEEE Computer Society, 2000. [3] I. Foster and C. Kesselman. Globus: A metacomputing infrastructure toolkit. The International Journal of Supercomputer Applications and High Performance Computing, 11(2):115–128, 1997. [4] H. Garavel, M. Jorgensen, R. Mateescu, C. Pecheur, M. Sighireanu, and B. Vivien. CADP’97 - status, applications, and perspectives. In Proceedings of 2nd COST 247 Int. Workshop on Applied Formal Methods in System Design, 1997. [5] D. Garlan, R.T. Monroe, and D. Wile. ACME: An architecture description interchange language. In Proceedings of CASCON ’97, November 1997. [6] D. Garlan and M. Shaw. Software Architecture: Perspectives on an emerging Discipline. Prentice Hall, April 1996. [7] D. Giannakopoulou, J. Kramer, and S.C. Cheung. Behaviour analysis of distributed systems using Tracta. Journal of Automated Software Engineering, 6(1):7–35, January 1999. R. Cleaveland and D. Jackson, Eds. [8] A. Grimshaw, A. Ferrari, G. Lindahl, and K. Holcomb. Metasystems. Communications of the ACM, 41(11), 1998. [9] T. Haupt, E. Akarsu, and G. Fox. Webflow: a framework for web based metacomputing. In HPCN Europe ’99, April 1999. [10] ISO/IEC. Lotos — a formal description technique based on the temporal ordering of observational behaviour. International Standard 8807, ISO — Information Processing Systems — OSI, Gen`eve, September 1988. [11] R. Mateescu and H. Garavel. Xtl: A meta-language and tool for temporal logic model-checking. In Tiziana Margaria, editor, STTT’98 (Denmark), July 1998. [12] R.P. Mc Cormack, J.E. Koontz, and J. Devaney. Seamless computing with WebSubmit. Concurrency: Practice and Experience, 11(15):946–963, 1999. [13] M. Shaw and P. Clements. A field guide to boxology: Preliminary classification of architectural styles for software systems. In Proceedings COMPSAC, 1997. [14] J. Villacis, M. Govindaraju, D. Stern, A. Withaker, F. Berg, P. Deuskar, T. Benjamin, D. Gannon, and R. Bramley. Cat: A high performance, distributed component architecture toolkit for the grid. In Proceedings of the High Performance Distributed Computing Conference, 1999.
Javelin 2.0: Java-Based Parallel Computing on the Internet Michael O. Neary, Alan Phipps, Steven Richman, and Peter Cappello Department of Computer Science University of California, Santa Barbara Santa Barbara, CA 93106 {neary, evodius, joy, cappello}@cs.ucsb.edu
Abstract. This paper presents Javelin 2.0. It presents architectural enhancements that facilitate aggregating larger sets of host processors. It then presents: a branch-and-bound computational model, the supporting architecture, a scalable task scheduler using distributed work stealing, a distributed eager scheduler implementing fault tolerance, and the results of performance experiments. Javelin 2.0 frees application developers from concerns about complex interprocessor communication and fault tolerance among Internetworked hosts. When all or part of their application can be cast as a piecework or a branch-and-bound computation, Javelin 2.0 allows developers to focus on the underlying application.
1
Introduction
Our goal is to harness the Internet’s vast, growing, computational capacity for ultra-large, coarse-grained parallel applications. By providing a portable, secure programming system, Java holds the promise of harnessing this large heterogeneous computer network as a single, homogeneous, multi-user multiprocessor [1]. Some research projects that are designed to exploit this include Charlotte [4], Atlas [3], Popcorn [6], Javelin [7], Bayanihan [12], Manta [13], Ajents [8], and Globe [2]. Javelin 2.0 is designed to achieve two goals: 1) Obtain the performance of a massively parallel implementation; 2) Provide a simple API, allowing designers to focus on a recursive decomposition/composition of the parallelizable part of the computation. The application programmer gets the performance benefits of massive parallelism, without adulterating the application logic with interprocessor communication details and fault tolerance schemes. The resulting code should run well on a set of processors that changes during execution. Javelin 2.0 handles all interprocessor communication and fault tolerance for the application programmer, when the parallelizable computation can be cast as a branch-andbound (or piecework) computation. This is a broad class of computations. We focus here on 2 fundamental issues: – Scalable Performance — If there is no niche where Java-based global computing outperforms existing multiprocessor systems, then there is no reason to use it. The architecture must scale to a higher degree than existing multiprocessor architectures, such as networks of workstations. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1231–1238, 2000. c Springer-Verlag Berlin Heidelberg 2000
1232
Michael O. Neary et al.
– Fault tolerance — An architecture that scales to thousands of hosts must be fault tolerant, particularly when hosts, in addition to failing, may dynamically disassociate from an ongoing computation. Javelin 2.0 extends the piecework computational model to a branch-and-bound model, which is implemented using a weak form of shared memory that itself is implemented via the pipelined RAM [9] model of cache consistency. This shared memory model is strong enough to support branch-and-bound computation (in particular, bound propagation), but weak enough to be fast. Using this cache consistency model, we present a high-performance, scalable, fault tolerant Internet architecture for branch-and-bound computations, such as are used to solve NP-complete problems. For such an architecture to succeed, the architects must be diligently cognizant of the central technical constraint: On the Internet, communication latency is large.
2
Model of Computation
The branch-and-bound method, which generalizes the piecework model of computation [10], intelligently enumerates all feasible points of a combinatorial optimization problem: not all feasible solutions are examined. Branch-and-bound, in effect, proves that the best solution is found without necessarily examining all feasible solutions. The method successively partitions the solution space (branches), and prunes a subspace, when there is sufficient information to infer that none of the subspace’s solutions are as good as a current solution (bound). (See Papadimitriou and Steiglitz [11] for a more complete discussion of branchand-bound.) The computational model implies the following requirements: 1) Tasks (elements of the activeset) are generated during the host computation; 2) When a host discovers a new best cost, it propagates it to the other hosts; 3) Detecting termination in a distributed implementation requires knowing when all subspaces (children) have been either fully examined or killed. The challenge, in sum, is, with a minimum of communication, to enable: a) hosts to create tasks, which subsequently can be stolen; b) hosts to propagate new bounds rapidly to all hosts; c) the eager scheduler to detect tasks that have been completed or killed. The last item is needed not just for termination detection, but for fault tolerance, to determine which tasks need to be rescheduled.
3
Architecture
The Javelin 2.0 system architecture retains the basic structure of its predecessors, Javelin [7] and Javelin++ [10]. There are three system entities — clients, brokers, and hosts. A client is a process seeking computing resources; a host is a process offering computing resources; a broker is a process that coordinates the allocation of computing resources.
Javelin 2.0: Java-Based Parallel Computing on the Internet
3.1
1233
Javelin Broker Name Service
When a host (or client) wants to connect to Javelin, it first must find a broker that is willing to serve it. The JavelinBNS system is a scalable, fault tolerant directory service that enables the discovery of a nearby Javelin broker, without any prior knowledge of the broker network structure. It is designed not only to aid hosts who are searching for brokers, but also to aid brokers who are looking for neighboring brokers. A JavelinBNS system consists of at least two fully replicated JavelinBNS servers. Each server is responsible for managing a list of available brokers, responding to broker lookup requests, and ensuring that the other JavelinBNS nodes contain the same information. The JavelinBNS system thus serves as an information backbone for the entire Javelin 2.0 system. Since the information stored for each broker is relatively small, the service will scale to a very large number of brokers. A small number of BNS servers will therefore be capable of administering thousands of broker entries, so a fully connected network of BNS servers will not be a bottleneck. At regular intervals, information is exchanged by the BNS servers. If a BNS server crashes and subsequently restarts, it can simply reload its tables with the information from its neighbors, thus providing for fault tolerance. Figure 1 shows the steps involved in a broker lookup operation.
BNS
2. BNS lookup
1. Register with BNS
3. Broker list
Broker
5. Connect to selected broker
Host
4. Ping brokers
Fig. 1. JavelinBNS lookup sequence.
3.2
Broker Network & Host Tree Management
The topology of the broker network is an unrestricted graph of bounded degree. Thus, at any time a broker can only communicate with a constant number of other brokers. Similarly, a broker can only handle a constant number of hosts. If that limit is exceeded adequate steps must be taken to redirect hosts to other brokers. The bounds on both types of connection give the broker network the potential to scale to arbitrary numbers of participants. At the same time, the degree of connectivity is higher than in a tree-based topology. When a host connects to a broker, the broker enters the host in a logical tree structure. The top-level host in the tree will not receive a parent; instead it will
1234
Michael O. Neary et al.
later become a child of the client. This way, the broker maintains a preorganized tree of hosts which are set on standby until a client becomes active. When a client connects, or client information is remotely received from a neighboring broker, the whole tree is activated in a single operation and the client information is passed to the hosts. Brokers can individually set the branching factors of their trees, and decide how many hosts they can administer. In case of a host failure, the failed node is detected by its children and the broker restructures the tree in a heap-like operation (for details, see [10]).
4 4.1
Scalable Computation & Fault Tolerance The Scheduler
The fundamental concept underlying our approach to task scheduling is work stealing, a distributed scheduling scheme made popular by the Cilk project [5]. Work stealing is entirely demand driven — when a host runs out of work it requests work from some host that it knows. Work stealing balances the computational load, as long as the number of tasks is high relative to the number of hosts — a property well suited for adaptively parallel systems. In Javelin 2.0, tasks get split in a double-ended task queue until a certain minimum granularity — determined by the application — is reached. Then, they are processed. When a host runs out of local tasks, it selects a neighboring host and requests work from that host. Since the hosts are organized as a tree, the selection of the host to steal work from follows a deterministic algorithm based on the tree structure. Initially, each host retrieves work from its parent, and computes one task at a time. When a host finishes all the work in its deque, it attempts to steal work, first from its children, if any, and, if that fails, from its parent. This strategy ensures that all the work assigned to the subtree rooted at a host gets done before that host requests new work from its parent. Work stealing helps each host get a quantity of work that is commensurate with its capabilities. The client is the root of its tree of hosts. 4.2
Shared Memory
For branch-and-bound computation, only a small amount of shared memory is needed and a weak shared memory model suffices. The small amount is because only one integer is needed to represent a solution’s cost. The weak model suffices because if a host’s copy of best cost is stale, correctness is unaffected. Only performance may suffer — we might search a subspace that could be pruned. It thus suffices to implement the shared memory using a pipelined RAM (aka PRAM) model of cache consistency. This weak cache consistency model can be implemented with scalable performance, even in an Internet setting. There are several methods to propagate bounds among hosts. We use the following: When a host discovers a solution with a better cost than its cached best cost, it sends this solution to the client. If the client agrees that this indeed
Javelin 2.0: Java-Based Parallel Computing on the Internet
1235
is a new best cost solution (it may not be, due to certain race conditions), it updates its cached best cost solution, and “broadcasts” the new best cost to its entire tree of hosts. That is, it propagates the new best cost to its children, who in turn propagate it to their children, level by level down the host tree. 4.3
Fault Tolerance
Eager scheduling reschedules a task to an idle processor in case its result has not been reported. It was introduced and made popular by the Charlotte project [4], and also has been used successfully in Bayanihan [12]. Javelin++ [10] also uses eager scheduling to achieve fault tolerance and load balancing. It efficiently and relentlessly progresses towards the overall solution in the presence of host and link failures, and varying host processing speeds. The Javelin 2.0 eager scheduler is located on the client. Although this may seem like a bottleneck with respect to scalability, it is not, as we shall explain below. Eager scheduling, however, is more challenging for branch-and-bound computation (as compared to piecework computation). Besides detecting positive results, (i.e., new best cost solutions), the eager scheduler must detect negative results: solution [subspaces] that have been examined and do not contain a new best cost solution, and solution subspaces that have been pruned. Performance, though, requires avoiding unnecessary communication and computation. In a branch-and-bound computation, the size of the feasible solution space is exponential in the size of the input. In principle, the algorithm may need to examine all of these exponentially many feasible solutions to find the minimum cost solution. In practice, a partial solution, p, is “killed” (a subspace is pruned) when the lower bound on the cost of any feasible solution that is an extension of p must be more costly than the currently known minimum cost solution. The algorithm nonetheless must gather sufficient information to detect that the minimum cost solution has indeed been found. This implies that killed nodes and sub-optimal solutions must be detected by the eager scheduler. If a separate communication is required to detect each such event, the overall quantity of communication would nullify the benefits of parallelism. We cope with this communication overload by aggregating portions of the search space into atomic tasks, and similarly aggregating negative results into one communication per atomic task. This lets the eager scheduler know that this part of the problem tree has been searched, and hence need not be rescheduled. The number of negative communications consequently is equal to or less than the number of atomic tasks. In practice, it is much less than the number of atomic tasks; many are killed. We can adjust the computation/communication ratio by adjusting the size of atomic tasks, in order to decrease the overall run time. Performance is quite sensitive to atomic task size, so finding good size values is important. For performance reasons, we balance the computational size of the hosts’ atomic tasks with the client’s computation of result handling and eager scheduling, so that neither the client nor the hosts have to wait for one another. Additionally, we want the number of atomic tasks to be much larger than the number of hosts, to keep them all well utilized, even when some are much faster than others.
1236
5
Michael O. Neary et al.
Experimental Results
All experiments were run in campus computer labs under a typical workload. The heterogeneous test environment consists of 4 Sun Enterprise 450 dual/quadprocessors with a processor speed of 400 MHz; 49 Celeron 466/500 MHz processors; and a Beowulf cluster of 42 nodes, with 6 Pentium III 500 MHz quadprocessors, and 36 Pentium II 400 MHz dual processors. The cluster is running Red Hat Linux 6.0. All other machines are running Solaris 2.7. We used JDK 1.2 with active JIT for our experiments. We tested the performance of Javelin 2.0 with a TSP application. The test graphs are complete, undirected, weighted graphs of 22 and 24 nodes, with randomly generated integer edge weights. These graphs are complex enough to justify parallel computing, but small enough to enable us to run tests in a reasonable amount of time. The 22-node graph took approximately 3 hours to process on a Sun E450. The 24-node graph took just under 10 hours on the same processor. The term “speedup” is somewhat confusing here. Traditionally, speedup is measured on a dedicated multiprocessor, where all processors are homogeneous in hardware and software configuration, and varying workloads between processors do not exist. Thus, speedup is well defined as T1 /Tp , where T1 is the time a program takes on one processor and Tp is the time the same program takes on p processors. Therefore, strictly speaking, in a heterogeneous environment like ours the term speedup cannot be used anymore. Even if one tries to run tests in as homogeneous a hardware setup as possible, the varying workloads on both the OS and the network can amount to big differences in the individual performance of hosts. However, from a practical standpoint, a user running an application on Javelin 2.0 with a large set of hosts will definitely see “speedup”; the application will run faster than on a single machine. We will use the term practical speedup to distinguish between the two scenarios. In the following, we may omit the word “practical” when the meaning is clear from the context. We now give a more formal definition of our notion of practical speedup: Let M1 , . . . , Mk denote k different processor types. Let T1 (i) denote the time to complete the problem using 1 processor of type Mi . Conventional speedup, using p processors of type Mi can be defined as T1 (i)/Tp (i). To compute speedup when we have more than one type of processor, we generalize this formula. Let a problem be solved concurrently using k types of processors, where there are pi processors of type Mi : The total number of processors is p = p1 + · · · + pk . Let Tp (p1 , . . . , pk ) denote the execution time when using this mix of p processors. We define a composite base case that reflects this mix of processors: T1 (p1 , . . . , pk ) =
p1 T1 (1) + · · · + pk T1 (k) . p1 + · · · + pk
Finally, we define the speedup S as S = T1 (p1 , . . . , pk )/Tp (p1 , . . . , pk ). While this definition does not incorporate machine and network load factors, it does reflect the heterogeneous nature of the set of machines.
Javelin 2.0: Java-Based Parallel Computing on the Internet
1237
Figure 2 shows the speedup we measured in our experiments and calculated according to the above formula. For the 22-node graph, speedup was superlinear at first, until it topped out at 77.26 for 100 hosts, when communication became a significant bottleneck. To observe superlinear speedup for the parallel TSP is quite common, due to the inherent irregularity of the input graph. The results for the 24-node graph illustrate this even further. Here, speedup was nowhere near as good, reaching only 23.35 for 80 hosts. However, the curve still shows a steady rate of improvement, and the larger graph has the potential to scale better due to its higher computation complexity.
100
Speedup
80
60
graph22 ideal graph24
40
20
0 0
20
40
60
80
100
Processors
Fig. 2. Practical Speedup for TSP on Javelin 2.0.
To sum up, a graph that took about 3 hours to calculate on a single computer took just under 3 minutes on 100 processors under their normal workloads. These results are encouraging, although they need to be evaluated with different input graphs and more hosts.
6
Conclusion
To enlarge the set of applications that can benefit from Javelin, Javelin 2.0 extends Javelin++’s piecework model of computation to a branch-and-bound model. The technical challenge is to implement a distributed shared memory that enables hosts to share bounds. We implemented the pipelined RAM model of cache consistency among hosts sharing the bound. Our experiments indicate that limited use of this weak shared memory poses no performance problem. To facilitate aggregating large numbers of hosts, Javelin 2.0 enhances host registration: The host can request the broker name system to return k broker names, where k is chosen by the host. Currently, the host then pings these brokers to discover the “nearest”. In Javelin 2.0, with but one Java RMI call on a broker, a client gets a handle to the broker’s entire preorganized host tree. Other brokers convey their host trees with a similar economy of communication.
1238
Michael O. Neary et al.
The TSP experiments suggest that branch-and-bound can be sped up efficiently, even with large numbers of Internetworked hosts. Many combinatorial optimization versions of NP-hard problems are solved with branch-and-bound. Our distributed deterministic work stealing scheduler integrates smoothly, not only with bound caching, but also with the distributed eager scheduler, which provides essential fault tolerance.
References [1] A. Alexandrov, M. Ibel, K. E. Schauser, and C. Scheiman. SuperWeb: Research Issues in Java-Based Global Computing. Concurrency: Practice and Experience, 9(6):535–553, June 1997. [2] A. Bakker, M. van Steen, and A. S. Tanenbaum. From Remote Object to Physically Distributed Objects. In Proc. 7th IEEE Workshop on Future Trends of Distributed Computing Systems, Cape Town, South Africa, Dec. 1999. [3] J. E. Baldeschwieler, R. D. Blumofe, and E. A. Brewer. ATLAS: An Infrastructure for Global Computing. In Proceedings of the Seventh ACM SIGOPS European Workshop on System Support for Worldwide Applications, 1996. [4] A. Baratloo, M. Karaul, Z. Kedem, and P. Wyckoff. Charlotte: Metacomputing on the Web. In Proceedings of the 9th Conference on Parallel and Distributed Computing Systems, 1996. [5] R. D. Blumofe, C. F. Joerg, B. C. Kuszmaul, C. E. Leiserson, K. H. Randall, and Y. Zhou. Cilk: An Efficient Multithreaded Runtime System. In 5th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPOPP ’95), pages 207–216, Santa Barbara, CA, July 1995. [6] N. Camiel, S. London, N. Nisan, and O. Regev. The POPCORN Project: Distributed Computation over the Internet in Java. In 6th International World Wide Web Conference, Apr. 1997. [7] B. O. Christiansen, P. Cappello, M. F. Ionescu, M. O. Neary, K. E. Schauser, and D. Wu. Javelin: Internet-Based Parallel Computing Using Java. Concurrency: Practice and Experience, 9(11):1139–1160, Nov. 1997. [8] M. Izatt, P. Chan, and T. Brecht. Ajents: Towards an Environment for Parallel, Distributed and Mobile Java Applications. In ACM 1999 Java Grande Conference, pages 15–24, San Francisco, June 1999. [9] Lipton and Sandberg. PRAM: A scalable shared memory. Technical report, Princeton University: Computer Science Department, CS-TR-180-88, Sept. 1988. [10] M. O. Neary, S. P. Brydon, P. Kmiec, S. Rollins, and P. Cappello. Javelin++: Scalability Issues in Global Computing. Concurrency: Practice and Experience, to appear, 2000. [11] C. H. Papadimitriou and K. Steiglitz. Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Inc., Englewood Cliffs, NJ, 1982. [12] L. F. G. Sarmenta and S. Hirano. Bayanihan: Building and Studying Web-Based Volunteer Computing Systems Using Java. Future Generation Computer Systems, 15(5-6):675–686, Oct. 1999. [13] R. van Nieupoort, J. Maassen, H. E. Bal, T. Kielmann, and R. Veldema. WideArea Parallel Computing in Java. In ACM 1999 Java Grande Conference, pages 8–14, San Francisco, June 1999.
Data Distribution for Parallel CORBA Objects Tsunehiko Kamachi1, Thierry Priol2 , and Christophe Ren´e2 1
C&C Media Research Laboratories, NEC Corporation, 4-1-1 Miyazaki, Miyamae-ku, Kawasaki, Kanagawa 216-8555 Japan 2 IRISA/INRIA, Campus de Beaulieu - 35042 Rennes Cedex, France
Abstract. The design of application for Computational Grids relies partly on communication paradigms. In most of the Grid experiments, message-passing has been the main paradigm either to let several processes from a single parallel application to exchange data or to allow several applications to communicate between each others. In this article, we advocate the use of a modern approach for programming a Grid. It is based on the use of distributed objects, namely parallel CORBA objects. We focus our attention on the handling of distributed data within parallel CORBA objects. We show some performance results that were obtained using a NEC Cenju-4 parallel machine connected to a PC cluster.
1
Introduction
With the availability of high-performance networking technologies, it is nowadays feasible to couple several computing resources together to offer a new kind of computing infrastructure that is called a Computational Grid [4]. Such system can be made of a set of heterogeneous computing resources that are interconnected together through multi-gigabit networks. Software infrastructures, such as Globus [3] or Legion [6], provide a set of basic services to support the execution of distributed and parallel programs. One of the problem that arises immediately is how to program such a computational Grid and what is the most suitable communication model for Grid-enabled applications ? It is very tempting to extend existing message-passing libraries so that they can be used for distributed programming. We believe that this approach cannot be seen as a viable solution for the future of Grid Computing. Instead, we advocate an approach that allows the combination of communication paradigms for parallel and distributed programming. This approach, called PaCO, is based on an extension to a well known and mature distributed object technology, namely CORBA.
2
Communication within a Computational Grid
There exist two main approaches to communicate within a computational grid. The first approach is to allow the execution of a parallel code over heterogeneous machines taking benefit of the available computing resources. Research works have recently lead to extend existing message passing libraries to be able A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1239–1249, 2000. c Springer-Verlag Berlin Heidelberg 2000
1240
Tsunehiko Kamachi, Thierry Priol, and Christophe Ren´e
to exchange data between heterogeneous computing resources, such as MPICH-G [5], PACX [1] or PLUS [11]. A parallel code, based on one of these communication libraries, can be executed on a Grid with some minor modifications. We think that such approach is relevant since the purpose of these Grid-enabled communication libraries is to allow parallel programming at a larger scale. Such libraries can also be used to connect several parallel codes together to perform coupled simulations. It constitutes the second approach. The objective is to solve new kind of problems that were not affordable due to the lack of computing resources. The aggregating of computing resources may allow the simulation, in a shorter time frame, of complex manufactured products for which different physical behaviors have to be taken into account (structural mechanics, computational fluid dynamics, electromagnetism, noise analysis, etc...). Moreover, distributed execution of simulation codes are nowadays imposed by the way the industrial companies work together to design manufactured products. It requires that each company, participating to the design of a manufactured product, to contribute to the simulation of the whole product by providing access to its own simulation tools. However, a company is often reluctant to give both its simulations tools and the necessary simulation data to other companies (that may act as competitors later on). Therefore, there is a strong need to have part of the simulation of the whole product performed on their own computing resources to avoid the exchange of confidential data (i.e. the model of the object to be simulated). Thus, there is a clear need to have a mechanism to let simulation codes to communicate together. However, such mechanism requires that it is capable of both transferring data and control efficiently between codes. We think that message-passing is not suitable to connect several parallel codes together. Indeed, message-passing was mainly designed for parallel programming and not for distributed programming; it is mainly to transfer data but not the control. As for instance, if one code would like to call a particular function into another code, this latter has to be modified in such a way that a message type is associated to this particular function. Such modification requires a deep understanding of the code. Moreover, entry points in a code are not really exposed to potential users that would like to include such code into its application. Communication paradigms, such as RPC or distributed objects, offer a much more attractive solution since the transfer of control is implemented by remote invocation that is as simple as calling a function or a method. However, they are not suitable for parallel programming due to their higher communication cost. It is thus clearly difficult to have a single communication paradigm for the programming of computational grids. We advocate an approach, like others [8], that consists in merging several communication paradigms in a coherent way so that they fit the requirements mentioned previously. The remainder of this paper is structured as follows. Section 2 discusses communication issues for Computational Grids. Section 3 gives an overview of the parallel CORBA object concept. Section 4 describes data redistribution within a parallel CORBA object. Section 5 provides some experimental results. Finally, we conclude in section 6 by laying the grounds for future enhancement.
Data Distribution for Parallel CORBA Objects
3
1241
Overview of Parallel CORBA Object
CORBA is a specification from the OMG (Object Management Group) to support distributed object-oriented applications. CORBA acts as a middleware that provides a set of services allowing the distribution of objects among a set of computing resources connected to a common network. Transparent remote method invocations are handled by an Object Request Broker (ORB) which provides a communication infrastructure independent of the underlying network. An object interface is specified using the Interface Definition Language (IDL). An IDL file contains a list of operations for a given object that can be invoked remotely. An IDL compiler is in charge of generating a stub for the client side and a skeleton at the server side. A stub is simply a proxy object that behaves as the object implementation at the server side. Its role is to deliver requests to the server. Similarly, the skeleton is an object that accepts requests from the ORB and delivers them to the object implementation. The concept of parallel CORBA object1
Machine A
interface[*:2*n] MatrixOperations { const long SIZE=100; typedef double Vector[SIZE]; typedef double Matrix[SIZE][SIZE]; void multiply(in dist[BLOCK][*] Matrix A, in Vector B, out dist[BLOCK] Vector C); void skal(in dist[BLOCK] Vector C, out csum double skal); };
Cluster of PCs Parallel CORBA object MPI Communication layer Object impl.
Object impl.
Object impl.
Object impl.
SPMD code
SPMD code
SPMD code
SPMD code
Skel.
Skel.
Skel.
Skel.
PBOA
PBOA
PBOA
PBOA
Client Extended -IDL compiler Stub
CORBA ORB
Fig. 1. Encapsulation of MPI-based parallel codes into CORBA objects.
is simply a collection of identical CORBA objects as shown in figure 1. It aims at encapsulating a MPI code into CORBA objects so that a MPI code can be fully integrated into a CORBA-based application. Our goal is to hide as much as possible of the problems that appear when dealing with coarse-grain parallelism on a distributed memory parallel architecture like a cluster of PCs. However, this is done without entailing a lost of performance when communicating with the MPI code. First of all, the calling of an operation by a client will result in the execution of the associated method by all objects belonging to the collection at the server side. Execution of parallel objects is based on the SPMD execution model. This parallel activation is done transparently by our system. Data distribution between the objects belonging to a collection is entirely handled by the system. However, to let the system to carry out parallel execution and data distribution between the objects of the collection, some specifications have to be added to the component interface. A parallel object interface is thus described 1
we will use parallel object from now on
1242
Tsunehiko Kamachi, Thierry Priol, and Christophe Ren´e
by an extended version of IDL, called Extended–IDL as shown in figure 1. It is a set of new keywords (in bold in the figure), added to the IDL syntax2 , to specify the number of objects in the collection, the shape of the virtual node array where objects of the collection will be mapped on, the data distribution modes associated with parameters and the collective operations applied to parameters of scalar types.
4
Data Redistribution in a Parallel Object
Application programmers have to specify, for each operation, parameters that have to be distributed among the collection and how they are distributed. Since they can define data distribution for a parameter in each parallel object differently, parameter values need to be redistributed. This problem is made difficult due to various configurations at both the client and the server side (data distribution modes, distributed dimension, object collection size or its virtual shape). It is thus necessary to provide a data redistribution mechanism, as part of the operation invocation, to facilitate the coupling of several parallel objects. Furthermore, while users are responsible for data distribution management within a parallel object, data redistribution should have to be handled by a runtime system in order to hide all the operations associated with the invocation of operations from or to a parallel object. MPI
Parallel Client Client Object
Stub
1
ORB 2
Parallel Server 3
Skeleton
4
1
4
Parallel Client Client Object
ORB
Stub
Parallel Server Skeleton
MPI
Server Object
4
Fig. 2. Master/slave approach (a)
4.1
MPI
4
1
1
MPI
Server Object
Through the CORBA ORB (b)
Design Considerations
The most obvious way to perform data redistribution is to use gather and scatter operations at the client and the server sides. Figure 2-a illustrates such a technique for an operation invocation with a in parameter array. Four steps are required: first, one of the stubs gathers distributed data from the client objects using the MPI communication layer (1). Then, it invokes one of the server objects 2
A more complete description of these extensions is given in [10, 12]
Data Distribution for Parallel CORBA Objects
1243
and sends gathered data to it through the CORBA ORB (2). During the third step, the skeleton of the activated server object receives the data from the ORB (3). Finally, the skeleton activates the remaining server objects and scatters data to them using MPI (4). Although this technique is simple to implement, it has some severe drawbacks. The gathering and scattering of data values associated with distributed parameters do not offer a good scalability when the number of objects increases. Data transfer between two collections is serialized through only one object. Furthermore, the gathering of data by one stub is memory consuming. To avoid this problem, we need to incorporate both a parallel invocation of operations and a data redistribution strategy into parallel objects. One possible approach is to let client objects to send data values of the distributed parameters to the server objects through the ORB as shown in figure 2-b. In this approach, each client splits its own data according to the data distribution at the server side, and sends them to the relevant server objects directly. On the server side, each object receives pieces of data which it should own from several client objects. This approach suffers from a higher number of ORB requests compared to the master/slave approach. Since the ORB is usually much more slower than message-passing layers, we have to keep the number of requests as low as possible. Taken into account these remarks, we propose the following technique. Data MPI
Parallel Client Client Object
Stub
ORB
Parallel Server Skeleton
MPI
Server Object
Fig. 3. Data redistribution in the client stub. redistribution is performed by the client side or the server side before or after the sending of the request associated with the operation invocation. Figure 3 shows the case of data redistribution at the client side. Accordingly, all communications needed for the redistribution are carried out over the high-speed network of the parallel machines on which the parallel object, acting as the client, is running. The reason why we incorporated data redistribution into both the client and server sides is to obtain the maximum performance from the available communication resources. This allows us to select the most suitable place to perform data redistribution depending on the performance of the network at the client or the server side. More precisely, when data redistribution is performed at the client side, it is handled by the stubs that are aware of the data distribution mode of both the client and the server. First, all the stubs exchange their own data using the MPI communication layer to prepare data meeting the distribu-
1244
Tsunehiko Kamachi, Thierry Priol, and Christophe Ren´e
tion mode at the server object, then each stub sends the redistributed data to the relevant server object through the ORB. When the data redistribution is performed by the server, the client has to provide to the server extra information associated with the distribution of the parameter along with the parameter data values. Such information includes, for each distributed parameter, the distribution mode and the distributed dimension; the virtual shape of the objects is also part of this information. When the skeleton receives such information, it redistributes the parameter data values to adapt the client data mapping to the server one.
4.2
Implementation
The generation of the stub code is rather complex due to various possibilities at the client side. As for instance, a client can be either a standard CORBA object or a parallel object. In the later case, data that have to be sent when invoking an operation on a parallel object are distributed among the objects of the client collection. Therefore, one possibility is to perform the data redistribution by the stub, using a data redistribution library, so that it fits the one at the server side. Another modification to the stub code concerns the invocation mechanism. Since the number of objects of the collection at the client side does not often coincides with the one at the server side, we added a mechanism that associates an object of the client collection to one of the object of the server collection. When there is only one object at the client side, the stub generates a request for each object of the server collection. Similarly, when there is only one object in the server collection, one object of the client collection is associated with this single object. As data redistribution has been done by the stub, the skeleton performs roughly the same work as a standard skeleton. It is worth mentioning that in this situation, skeletons do not need to communicate between each other within the server collection. Another possibility is to perform data redistribution by the skeleton instead of the stub. In that case, the modification of the stub code generation is very simple. Each object of the client collection sends the data it owns to another object of the server collection. Before sending this data, the stub includes, for each parameter, data distribution information at the client side. It can then call a data redistribution library providing both the client and the server data distribution mode. Once redistribution is performed, the skeleton invokes the implementation method as a standard skeleton. If the parameter has an inout or out attribute, the skeleton builds the reply where it puts distributed data values following data distribution information sent by the client. It uses again the data redistribution library. This approach has the drawback of adding extra information (data distribution information) to the data sent by the client to the server. Moreover, such technique cannot be used when a sequential object has to invoke a method implemented by a parallel object. In such case, the client has to set up a request for each object of the server collection and thus has to distribute the data to each object of the collection.
Data Distribution for Parallel CORBA Objects
1245
To avoid implementing a new data redistribution library, we decided to adapt our stub and skeleton code generation process in such a way that we can exploit existing libraries. These libraries were developed for the High Performance Fortran (HPF) compilation system such as the NEC HPF/SC[7] and the GMD Adaptor system[2]. These two systems support all patterns of data redistribution in the scope of the HPF-1.1 specification. Since our extension for describing data distribution can be seen as a subset of HPF-1.1, all data redistribution patterns are covered. However, there are some limitations in our current implementation due to the difference of execution model between HPF and a parallel object. These data redistribution libraries are intended to be used within stand alone parallel program in which the number of processes is constant. However, we want to use these libraries to reorganize data when two parallel objects communicate. Such parallel programs may run on different number of processors and therefore we may have to redistribute data between two parallel objects that run on different number of processors. Moreover, such libraries were intended to be used with Fortran programs whereas we are using C++. Therefore, using these libraries requires extra memory copy operations to map C++ arrays to Fortran arrays.
5
Experimental Results
We performed several experiments using two platforms (Figure 4); namely, the NEC distributed-memory parallel computer Cenju-4[9] and a PC cluster. The Cenju-4 has 16 PEs (processing elements) connected via a multistage interconnection network as well as a 100 Mb/s Ethernet network. Each PE consists of
PC Cluster
PE15
Cenju-4
(a) Using a 100 Mb/s Ethernet network
PC1
.. .
1 Gigabit/s
PC5
PC Cluster
PE0 Switch
.. .
Host
PE1
.. .
PE15
Multistage interconnection Network
PC15
PE1
PC0 Switch
.. .
PE0 Switch
PC1
Host Switch
PC0
Multistage interconnection Network
100 Mb/s
Cenju-4
(b) Using a 1Gb/s Ethernet network
Fig. 4. Experimental environment.
a 200MHz VR10000 RISC microprocessor. The PC cluster is a set of PCs connected to a 100 Mb/s Ethernet network. Each PC is equipped with two 450MHz Pentium III processors running Linux. ORB communications between the PEs of the Cenju-4 and the PCs of the PC cluster can go either through the Cenju-4 host machine using a 100 Mb/s network or through a 1Gb/s network. For the
1246
Tsunehiko Kamachi, Thierry Priol, and Christophe Ren´e
experiment, we used the DALIB redistribution library [2]. The MPI communication layer was implemented on the multistage interconnection network on the Cenju-4 and on the fast Ethernet network on the PC cluster. In order to illustrate the effectiveness of our approach, we experimented a simple code; a parallel client issues a single invocation operation to a parallel server with a two dimensional distributed array parameter (long) with a in attribute. Distribution of the matrix at the client side follows a [BLOCK][*] distribution mode whereas at the server side the matrix to be distributed using a [*][BLOCK] distribution mode. 5.1
Comparison with the Master/Slave Approach
In this experiment, we ran both a parallel client and a parallel server on the PC cluster to compare the performance of the master/slave approach with that of our approach. Results presented in Figure 5 make it evident that our approach provides a scalable solution as compared with the master/slave approach. The other point we can observe is that a distinct difference between the performance of the client side redistribution and that of the server side redistribution cannot be seen. This result tells us the overhead of sending extra data in case of the server side redistribution is insignificant with our experimental environment. 5.2
Redistribution at the Client versus the Server
We measured the performance of a single operation invocation similar to what have be done to compare the master/slave approach with the parallel object one. However, this time we mapped the client onto the PC cluster and the server onto the Cenju-4. Data redistribution is performed either by the stub (using the MPI layer with the Ethernet network of the PC Cluster) or by the skeleton (using the MPI layer with the multistage network of the Cenju-4 parallel system). Results are presented in Figures 6. They were obtained by using either the 100 Mbit/s or the 1 Gbit/s Ethernet network. The time associated with the ORB communication, the memory copy and the data redistribution in the invocation are measured separately. ORB communication time corresponds to Matrix 1000x1000
Matrix 2000x2000
10000 Elapsed Time (ms)
Elapsed Time (ms)
2500 2000 1500 1000 500 0
8000 6000 4000 2000 0
1
2
3
4
5
6
7
8
1
2
3
Num ber of Objects Master/Slave approach Redistribution in the stub
Redistribution in the skeleton
4
5
6
7
Num ber of Objects Master/Slave approach
Redistribution in the skeleton
Redistribution in the stub
Fig. 5. Comparison our approach with the master/slave approach.
8
Data Distribution for Parallel CORBA Objects
Matrix 1000x1000 (Block distribution)
Matrix 2000x2000 (Block distribution)
14000
1500
1000
10000
500
Stub (PC Cluster)
2000
Redistribution Memory Copy ORB Communication
12000
Elapsed Time (ms)
Stub (PC Cluster)
2500
Skeleton (NEC Cenju-4)
Redistribution Memory Copy ORB Communication
8000 6000
Skeleton (NEC Cenju-4)
3000
Elapsed Time (ms)
1247
4000 2000
0
0 2
4
6
8
16
2
4
Number of Objects
6
8
16
Number of Objects
Matrix 1000x1000 (Block distribution)
Matrix 2000x2000 (Block distribution)
2500 Redistribution Memory Copy ORB Communication
Redistribution Memory Copy ORB Communication
10000
2000
500
Skeleton (NEC Cenju-4)
6000
Stub (PC Cluster)
1000
Elapsed Time (ms)
Skeleton (NEC Cenju-4)
Stub (PC Cluster)
Elapsed Time (ms)
8000 1500
4000
2000
0
0 2
4
6
Number of Objects
2
4
6
Number of Objects
Fig. 6. Comparison of communication costs (top: 100 Mbit/s, bottom: 1 Gbit/s).
the invocation time without redistribution and memory copy. We cannot see a significant difference in the two test cases (data redistribution within the stub or the skeleton). However, as compared with the results measured within the PC cluster, the ORB communication time between the Cenju-4 and the PC cluster is slow. In addition, when using the 100 Mbit/s Ethernet network, it does not provide a good speedup when number of objects increases. This is because, as shown in Figure 4, all the ORB communications between the PEs in the Cenju-4 and the PCs in the PC cluster have to go through the Cenju-4 host computer. If the PEs are connected to the network directly (using the 1 Gbit/s Ethernet network), the performance and the speedup ratio is improved. To handle both distributed data and the information about its distribution mode, we introduced a new data structure, called darray [12], based on the CORBA sequence data structure. Unlike an array, data in the darray is stored non-continuously into the memory if the darray realizes an array which has more than two dimensions in the same way as the sequence. Therefore, two memory
1248
Tsunehiko Kamachi, Thierry Priol, and Christophe Ren´e
copy operations are required in the redistribution process, that is, copying data from darray structure to a Fortran array before redistribution and copying redistributed data from the Fortran array to darray structure after redistribution. Moreover, since the difference of memory mapping scheme between C++ and Fortran arrays forces this memory copy by element, its overhead increases. The results clearly show that this memory copy causes serious overhead. In addition, we see from the figure that the parallel machines provide quite different results due mainly to the performance of the processors and the memory hierarchy that equippe each computing node. Consequently, as the Cenju-4 suffered from overhead of the memory copy, the PC cluster achieves better performance of the invocation in total, even if the Cenju-4 provides good performance for the data redistribution. Redistribution time is the communication time for exchanging data to perform the redistribution using the MPI interface. Experiments shows that the time of redistribution on the Cenju-4 is up to 9 times faster than that on the PC cluster. However, as compared with the overhead of the memory copy, this performance difference gives less impact to the total time of the invocation.
6
Conclusion and Future Works
This paper discusses the implementation of data redistribution within a parallel CORBA object. We implemented a capability of performing data redistribution within parallel object in both a stub and a skeleton. This allows programmers to obtain the maximum performance under their distributed computing environment. In our current implementation, the selection is performed at compile time by specifying Extended-IDL compiler options. This means that programmers are responsible for deciding which side should perform redistribution. Although it is important to provide means to control data redistribution to the programmers, it is usually difficult for them to know which side provides better performance. This is because they have to take into account a lot of factors related to characteristics of data redistribution and their underlying computing environment (communication network, memory hierarchy and processor). In addition, the fact that such factors can vary at run-time due to the network contention makes this problem much harder. In order to relieve programmers from the pains of making such decision, as well as to provide the maximum performance automatically, we are developing a run-time service system to manage static and dynamic system information during the execution of parallel objects. This information will be used do decide at runtime the best place to carry out data redistribution. Acknowledgments We would like to thank Satoshi Goto and Toshiyuki Nakata for their continuous valuable advice. This work was carrying out within the INRIA-NEC collaboration framework under contract 099C1850031308065.
Data Distribution for Parallel CORBA Objects
1249
References [1] T. Beisel, E. Gabriel, and M. Resch. An extension to MPI for distributed computing on MPPs. Lecture Notes in Computer Science, 1332, 1997. [2] T. Brandes and F. Zimmermann. Adaptor — A transformation tool for HPF programs. In Programming environments for massively parallel distributed systems: working conference of the IFIP WG10.3, pages 91–96, April 1994. [3] I. Foster and C. Kesselman. Globus: A metacomputing infrastructure toolkit. The International Journal of Supercomputer Applications and High Performance Computing, 11(2):115–128, Summer 1997. [4] I. Foster and C. Kesselman, editors. The Grid: Blueprint for a New Computing Infracstructure. Morgan Kaufmann Publishers, Inc, 1998. [5] Ian Foster, Jonathan Geisler, William Gropp, Nicholas Karonis, Ewing Lusk, George Thiruvathukal, and Steven Tuecke. Wide-area implementation of the Message Passing Interface. Parallel Computing, 24(12–13):1735–1749, November 1998. [6] A. S. Grimshaw, W. A. Wulf, and the Legion team. The Legion Vision of a Worldwide Virtual Computer. Communications of the ACM, 1(40):39–45, January 1997. [7] T. Kamachi, K. Kusano, K. Suehiro, Y. Seo, M. Tamura, and S. Sakon. Generating realignment-based communication for HPF programs. In 10th International Parallel Processing Symposium, 1996. [8] K. Keahey and D. Gannon. Developing and Evaluating Abstractions for Distributed Supercomputing. Cluster Computing, 1(1):69–79, May 1998. [9] T. Nakata, Y. Kanoh, K. Tatsukawa, S. Yanagida, N. Nishi, and H. Takayama. Architecture and software environment of parallel computer cenju-4. NEC Research & Development, 39(4):385–390, October 1998. [10] T. Priol and C. Ren´e. Cobra: A CORBA-compliant Programming Environment for High-Performance Computing. In Euro-Par’98, pages 1114–1122, September 1998. [11] A. Reinefeld, J. Gehring, and M. Brune. Communicating across parallel messagepassing environments. Journal of Systems Architecture, 44:261–272, 1998. [12] C. Ren´e and T. Priol. MPI code encapsulating using parallel CORBA object. In Proceedings of the Eighth IEEE International Symposium on High Performance Distributed Computing, pages 3–10, August 1999.
Topic 20 Parallel I/O and Storage Technology Rajeev Thakur, Rolf Hempel, Elizabeth Shriver, and Peter Brezany Topic Chairpersons
Introduction In recent years, it has become increasingly clear that the overall time to completion of parallel applications may depend to a large extent on the time taken to perform I/O in the program. This is because many parallel applications need to access large amounts of data, and although great advances have been made in the CPU and communication performance of parallel machines, similar advances have not been made in their I/O performance. The densities and capacities of disks have increased significantly, but improvement in performance of individual disks has not followed the same pace. For parallel computers to be truly usable for solving real, large-scale problems, the I/O performance must be scalable and balanced with respect to the CPU and communication performance of the system. The parallel I/O and storage research community is pursuing research in several different areas in order to solve the problem. Active areas of research include disk arrays, network-attached storage, parallel and distributed file systems, theory and algorithms, compiler and language support for I/O, runtime libraries, reliability and fault tolerance, large-scale scientific data management, database and multimedia I/O, realtime I/O, and tertiary storage. The MPI-IO interface, defined by the MPI Forum as part of the MPI-2 standard, aims to provide a standard, portable API that enables implementations to deliver high I/O performance to parallel applications. The Parallel I/O Archive at Dartmouth, http://www.cs.dartmouth.edu/pario is an excellent resource for further information on the subject. It has a comprehensive bibliography and links to various I/O projects.
Papers in This Track The Parallel I/O and Storage Technology track at Euro-Par 2000 contains six papers that address different aspects of the I/O problem: 1. “Towards a High-Performance and Robust Implementation of MPI-IO on top of GPFS,” by Jean-Pierre Prost, Richard Treumann, Richard Hedges, Alice Koniges, and Alison White describes IBM’s implementation of the MPI-IO standard for the GPFS file system. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1251–1252, 2000. c Springer-Verlag Berlin Heidelberg 2000
1252
Rajeev Thakur et al.
2. “Design and Evaluation of a Compiler-Directed Collective I/O Technique,” by Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary presents a compiler-directed collective I/O approach that detects opportunities for using collective I/O in a program and inserts the appropriate collective I/O calls. 3. “Effective File-I/O Bandwidth Benchmark,” by Rolf Rabenseifner and Alice E. Koniges describes a benchmark designed to measure the effective I/O bandwidth achievable by applications on a given parallel machine and file system. 4. “Instant Image: Transitive and Cyclical Snapshots in Distributed Storage Volumes,” by Prasenjit Sarkar presents an algorithm for handling snapshots of storage volumes in a distributed storage system. 5. “Scheduling Queries for Tape-Resident Data,” by Sachin More and Alok Choudhary investigates issues in optimizing I/O time for a query whose data resides on an automated tertiary storage system containing multiple storage devices. 6. “Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays,” by Y. Chen, W. Hsu, and H. Young presents a disk-array architecture that uses logging techniques to solve the small-write problem in parity-based disk arrays.
Towards a High-Performance Implementation of MPI-IO on Top of GPFS Jean-Pierre Prost1 , Richard Treumann2 , Richard Hedges3 , Alice Koniges3, and Alison White2 1
IBM T.J. Watson Research Center, Route 134,Yorktown Heights, NY 10598 IBM Enterprise Systems Group, 2455 South Road, Poughkeepsie, NY 12601 Lawrence Livermore National Laboratory, 7000 East Avenue, Livermore, CA 94550 2
3
Abstract. MPI-IO/GPFS is a prototype implementation of the I/O chapter of the Message Passing Interface (MPI) 2 standard. It uses the IBM General Parallel File System (GPFS) as the underlying file system. This paper describes the features of this prototype which support its high performance. The use of hints allows tailoring the use of the file system to the application needs.
1
Introduction
To provide users with a portable and efficient interface for parallel I/O, an I/O chapter was introduced in the Message Passing Interface 2 standard [3], based upon earlier collaborative work between researchers at the IBM T.J. Watson Research Center and the Nasa Ames National Laboratory [2]. Since approval of the MPI-2 standard, IBM has been working on both prototype and product implementations of MPI-IO for the IBM SP system, using the IBM General Parallel File System [4] as the underlying file system. This paper describes features of the prototype, referred to as MPI-IO/GPFS1 . The use of GPFS as the underlying file system offers the potential for maximum performance through a tight interaction between MPI-IO and GPFS. GPFS is a high performance file system which presents a global view of files to every client node. It provides parallel access to disk data managed by server nodes through the IBM Virtual Shared Disk interface. GPFS provides coherent caching at the client and uses optimized prefetching techniques. To avoid file block contention among tasks, MPI-IO uses data shipping. This technique binds each GPFS block to a single I/O agent, making it responsible for all accesses to this block. The GPFS blocks are bound to a set of I/O agents 1
IBM product development work draws on the knowledge gained in this prototype project but features of the prototype discussed in this paper and features of eventual IBM products may differ. Any performance data contained in this paper was obtained using prototype software. Therefore, the results obtained with product software may vary significantly. Measurements quoted in this presentation may have been made on development-level systems. There is no guarantee that these measurements will be the same on generally-available systems. Actual results may vary.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1253–1262, 2000. c Springer-Verlag Berlin Heidelberg 2000
1254
Jean-Pierre Prost et al.
with a round-robin striping scheme and MPI-IO/GPFS transfers data between the tasks where MPI-IO calls occur and the responsible agents as needed. MPIIO/GPFS allows the user to define the stripe size. The stripe size also controls the amount of buffer space each I/O agent uses in each data access operation. MPI-IO/GPFS is also a robust and user-friendly implementation. It prevents deadlocks when an error occurs only on a subset of the tasks participating in a collective I/O operation. In addition, errors which occur at the file system level can be traced on a per I/O agent basis through an optional error reporting feature. These robustness features are beyond the scope of this paper. The paper is organized as follows. Section 2 details how data shipping is implemented in MPI-IO/GPFS. Section 3 presents performance measurements to demonstrate the benefit of using data shipping appropriately. Section 4 briefly describes features that are being implemented in the prototype in order to achieve a tighter integration between MPI-IO and GPFS. Section 5 presents some conclusions and suggests possible future research directions for the MPIIO/GPFS prototype. All figures referenced in the text are gathered at the end of the paper.
2
MPI-IO/GPFS Features
The design foundation of MPI-IO/GPFS is the technique we call data shipping. To prevent conflicting access of GPFS file blocks by multiple tasks, normally residing on separate nodes, MPI-IO/GPFS binds each GPFS file block to a single I/O agent, which will be responsible for all accesses to the block. For an MPI-IO write call, the MPI task at which the call occurs ships to one or more I/O agents, a command with that agent’s write assignment and the data to be written. The agents then perform their assigned file writes. For an MPI-IO read call, the task ships commands with I/O agent read assignments. The agents read the file as instructed and ship the data to the commanding tasks. The binding scheme implemented by MPI-IO/GPFS consists in assigning the GPFS blocks to the set of I/O agents according to a round-robin striping, illustrated in Figure 1. I/O agents are multi-threaded and are also responsible for combining data access requests issued by all participating tasks in a collective MPI-IO operation. On a per file basis, the user can define the stripe size used in the allocation of GPFS blocks to I/O agents. The stripe size is the value of file hint IBM io buffer size, which can be specified when the file is opened, or when the MPI FILE SET INFO and MPI FILE SET VIEW functions are called. It is possible for a program to change the stripe size of an opened file as long as no I/O operation is pending on the file. The stripe size also controls the amount of buffer space used by each I/O agent in data access operations, justifying its name. The stripe size given by the user is rounded up by MPI-IO/GPFS to an integral number of GPFS blocks, and its default size is the number of bytes contained in 16 GPFS blocks. GPFS block size defaults to 256K unless set to some other value by a system administrator when the GPFS file system is configured.
Towards a High-Performance Implementation of MPI-IO on Top of GPFS
1255
Finally, on a per file basis, the user can enable or disable MPI-IO data shipping by setting file hint IBM largeblock io to “false” or “true”, respectively. MPI data shipping is enabled by default. When it is disabled, tasks issue read/write calls to GPFS directly. This saves the cost of MPI-IO data shipping but risks GPFS block ping-ponging among nodes if tasks located on distinct nodes contend for GPFS blocks in read-write or write-write sharing mode. In addition, collective data access operations are done independently. Therefore, we recommend to disable data shipping on a file only when accesses to the file are performed in large chunks or when tasks access disjoint large regions of the file. In such cases, MPI-IO coalescing of the I/O requests of a collective data access operation cannot provide benefit and GPFS block contention is not a concern. It is worth commenting on the difficulty of describing the purpose of a file hint in terms which are meaningful to the user. The relationship between MPI calls to do I/O and the activity at the file system level is complex and largely opaque to the MPI programmer. This makes the choice of a meaningful name and value format for a file hint somewhat challenging. The name should be mnemonic and the set or range of values the user selects from should have a recognizable relationship to the user’s understanding of her program’s I/O behavior. The user should not be expected to understand the internals of the MPI-IO implementation. In the case of data shipping, the user can use two file hints which are called IBM largeblock io and IBM io buffer size. These hints do not refer directly to MPI-IO/GPFS’s data shipping mode. Instead, they allow the user to express the general I/O pattern of her application on a per file basis and how much buffer space should be made available to MPI-IO/GPFS to process each data access operation.
3 3.1
Performance Measurements Benchmark Description
Let us first describe the two benchmarks used in our experimentation. Each is aimed at evaluating the benefit of the high performance features of MPIIO/GPFS for a particular class of application I/O patterns. For all tests, the metric is read or write bandwidth expressed as Mbyte/second for the job as a whole. The benchmarks run with any number of tasks and provide every task with the same amount of I/O to do. In our tests, the file size scales with the number of tasks. To provide a consistent approach to measurement, in each benchmark, MPI Barrier is called before beginning the timing, making each task’s first call to the MPI read or write routine well synchronized. In MPI semantics, the return from an MPI write operation does not guarantee that data has been committed to disk. To ensure that the entire write time is counted, each test does an MPI File sync and MPI Barrier before taking the end time stamp. In noncollective I/O it is possible that some tasks do I/O faster than others. Perhaps an argument could be made for averaging the times but we chose not to do that. The synchronizations ensure, in both collective and noncollective tests, we measure job time which we consider most meaningful.
1256
Jean-Pierre Prost et al.
Contiguous Benchmark In the contiguous benchmark, each task reads or writes sequentially a block of data from/to a single contiguous region of a preexisting file. Region size, type of file access (read or write), and type of I/O operation (collective or noncollective) are parameters to the benchmark program. The number of tasks in the parallel job together with the region size determine the size of the file that is accessed. Discontiguous Benchmark In the discontiguous benchmark, each task reads or writes an equal number of 1024-byte blocks, where the set of blocks that any one task reads or writes is scattered randomly across the entire file. Working together, the tasks tile the entire file without gaps or overlap. Parameters control file size, type of file access (read or write), type of I/O operation (collective or noncollective). The benchmark program reads or writes the file region by region, using a two gigabyte region size. The region size is chosen as the maximum that could be mapped with an MPI Datatype. At program start, each task is assigned a list of 1024-byte blocks to read or write to the file. At each step, the task creates an MPI Datatype to map the blocks that belong to the current region into their proper file locations. This is accomplished by passing the MPI Datatype that maps the blocks of the current region as a new filetype argument to MPI File set view at each step. 3.2
Experimental Platform
All measurements were made on Lawrence Livermore National Laboratory ASCI Blue Pacific systems. These IBM RS6000/SP systems are based on 4-way nodes utilizing 332 Mhz 604e Power2 processors. Each of the nodes has 1.5 Gb of memory. The particular system hosting the experimentation includes 425 compute nodes and 56 I/O nodes. The GPFS configuration includes 38 VSD servers. GPFS page pools occupy 50 MB on each compute node. The software configuration includes AIX 4.3.2+, PSSP 3.1, and GPFS 1.2. A more detailed description of the file system configuration and GPFS performance statistics can be found elsewhere [5]. While it is preferable to make performance measurements on a dedicated system, the ASCI Blue systems are so heavily used that this was not possible. Thus, these measurements were carried out on systems in normal production operation. Despite this normal workload, efforts were made to make the measurements at times when other jobs were not contending for the GPFS file system. The peak performance on the contiguous benchmark with data shipping could be somewhat higher than our numbers, since it is affected by switch traffic and we are not in a dedicated environment. The peak performance we report for no data shipping is unlikely to change in a dedicated environment because our results are very reproducible. In order to obtain an estimate of the peak performance of the file system, we ran each of our experiments several times and then report the highest value, plotted in graphs collected at the end of the paper. We also use
Towards a High-Performance Implementation of MPI-IO on Top of GPFS
1257
the data obtained from several replications of the same runs in order to evaluate the impact of contention on the measurement results. The contiguous and discontiguous benchmarks were run on 4, 8, 16, 32, 64, and 128 MPI tasks. Each MPI task had a node of the machine dedicated to it. Measurements for MPI-IO/GPFS are compared to the same tests using the ROMIO (MPICH) implementation [6,7] and to analogous tests utilizing a traditional POSIX I/O interface. By analogous, we mean that the POSIX tests use the same data layouts (across task memory and in the file) as the MPI-IO tests. For all of the MPI-IO/GPFS measurements and for the POSIX version of the contiguous benchmark, I/O bandwidth was measured for seven replications of the same experiment. For the POSIX version of the discontiguous benchmark, the maximum achieved I/O bandwidths (< 10 MB/sec) ruled out multiple measurements. In this case, a single run of each experiment was performed. The maximum measured aggregated I/O bandwidth is reported for each combination of benchmark, read or write operation, MPI-IO implementation, and number of tasks. 3.3
Benchmark Results
The performance data is presented in graphs at the end of the paper, to provide a comparison of the aggregated bandwidths obtained with and without use of MPI-IO data shipping. Contiguous Benchmark Results For the contiguous benchmark, described in Section 3.1, a region size of 333 MB was used. Each MPI task wrote a single region. In this situation, noncollective operations give the best performance. The data transfers are independent and can proceed in parallel with no coordination. To illustrate the effect of using MPI-IO data shipping (controlled by the IBM largeblock io file hint), Figure 2 compares the results of running the contiguous benchmark with MPI-IO data shipping enabled (IBM largeblock io set to “false”) to the same runs with MPI-IO data shipping disabled (IBM largeblock io set to “true”). For comparison, we also include results obtained with ROMIO and POSIX. With data shipping disabled, the performance of MPI-IO/GPFS is strikingly similar to the ROMIO and POSIX versions of the benchmark. These three versions exhibit a saturation of GPFS’s ability to process the data with saturation occurring between 32 and 64 MPI tasks. For the contiguous benchmark with large file regions, there is no contention of tasks writing to the same GPFS block. Therefore, the setup logic and data movement involved in data shipping is purely overhead. For task numbers below 64, performance is less with data shipping enabled. The overhead of data shipping has an interesting effect for larger task numbers (64 and 128). We surmise that the overhead reduces the rate of queueing requests to GPFS providing a sort of flow control which leads to better performance with these large task numbers. In Figure 3, results of the contiguous benchmark for read operations are displayed. Again with data shipping disabled, MPI-IO/GPFS performs like the
1258
Jean-Pierre Prost et al.
POSIX or ROMIO versions of the benchmark. In the case of reads, the data layout is such that no data shipping is needed for optimal performance. Flow control (or lack of it) is not an issue for the read tests. Enabling data shipping introduces additional overhead, and the read performance with data shipping enabled is consistently lower than with the other three versions of the benchmark. The range of measured write bandwidths for the contiguous benchmark was observed to depend on the status of data shipping (controlled by the value of the IBM largeblock io hint). This variability of measured performance is particularly evident for larger numbers of tasks. In Figure 4, we produce the standard deviation, σ, of the aggregated write bandwidth for the two cases, nΣx2 − (Σx)2 σ= n(n − 1) where x is a single bandwidth measurement and n is the number of measurements. (We removed one data point from our sample that had a very low value potentially due to system problems.) The standard deviation with data shipping enabled exceeds the standard deviation with data shipping disabled for experiments with 32 or greater number of tasks. For 128 tasks, the standard deviation with data shipping is 35 times larger. The data is for MPI-IO/GPFS only, since the results for ROMIO and POSIX are very similar to those obtained with MPI-IO/GPFS and data shipping disabled. A possible explanation for the dependence on data shipping, of the variability (expressed in terms of the standard deviation) of the measured bandwidth is the following. When data shipping is enabled, all data must be sent from the tasks to the I/O agents, and write calls are issued by the I/O agents. We notice an increasing variability in the bandwidth with an increasing number of tasks since there is more contention for switch access and because all tasks are trying to send data concurrently to each I/O agent. When data shipping is disabled, i.e., the IBM largeblock io is set to “true”, there is no message passing occurring between the MPI tasks and the I/O agents during data access operations. Here, tasks are issuing I/O calls to GPFS directly. In this latter case, the variability in the measurements goes down to an almost insignificant level at larger numbers of tasks. The turnover in the standard deviation curve without data shipping occurs when the write rates begin to saturate because of the lack of flow control in GPFS 1.2. In summary, users on a production machine such as this one may see a large variability in their write performance for large numbers of tasks with data shipping, and significantly reduced variability in their write performance without data shipping. Discontiguous Benchmark Results The discontiguous benchmark, described in Section 3.1, is most efficient with use of the optimized collective operations and data shipping enabled. The file size in this benchmark amounts to 333 MBytes per MPI task. Again, the POSIX and ROMIO versions of this benchmark are compared to MPI-IO/GPFS.
Towards a High-Performance Implementation of MPI-IO on Top of GPFS
1259
In Figure 5, results of the write performance tests for the discontiguous benchmark are presented. Initially, the most striking aspect of the plot is the dismal performance when using the straight POSIX interface. It is however not so surprising that many tasks writing 1KB blocks to a GPFS file system leads to very poor performance. Experimental results show that MPI-IO/GPFS with data shipping disabled leads to qualitatively similar performance. In Figure 6, results of the read performance for the discontiguous benchmark are presented. Again, the performance of the POSIX version of the test shows poor performance as we would expect. MPI-IO/GPFS and ROMIO show good performance, achieving over 375 MB/sec and 250 MB/sec respectively. ROMIO shows substantial optimization of the I/O in comparison to the POSIX results. The performance of ROMIO shows qualitatively different behaviors for the write and the read tests. Where performance of the read test is scaling well, performance of the write test peaks in the neighborhood of 32 tasks and then declines as the number of tasks increases. We suspect that the additional load to GPFS of the read - modify - write cycle for data sieving in ROMIO has led to the same sort of flow control issues noted in the write performance of the contiguous test [6]. MPI-IO/GPFS shows the best write performance in the discontiguous benchmark reaching over 300 MB/sec and the best performance in the read test, reaching over 375 MB/sec. This demonstrates the benefit of using data shipping combined with collective operations, leading to good scalability for both read and write operations.
4
Work in Progress
We are currently experimenting with a tighter integration of MPI-IO with GPFS, through the use of prototyped GPFS directives and hints. A new set of GPFS directives allow to set what is referred to as file partitioning mode at file open. When this mode is set, the file is partitioned into a number of large pieces. Each piece is accessed under the responsibility of a single node. In this mode, there is never any GPFS block shared between nodes so a single shared lock on the file is issued to each node. This eliminates any need to manage lock conflicts and greatly reduces the complexity and size of the state the GPFS lock manager maintains. MPI-IO/GPFS’s data shipping feature is a natural match to GPFS file partitioning mode and can take advantage of the performance gain associated with it. A new GPFS hint is being prototyped. It allows to prefetch selectively ranges of data contained in GPFS file blocks. MPI-IO/GPFS can use this hint at each I/O agent once it knows the GPFS blocks to be accessed for the current data access operation. If these blocks do not correspond to a sequential or strided sequential patterns, for which GPFS prefetching policy has been optimized, use of this hint has the potential to improve noticeably the I/O performance of the
1260
Jean-Pierre Prost et al.
MPI application. Because this hint is a part of the GPFS interface and is used by MPI-IO/GPFS, its exploitation is transparent to the MPI program making MPI-IO calls.
5
Conclusion
This paper illustrates how careful use of file hints can allow users to improve the performance of their MPI-IO applications on top of the IBM General Parallel File System. Data shipping is a clear winner when data is read-write or writewrite shared among several tasks. Even though not illustrated in this paper, we also observed that increasing the stripe size and the I/O agent buffer size leads to better performance, provided the application can spare the buffer space. On the other hand, for applications in which tasks access either large pieces of data at once or disjoint regions of the file, enabling data shipping may reduce the performance. We are currently investigating whether double buffering at the I/O agent can lead to increased performance. We are also examining whether data sieving, already used in ROMIO [6], could be beneficial to our prototype implementation. We are planning to study whether feedback from GPFS about the block hit ratio induced by its buffer cache replacement and prefetching policies or about the I/O request service time at the server side can be exploited by MPI-IO so that it can adapt its own prefetching policy or help GPFS control the I/O request flow to an overloaded server. We also started to define synthetic benchmarks, which allow us to control the level of overlap between I/O and computation by using several user threads and adjusting the number of threads performing data access operations versus the number of computing threads. Through these synthetic benchmarks, we will also evaluate the impact of thread scheduling policies on the application performance.
6
Acknowledgements
All prototype and benchmark development was done by IBM personnel as part of ongoing research and development. Measurements and observations in the sections for Contiguous Benchmark Results and Discontiguous Benchmark Results have been provided by Lawrence Livermore National Laboratory. This portion of the work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract no. W-7405-Eng-48.
Towards a High-Performance Implementation of MPI-IO on Top of GPFS
1261
MPI
MPI
Main Thread
Main Thread
Task
MPI Task
MPI
MPI
Responsible for accesses to GPFS blocks #
I/O Agent
I/O Agent
0-15, 32-47, 64-79, ...
I/O buffer
I/O buffer
read()/write()
Responsible for accesses to GPFS blocks # 16-31, 48-63, 80-95, ...
read()/write()
GPFS
GPFS
Standard Deviation of Aggregated Bandwidth
250
200
150
50
0
Fig. 1. GPFS block allocation used by MPI-IO/GPFS for a two task MPI job, in data shipping mode (using the default stripe size)
MPI-IO/GPFS with data shipping MPI-IO/GPFS without data shipping
100
4 8 16
32
64 Number of Tasks
128
Fig. 4. Effect of data shipping on measured standard deviation for the contiguous write benchmark 350
300 Aggregated Bandwidth (MB/s)
Aggregated Bandwidth (MB/s)
1400 1200 1000 800 600
MPI-IO/GPFS with data shipping MPI-IO/GPFS without data shipping POSIX ROMIO
400
200
250
200
100
50
0 0
4 8 16
32
64 Number of Tasks
32
64 Number of Tasks
128
Fig. 5. Maximum bandwidths measured for write operations in the discontiguous benchmark
1800
400
1600
350
1400
Aggregated Bandwidth (MB/s)
Aggregated Bandwidth (MB/s)
4 8 16
128
Fig. 2. Maximum bandwidths measured for write operations in the contiguous benchmark
1200
1000
800
600
MPI-IO/GPFS with data shipping MPI-IO/GPFS without data shipping POSIX ROMIO
400
200
0
MPI-IO/GPFS POSIX ROMIO
150
4 8 16
32
64 Number of Tasks
300 250
200
150
MPI-IO/GPFS POSIX ROMIO
100
50 128
Fig. 3. Maximum bandwidths measured for read operations in the contiguous benchmark
0
4 8 16
32
64 Number of Tasks
128
Fig. 6. Maximum bandwidths measured for read operations in the discontiguous benchmark
1262
Jean-Pierre Prost et al.
References 1. www.llnl.gov/asci/. 2. P. Corbett, D. Feitelson, S. Fineberg, Y. Hsu, B. Nitzberg, J.-P. Prost, M. Snir, B. Traversat, and P. Wong, Chapter in Input/Output in Parallel and Distributed Computer Systems, Ravi Jain, John Werth, and James C. Browne, Eds., Kluwer Academic Publishers, June 1996, pp. 127–146. 3. Message Passing Interface Forum, Mpi-2: A message-passing interface standard, Standards Document 2.0, University of Tennessee, Knoxville, July 1997. www.mpi-forum.org/docs/docs.html. 4. IBM General Parallel File System for AIX: Installation and Administration Guide, IBM Document SA22-7278-02, October 1998. www.rs6000.ibm.com/resource/aix resource/sp books/gpfs/install admin/install admin v1r2/gpfs1mst.html. 5. T. Jones, A. Koniges, and K. Yates, Performance of the IBM General Parallel File System, Proc. International Parallel and Distributed Processing Symposium, May 2000. Accepted. 6. R. Thakur, W. Gropp, and E. Lusk, Data Sieving and Collective I/O in ROMIO, Proc. 7th Symposium on the Frontiers of Massively Parallel Computation, February 1999, pp. 182–189. 7. R. Thakur, W. Gropp, and E. Lusk, On Implementing MPI-IO Portably and with High Performance, Proc. of the Sixth Workshop on I/O in Parallel and Distributed Systems, May 1999, pp. 23–32.
Design and Evaluation of a Compiler-Directed Collective I/O Technique Gokhan Memik1 , Mahmut T. Kandemir2 , and Alok Choudhary1 1
Department of Electrical and Computer Eng. Northwestern University, Evanston IL 60208, USA 2 Dept. of Computer Science and Engineering, Pennsylvania State University, University Park PA 16802, USA
Abstract. Current approaches to parallel I/O demand extensive user effort to obtain acceptable performance. This is in part due to difficulties in understanding the characteristics of a wide variety of I/O devices and in part due to inherent complexity of I/O software. While parallel I/O systems provide users with environments where large datasets can be shared between parallel processors, the ultimate performance of I/O-intensive codes depends largely on the relation between data access patterns and storage patterns of data in files and on disks. Collective I/O is one of the most popular methods to access the data when the storage and access patterns do not match. In this strategy, each processor does I/O on behalf of other processors if doing so improves the overall performance. While it is generally accepted that collective I/O and its variants can bring impressive improvements as far as the I/O performance is concerned, it is difficult for the programmer to use collective I/O effectively. In this paper, we propose and evaluate a compiler-directed collective I/O approach which detects the opportunities for collective I/O and inserts the necessary I/O calls in the code automatically. An important characteristic of the approach is that instead of applying collective I/O indiscriminately, it uses collective I/O selectively, only in cases where independent parallel I/O would not be possible. We have conducted several experiments using an IBM SP-2 distributed-memory message-passing machine with 128 nodes. Our compiler directed collective I/O scheme was able to perform 18% better in average than an indiscriminate collective I/O scheme in our base configuration.
1
Introduction
Todays’ parallel architectures comprise fast microprocessors, powerful network interfaces, and storage hierarchies that typically have multi-level caches, local and remote main memories, and secondary and tertiary storage devices. In going from upper levels of a storage hierarchy to lower levels, average access times
This work is supported by the Department of Energy’s Accelerated Strategic Computing Initiative (ASCI) program under a subcontract No W-7405-ENG-48 from Lawarence Livermore National Laboratories and by NSF CDA-9703228 and NSF ACI-9707074.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1263–1272, 2000. c Springer-Verlag Berlin Heidelberg 2000
1264
Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary
increase dramatically. Because of their cost effectiveness, magnetic disks have dominated the secondary storage market for the last several decades. Unfortunately, their access times have not kept pace with performance of the processors used in parallel architectures. Consequently, a large performance gap between secondary storage access times and processing unit speeds has emerged. To address this imbalance, hardware designers focus on improving parallel I/O capabilities using multiple disks, I/O processors, and large bandwidth I/O busses [6]. An optimized I/O software can also play a major role in bridging this performance gap. In order to eliminate the difficulty in using a parallel file system directly, several research groups proposed high-level parallel I/O libraries and runtime systems that allow programmers to express access patterns of their codes using program-level data structures such as rectilinear array regions [4,13,5]. While all these software supports provide an invaluable help to boost the I/O performance in parallel architectures, it remains still programmer’s responsibility to select appropriate I/O calls to use, to insert these calls in appropriate locations within the code, and to manage the data flow between parallel processors and parallel disks. One of the most important optimizations in MPI-IO [5] is collective I/O, an optimization that allows each processor to do I/O on behalf of other processors [4]. This optimization has many variants [12,11,13]; the one used in this study is two-phase I/O. In this implementation, I/O is performed in two phases: an I/O phase and a communication phase. In the I/O phase, processors perform I/O in a way that is most beneficial from the storage layout point of view. In the second phase, they engage in a many-to-many type of communication to ensure that each piece of data arrives in its final destination. While collective I/O and its variants are very beneficial if used properly, almost all previous studies considered a useroriented approach in applying collective I/O. For example, Thakur et al. suggest programmers to use collective I/O interfaces of MPI-IO instead of easy-to-use Unix-like interfaces [14]. Apart from determining the most suitable collective I/O routine and its corresponding parameters, this also requires, on the user part, analyzing access patterns of the code, detecting parallel I/O opportunities, and finally deciding a parallel I/O strategy. In this paper, we propose and evaluate a compiler-directed collective I/O strategy whereby an optimizing compiler and MPI-IO cooperate to improve I/O performance of scientific codes. The compiler’s responsibility in this work is to analyze the data access patterns of individual applications and determine suitable file storage patterns and I/O strategies. Our approach is selective because it activates collective I/O selectively, only when necessary. In other cases, it ensures that processors perform independent parallel I/O, which has almost the same I/O performance as collective I/O but without extra communication overhead. The remainder of this paper is organized as follows. In the next section, we review collective I/O. In Section 3, we explain our compiler analyses to detect access patterns and suitable storage patterns for multidimensional datasets considering multiple, related applications together. In Section 4, we describe
Design and Evaluation of a Compiler-Directed Collective I/O Technique
1265
our experimental framework, our benchmarks, and different code versions, and present our experimental results. In Section 5, we present our conclusions.
2
Collective I/O
In many I/O-intensive applications that access large, multidimensional, diskresident datasets, the performance of I/O accesses depends largely on the layout of data in files (storage pattern) and distribution of data across processors (access pattern). In cases where these patterns are the same, potentially, each processor can perform independent parallel I/O. However, the term ‘independent parallel I/O’ might be misleading, as, depending on the I/O network bandwidth, the number of parallel disks available, and the data striping strategies employed by the parallel file system, two processors may experience a conflict in accessing different data pieces residing on the same disk [6]. What we mean by ‘independent parallel I/O’ instead is that the processors can read/write their portions of the dataset (dictated by the access pattern) using only a few I/O requests in the code, each for a large number consecutive data items in a file. These independent source-level I/O calls to files are broken up into several system-level calls to parallel disks. This last aspect, however, is architecture and operating system dependent and is not investigated in this paper. Note that, in independent parallel I/O, there is no interprocessor communication or synchronization during I/O. In cases where storage and access patterns do not match, allowing each processor to perform independent I/O will cause processors to issue many I/O requests, each for a small amount of consecutive data. In this paper, an access pattern which is the same as the corresponding storage pattern is called a conforming access pattern. Collective I/O can improve the performance in nonconforming cases by first reading the dataset in question in a conforming (storage layout friendly) manner and then redistributing the data among the processors to obtain the target access pattern. Of course, in this case, the total data access cost should be computed as the sum of I/O cost and communication cost. The idea is that the communication cost is typically small as compared to I/O cost, meaning that the cost of accessing a dataset becomes almost independent from its storage pattern. Consider Figure 1 that shows both independent parallel I/O and collective I/O for a four processor case using a single disk-resident two-dimensional dataset. In Figure 1(a), the storage pattern is row-major (each circle represents an array element and the arrows denote the linearized file layout of elements) and the access pattern is row-wise (i.e., each of the four processors accesses two full-rows of the dataset). Since the access pattern and the storage pattern match, each processor can perform independent parallel I/O without any need of communication or synchronization. Figure 1(b), on the other hand, shows the case where collective I/O is required. The reason is that in this figure the storage pattern is row-major and the access pattern does not match it. As explained earlier, the I/O is performed in two phases. In the first phase, each processor accesses the
1266
Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary
data row-wise, as if this was the original access pattern), and in the second step, an all-to-all communication is performed between the processors and each data item is delivered to its final destination. Access Pattern
(a)
Access Pattern
Storage Pattern
(b)
Storage Pattern
Independent I/O
Interprocessor Communication
Collective I/O
Fig. 1. (a) Independent parallel I/O and (b) Collective (two-phase) I/O.
6
3
Visualization/Archiving Application
4
0
5
1
2
Simulation Applications Mesh Generation & Domain Decomposition Applications
Fig. 2. Scientific working environment.
3
Compiler Analysis
Our approach to collective I/O utilizes a directed graph called weighted communication graph (WCG). Each node of a weighted communication graph is a code block, which can be defined as a program fragment during which we can keep the datasets in memory; however, between executions of code blocks, the datasets should be stored on disks. Depending on the applications, the datasets in question, and the available memory, a code block can be as small as a loop nest or can be as large as a full-scale application. An example for the latter is shown in Figure 2 that depicts a typical scenario from a scientific working environment. There is a directed edge, e1,2 , between two nodes, cd1 and cd2 , of the WCG if and only if there exists at least a dataset that is produced (i.e., created and stored on disk) in cd1 and used (i.e., read from disk) in cd2 . In such a case cd1 is called producer and cd2 is called consumer. The weight associated with e1,2 (written as w1,2 ) corresponds to total number of dynamic control-flow transitions between code blocks cd1 and cd2 (e.g., how many times cd2 is run after cd1 in a typical setup). Depending on the granularity of code blocks, these weights
Design and Evaluation of a Compiler-Directed Collective I/O Technique
1267
can be calculated using profiling with typical input sets, can be approximated using weight estimation techniques, or can be entered by a user who observed the scientific working environment for a sufficiently long period of time. 3.1
Access Pattern Detection
Access patterns exhibited by each code block can be determined by considering individual loop nests that make up the code block. The crucial step in this process is taking into account the parallelization information [2]. Individual nests can either be parallelized explicitly by programmers using compiler directives [1,3], or can be parallelized automatically (without user intervention) as a result of intra-procedural and inter-procedural compiler analyses [2,8,9]. In either case, after the parallelization step, our approach determines the data regions (for a given dataset) accessed by each processor involved. For each array reference in each loop nest, our compiler determines an access pattern. Afterwards, it utilizes a conflict resolution scheme to resolve intra-nest and inter-nest access pattern conflicts. To achieve a reasonable conflict resolution, we associate a count with each reference indicating (or approximating) the number of times that this reference is touched in a typical execution. In addition to that, for each access pattern that exists in the code block, we associate a counter that is initialized to zero and incremented by a count amount each time we encounter a reference with that access pattern. In this way, for a given array, we determine the most preferable (or most prevalent) access pattern (also called representative access pattern) and mark the code block with that information. Although, at first glance, it seems that, in a typical large-scale application, there will be a lot of conflicting patterns that would make compiler’s job of favoring one of them over the others difficult, in reality, most scientific codes have a few preferable access patterns. 3.2
Storage Pattern Detection
Having determined an access pattern for each disk-resident dataset, the next step is to select a suitable storage pattern for each dataset in its producer code block. We have built a prototype tool to achieve this.1 For a given dataset, the tool takes the representative access patterns detected in the previous step by the compiler for each code block and runs a storage layout detection algorithm. Without loss of generality, in the following discussion, we focus only on a single dataset. The first step in our approach is to determine producer-consumer subgraphs (PCSs) of WCG for the dataset in question. A PCS for a dataset consists of a producer node and a number of consumer nodes that use the data produced by this producer. In the second step, we associate a count 1
Note that building a separate tool is necessary only if the granularity of code blocks is a full-application. If, on the other hand, the granularity is a single nested-loop or a procedure, the functionality of this tool can be embedded within the compiler framework itself.
1268
Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary
with each possible access pattern and initialize it to zero. Then, we traverse all the consumer nodes in turn and for each consumer node add its weight to the count of its access pattern. At the end of this step, for each access pattern, we obtain a count value. In the third step, we set the storage pattern in the producer node to the access pattern with the highest count. Note that, for a given dataset, we need to run the storage pattern detection algorithm multiple times, one for each producer node for this dataset. The next step is to determine suitable I/O strategies for each consumer node. Let us again focus on a specific dataset. If the access pattern (for this dataset) of a consumer node is the same as the storage pattern in the producer node, we perform independent parallel I/O in this consumer node. Otherwise, that is, if the access and storage patterns are different, we perform collective I/O. We perform this step for each dataset and each PCS. Once the suitable I/O strategies have been determined, the compiler automatically inserts corresponding MPI-IO calls in each code block. 3.3
Discussion
Although, our approach is so far mainly discussed for a setting where individual code blocks correspond to individual applications, it is relatively easy to adapt it to different settings as well. If we consider each code block as a procedure in a given application, then a WCG can be processed using algorithms similar to those utilized in processing weighted call graphs [7], where each node represents a procedure and an edge between two nodes correspond to dynamic control-flow transitions (e.g., procedure calls and returns) between the procedures represented by these two nodes. In an out-of-core environment, on the other hand, each node may represent an individual loop nest and edges might represent dynamic control-flow between nests; in this case, the WCG is similar to a control-flow graph. Another important issue that needs to be addressed is what to do (inside a code block) when we come across a reference whose access pattern is not the same as the representative (prevalent) access pattern for this code block. Recall that we assumed that, within a code block, we should be able to keep the datasets in memory. When the access pattern of an individual reference is different from the representative access pattern determined for a code block, we can re-shuffle the data in memory. This is not a correctness issue for shared-memory machines but it may cause performance degradation. In distributed-memory message-passing architectures, on the other hand, this data re-shuffling in memory is necessary to ensure the correct data accesses in subsequent computations.
4
Experiments
In this section, we first describe the experimental environment. Afterwards, we explain the setups for the experiments. Then, the results for the base configuration is given. Finally, we give results for different number of processors and data sizes.
Design and Evaluation of a Compiler-Directed Collective I/O Technique
cb1
cb2
cb3
cb5
cb4
cb7
cb6
cb0
cb7
cb7
cb6
cb6
cb5
cb5
1269
Setup 1
cb1
cb2
cb3
cb4
cb5
cb7
cb6
cb4
cb4
cb3
cb3
cb2
cb2
cb1
cb1
cb0
Setup 2
cb7
cb6
cb5
cb4
cb3
cb2
cb1
cb0
cb0
Setup 7
Setup 8
cb0
Setup 3 cb7
cb4
cb5
cb6
cb5
cb6
cb7
cb6
cb1
cb2
cb3
cb2
cb3
cb4
cb1
cb0
Setup 4
cb0
cb1
Setup 5
cb7
cb2
cb3
cb4
cb5
cb0
Setup 6
Fig. 3. Setups (communication graphs) used in the experiments. 4.1
Experimental Environment
We used the MPI-2 library and an IBM SP-2 in Argonne National Laboratories to evaluate our scheme proposed in this paper. The IBM SP-2 used in the experiments has 128 processors, 8 of which are I/O processors. The compute nodes are RS/6000 Model 370 processors with 128 MB memory, whereas the I/O nodes are RS/6000 Model 970 processors with 256 MB memory. The nodes are connected via 100 Mbs Ethernet, 155 Mbs ATM and 800 Mbs HiPPI networks. Each I/O server has a 9 GB of storage space resulting in 72 GB of total disk space. The operating system on each node is AIX 4.2.1. PIOFS provides the parallel access to files. It distributes a file across multiple I/O server nodes. 4.2
Setups
To evaluate the possible improvements with our scheme, we have designed 8 different setups (communication graphs), each built up using 8 different benchmark codes in different ways. We have selected 4 benchmarks from Specfp (tomcatv (cb0 ), vpenta (cb1 ), btrix (cb2 ), and mxm (cb3 )), 2 codes from the Perfect Club benchmark suite (tis (cb4 ) and eflux (cb5 )), 1 from Nwchem suite (transpose (cb6 )) and a miscellaneous code (cholesky (cb7 )). The setups built from these benchmarks are given in Figure 3.2
2
Although Setup 1 and Setup 2 look the same, they differ in the access and storage patterns they employ for different benchmarks. Similarly, Setup 7 and Setup 8 differ in their access and storage patterns. The details are omitted due to lack of space.
1270
Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary Na ve Strategy
Collective I/O
Compiler-directed Collective I/O 100
Execution Time [sec]
Execution Time [sec]
150 125 100 75 50 25
75 50 25 0
0 S1
S2
S3
S4
S5
S6
S7
S8
S1
A vg .
S2
S3
S4
150 125 100 75 50 25 0 S2
S3
S4
S5
S6
S7
S8
A vg .
Figure 7: Execution times for 4 processors Execution Time [sec]
Execution Time [sec]
Figure 6: Execution times for the base experiments 225 200 175
S1
S5 S etu p s
S etu p s
S6
S7
S8
S etu p s
Figure 8: Execution times for 16 processors
A vg .
300 275 250 225 200 175 150 125 100 75 50 25 0 S1
S2
S3
S4
S5
S6
S7
S8
A vg .
S etu p s
Figure 9: Execution times for larger data size (Data size is doubled)
Three bars for each setup represents the following access strategies, from left to right: naïve strategy (no collective I/O), indiscriminate collective I/O, compiler directed collective I/O
4.3
Base Experiments
For the base experiments, we have executed the setup explained in Section 4.2 using 8 processors of the IBM SP-2. In our experiments we are comparing the execution times of three different code versions as explained below. Version 1 In this version, each processor performs independent (non-collective) I/O regardless whether the access pattern and the storage pattern are the same or not. We call this version the naive I/O strategy. Version 2 This version performs indiscriminate collective I/O. Version 3 This is the strategy explained in this paper. The collective I/O is performed selectively, only if the access pattern and the storage pattern do not match. In all other cases, we perform independent parallel I/O. In Figures 6 through 9, the left-most bar represents the total execution time of version 1, the middle bar represents the total execution time of version 2 and the right-most bar represents the total execution time of version 3. The results for 8 processors are given in Figure 6. The average improvement over the indiscriminate collective I/O strategy is 18.01%. For setups 5 and 6, we are able to gain more than 21% over the version 2, which performs indiscriminate collective I/O. These two setups give the best results, because a change of the storage pattern effects the most applications. For example, when cb0 in Setup 1 changes its storage pattern, the weights of the favoring applications add up to 50% of the sum of all weights, whereas in Setup 6, the sum of weights of the favoring applications constitutes 60% of all the weights. So, there are 10% more favoring applications in Setup 6. Therefore, the improvement of Setup 6 is more than Setup 1 with our scheme. Note that, both the indiscriminate and selective collective I/O strategies perform well compared to a naive strategy, which does not use collective I/O at
Design and Evaluation of a Compiler-Directed Collective I/O Technique
1271
all. The improvement of indiscriminate collective I/O is 74.19% over the naive strategy, whereas our scheme brings 78.84% improvement. 4.4
Sensitivity Analysis
We have performed a second set of experiments to see how our strategy is affected by the number of processors and the data size. Figures 7 and 8 give the results for 4 and 16 processors, respectively. For 4 processors, our scheme brings 16.42% improvement over the indiscriminate collective I/O version, and 40.23% over the naive strategy. For 16 processors, on the other hand, our scheme brings an 26.27% improvement over the indiscriminate collective I/O version, and 80.74% over the naive strategy. As the number of processors increase, the improvement of our scheme increases, because with larger number of processors, the synchronization and communication costs increase. Figure 9 gives the results for a larger data size. For this experiment, we have doubled the size of the input and/or output data of all the benchmarks. When the data size is increased, the synchronization overhead is reduced. Similarly, communication and I/O can be better overlapped because the I/O calls are longer, so the overall communication cost also reduces. These factors decrease the percentage improvements of our scheme, but, it still performs the best by far. It brings a 13.57% improvement over the indiscriminate collective I/O version, and 75.21% improvement over the naive strategy.
5
Conclusions
In this paper, we present and evaluate a compiler-directed collective I/O strategy. Collective I/O plays a major role in parallel I/O systems. Therefore, increasing its performance is very important for many of data-intensive parallel applications. By adopting a selective collective I/O strategy, we are able to bring significant amounts of improvements. In average, our scheme performs 18.01% better than an indiscriminate collective I/O strategy in our base configuration. The scheme performs better as the number of processors increases. Although the improvement decreases with the increased data size, the scheme is still able to perform more than 13% better than an indiscriminate collective I/O strategy. The interface for parallel I/O systems are usually complex, and they are getting more complex, because the information required by the I/O calls increases. So, it becomes harder for an average user to detect the best possible I/O call for an application. Therefore, detecting the best possible storage and access patterns automatically is very useful for many programmers and increases the performance of I/O-intensive applications significantly.
References 1. A. Ancourt, F. Coelho, F. Irigoin, and R. Keryell. A linear algebra framework for static HPF code distribution. Scientific Prog., 6(1):3–28, Spring 1997.
1272
Gokhan Memik, Mahmut T. Kandemir, and Alok Choudhary
2. J. Anderson. Automatic Computation and Data Decomposition for Multiprocessors. Ph.D. dissertation, Computer Systems Lab., Stanford Univ., March 1997. 3. R. Chandra, D. Chen, R. Cox, D. Maydan, N. Nedeljkovic, and J. Anderson. Datadistribution support on distributed-shared memory multi-processors. In Proc. Prog. Lang. Design and Implementation, Las Vegas, NV, 1997. 4. A. Choudhary, R. Bordawekar, M. Harry, R. Krishnaiyer, R. Ponnusamy, T. Singh, and R. Thakur. Passion: Parallel and scalable software for input-output. NPAC Technical Report SCCS-636, Sept 1994. 5. P. Corbett et al. Overview of the MPI-IO parallel I/O interface, In Proc. Third Workshop on I/O in Par. and Dist. Sys., IPPS’95, Santa Barbara, CA, April 1995. 6. J. del Rosario and A. Choudhary. High performance I/O for parallel computers: problems and prospects. IEEE Computer, March 1994. 7. N. Gloy, T. Blackwell, M. D. Smith, and B. Calder. Procedure placement using temporal ordering information. In Proc. Micro-30, Research Triangle Park, North Carolina, December 1–3, 1997. 8. M. Gupta and P. Banerjee. Demonstration of automatic data partitioning techniques for parallelizing compilers on multicomputers. IEEE Transactions on Parallel and Distributed Systems, 3(2):179–193, March 1992. 9. M. W. Hall, B. Murphy, S. Amarasinghe, S. Liao, and M. Lam. Inter-procedural analysis for parallelization. In Proc. 8th International Workshop on Lang. and Comp. for Parallel Computers, pages 61–80, Columbus, Ohio, August 1995. 10. W. Kelly, V. Maslov, W. Pugh, E. Rosser, T. Shpeisman, and David Wonnacott. The Omega Library interface guide. Technical Report CS-TR-3445, CS Dept., University of Maryland, College Park, March 1995. 11. D. Kotz. Disk-directed I/O for MIMD multiprocessors. In Proc. the 1994 Symposium on Operating Systems Design and Implementation, pages 61–74, November 1994. Updated as Dartmouth TR PCS-TR94-226 on November 8, 1994. 12. B. J. Nitzberg. Collective Parallel I/O. PhD thesis, Department of Computer and Information Science, University of Oregon, December 1995. 13. K. E. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett. Server-directed collective I/O in Panda. In Proceedings of Supercomputing’95, December 1995. 14. R. Thakur, W. Gropp, and E. Lusk. A case for using MPI’s derived data types to improve I/O performance. In Proc. of SC’98: High Performance Networking and Computing, November 1998.
Effective File-I/O Bandwidth Benchmark Rolf Rabenseifner1 and Alice E. Koniges2 1
High-Performance Computing-Center (HLRS), University of Stuttgart Allmandring 30, D-70550 Stuttgart, Germany [email protected], www.hlrs.de/people/rabenseifner/ 2 Lawrence Livermore National Laboratory, Livermore, CA 94550 [email protected], www.rzg.mpg.de/∼ack
Abstract. The effective I/O bandwidth benchmark (b eff io) covers two goals: (1) to achieve a characteristic average number for the I/O bandwidth achievable with parallel MPI-I/O applications, and (2) to get detailed information about several access patterns and buffer lengths. The benchmark examines “first write”, “rewrite” and “read” access, strided (individual and shared pointers) and segmented collective patterns on one file per application and non-collective access to one file per process. The number of parallel accessing processes is also varied and wellformed I/O is compared with non-wellformed. On systems, meeting the rule that the total memory can be written to disk in 10 minutes, the benchmark should not need more than 15 minutes for a first pass of all patterns. The benchmark is designed analogously to the effective bandwidth benchmark for message passing (b eff) that characterizes the message passing capabilities of a system in a few minutes. First results of the b eff io benchmark are given for IBM SP, Cray T3E and NEC SX-5 systems and compared with existing benchmarks based on parallel Posix-I/O. Keywords. MPI, File-I/O, Disk-I/O, Benchmark, Bandwidth.
1
Introduction
Most parallel I/O benchmarks and benchmarking studies characterize the hardware and file system performance limits [2,4,5,6]. Often, they focus on determining under which conditions the maximal file system performance can be reached on a specific platform. Such results can guide the user in choosing an optimal access pattern for a given machine and file system, but do not generally consider the needs of the application over the needs of the file system. Our approach begins with consideration of the possible I/O requests of parallel applications. To formulate such I/O requests, the MPI Forum has standardized the MPI-I/O interface [7]. Major goals of this standardization are to express the user’s needs and to allow optimal implementations of the MPI-I/O interface on all platforms [3,8,11,12]. Based on this background, the effective I/O bandwidth benchmark (b eff io) should measure different access patterns, report A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1273–1283, 2000. c Springer-Verlag Berlin Heidelberg 2000
1274
Rolf Rabenseifner and Alice E. Koniges
these detailed results, and should calculate an average I/O bandwidth value that characterizes the whole system. This goal is analogous to the Linpack value reported in TOP500 [16] that characterizes the computational speed of a system, and also to the effective bandwidth benchmark (b eff), that characterizes the communication network of a distributed system [9,14,15]. A major difference between b eff and b eff io is the magnitude of the bandwidth. On well-balanced systems in high performance computing we expect an I/O bandwidth which allows for writing or reading the total memory in approximately 10 minutes. For the communication bandwidth, the b eff benchmark shows, that the total memory can be communicated in 3.2 seconds on a Cray T3E with 512 processors and in 13.6 seconds on a 24 processor Hitachi SR 8000. An I/O benchmark measures the bandwidth of data transfers between memory and disk. Such measurements are (1) highly influenced by buffering mechanisms of the underlying I/O middleware and filesystem details, and (2) high I/O bandwidth on disk requires, especially on striped filesystems, that a large amount of data must be transferred between such buffers and disk. Therefore a benchmark must ensure that a sufficient amount of data is transfered between disk and the application’s memory. The communication benchmark b eff can give detailed answers in about 2 minutes. Later we shall see that b eff io, our I/O counterpart, needs at least 15 minutes to get a first answer.
2
Multidimensional Benchmarking Space
Often, benchmark calculations sample only a small subspace of a multidimensional parameter space. One extreme example is to measure only one point, e.g., a communication bandwidth between two processors using a ping-pong communication pattern with 8 Mbyte messages, repeated 100 times. For I/O benchmarking, a huge number of parameters exist. We divide the parameters into 6 general categories. At the end of each category in the following list, a first hint about handling these aspects in b eff io is noted. The detailed definition of b eff io is given in section 4. 1. Application parameters are (a) the size of contiguous chunks in the memory, (b) the size of contiguous chunks on disk, which may be different in the case of scatter/gather access patterns, (c) the number of such contiguous chunks that are accessed with each call to a read or write routine, (d) the file size, (e) the distribution scheme, e.g., segmented or long strides, short strides, random or regular, or separate files for each node, and (f) whether or not the chunk size and alignment are wellformed, e.g., a power of two or a multiple of the striping unit. For b eff io, 36 different patterns are used to cover most of these aspects. 2. Usage aspects are (a) how many processes are used and (b) how many parallel processors and threads are used for each process. To keep these aspects outside of the benchmark, b eff io is defined as a maximum over these aspects and one must report the usage parameters used to achieve this maximum.
Effective File-I/O Bandwidth Benchmark
1275
3. The major programming interface parameter is specification of which I/O interface is used: Posix I/O buffered or raw, special filesystem I/O of the vendor’s filesystem, or MPI-I/O. In this benchmark, we use only MPI-I/O, because it should be a portable interface of an optimal implementation on top of Posix I/O or the special filesystem I/O. 4. MPI-I/O defines the following orthogonal aspects: (a) access methods, i.e., first writing of a file, rewriting or reading, (b) positioning method, i.e., explicit offsets, individual or shared file pointers, (c) coordination, i.e., accessing the file collectively by (all) processes or noncollectively, (d) synchronism, i.e., blocking or nonblocking. Additional aspects are: (e) whether or not the files are open unique, i.e., the file will not be concurrently opened by a different open call, and (f) which consistency is chosen for conflicting accesses, i.e., whether or not atomic mode is set. For b eff io there is no overlap of I/O and computation, therefore only blocking calls are used. Because there should not be a significant difference between the efficiency of using explicit offsets or individual file pointers, only the individual and shared file pointers are benchmarked. With regard to (e) and (f), unique and nonatomic are used. 5. Filesystem parameters are (a) which filesystem is used, (b) how many nodes or processors are used as I/O servers, (c) how much memory is used as bufferspace on each application node, (d) the disk block size, (e) the striping unit size, and (f) the number of parallel striping devices that are used. These aspects are also outside the scope of b eff io. The chosen filesystem, its parameters and any usage of non-default parameters must be reported. 6. Additional benchmarking aspects are (a) repetition factors, and (b) how to calculate b eff io, based on a subspace of the parameter space defined above using maximum, average, weighted average or logarithmic averages. To reduce benchmarking time to an acceptable amount, one can normally only measure I/O performance at a few grid points of a 1-5 dimensional subspace. To analyze more than 5 aspects, usually more than one subspace is examined. Often, the common area of these subspaces is chosen as the intersection of the area of best results of the other subspaces. For example in [5], the subspace varying the number of servers is obtained with segmented access patterns, and with well-chosen block sizes and client:server ratios. Defining such optimal subspaces can be highly system-dependent and may therefore not be as appropriate for a b eff io designed for a variety of systems. For the design of b eff io, it is important to choose the grid points based more on general application needs than on optimal system behavior.
3
Criteria
The benchmark b eff io should characterize the I/O capabilities of the system. Should we use, therefore, only access patterns, that promise a maximum bandwidth? No, but there should be a good chance that an optimized implementation
1276 type 0
Rolf Rabenseifner and Alice E. Koniges l
L 1 MB MP ART :=l 1 MB 2 MB 1 MB 1 MB 32 kB 1 MB 1 kB 1 MB 32 kB +8B 1 MB + 256B 1 kB +8B 1 MB + 8kB 1 MB +8B 1 MB + 8B 1 MB
U 0 4 4 4 2 2 2 2 2
type 1
l 1 MB
MP ART 1 MB 32 kB 1 kB 32 kB +8B 1 kB +8B 1 MB +8B
L :=l :=l :=l :=l :=l :=l :=l :=l
U 0 4 2 1 1 1 1 2
type 2
l 1 MB
MP ART 1 MB 32 kB 1 kB 32 kB +8B 1 kB +8B 1 MB +8B 3/4 see type=2
L :=l :=l :=l :=l :=l :=l :=l :=l
U 0 2 2 1 1 1 1 2
U = 64
Table 1. The pattern details used in b eff io
of MPI-I/O should be able to achieve a high bandwidth. This means that we should measure patterns that can be recommended to application developers. An important criterion is that the b eff io benchmark should only need about 10 to 15 minutes. For first measurements, it need not run on an empty system as long as concurrently running other applications do not use a significant part of the I/O bandwidth of the system. Normally, the full I/O bandwidth can be reached by using less than the total number of available processors or SMP nodes. In contrast, the communication benchmark b eff should not require more than 2 minutes, but it must run on the whole system to compute the aggregate communication bandwidth. Based on the rule for well-balanced systems mentioned in the introduction and assuming that MPI-I/O will attain at least 50 percent of the hardware I/O bandwidth, we expect that a 10 minute b eff io run can write or read about 16 % of the total memory of the benchmarked system. For this estimate, we divide the total benchmark time into three intervals based on the following access methods: initial write, rewrite, and read. However, a first test on a T3E900-512 shows that based on the pattern-mix, only about the third of this theoretical value is transferred. Finally, as a third important criterion, we want to be able to compare different common access patterns.
4
Definition of the Effective I/O Bandwidth
The effective I/O bandwidth benchmark measures the following aspects: – a set of partitions, – the access methods initial write, rewrite, and read, – the pattern types (see Fig. 1) (0) strided collective access, scattering large chunks in memory to/from disk, (1) strided collective access, but one read or write call per disk chunk, (2) noncollective access to one file per MPI process, i.e., on separated files, (3) same as (2), but the individual files are assembled to one segmented file, (4) same as (3), but the access to the segmented file is done with collective routines; for each pattern type, an individual file is used.
Effective File-I/O Bandwidth Benchmark
1277
– the contiguous chunk size is chosen wellformed, i.e., as a power of 2, and non-wellformed by adding 8 bytes to the wellformed size, – different chunk sizes, mainly 1 kB, 32 kB, 1 MB, and the maximum of 2 MB and 1/128 of the memory size of a node executing one MPI process. The total list of patterns is shown in Tab. 1. The column “type” refers to the pattern type. The column “l” defines the contiguous chunks that are written from memory to disk and vice versa. The value MP ART is defined as max(2 MB, memory of one node / 128). The column “L” defines the contiguous chunk in the memory. In case of pattern type (0), non-contiguous file views are used. If l is less than L, then in each MPI-I/O read/write call, the L bytes in memory are scattered/gathered to/from the portions of l bytes at the different locations on disk, see the left-most scenario in Fig. 1. In all other cases, the contiguous chunk handled by each call to MPI Write or MPI Read is equivalent in memory and on disk. This is denoted by “:=l” in the L column. U is a time unit. Each pattern is benchmarked by repeating the pattern for a given amount of time. This time is given by the allowed time for a whole partition (e.g., T = 10 minutes) multiplied by U/ U/3, as given in the table. This time-driven approach allows one to limit the total execution time. For the pattern types (3) and (4) a fixed segment size must be computed before starting the pattern of these types. Therefore, the time-driven approach is substituted by a size-driven approach, and the repeating factors are initialized based on the measurements for types (0) to (2). The b eff io value of one partition is defined as the sum of all transferred bytes divided by the total transfer time. If patterns do not need exactly the ideal allowed time, then the average is weighted by the unit U . At a minimum, 10 minutes must be used for benchmarking one partition. The b eff io of a system is defined as the maximum over any b eff io of a single partition of the system. This definition permits the user of the benchmark to freely choose the usage aspects and enlarge the total filesize as desired. The minimum filesize is given by the bandwidth for an initial write multiplied by 200 sec (= 10 minutes / 3 access methods). If a system complies with our rule that the total memory can be written in 10 minutes for each access pattern, then one third of the total memory is written by the complete benchmark, and in each single pattern with U =1, one 1/192 of the total memory is written. If all processors are used for this benchmark, then the amount written by each node is not very much, but a call to MPI File sync in each pattern may imply that the data is really written to disk. However this assumption is not valid on all systems. For example, on NEC SX systems, MPI File sync guarantees only the semantic stated in the MPI-2 standard. The data on the file must be visible to any other application, but the data can stay in a memory buffer controlled by the filesystem’s software. Therefore the benchmark rule, that at least 10 minutes are used for one run, had to be modified for this system. In the current version we use for the SX-5 measurements, we require that the total amount of data written with the initial write-calls must be at least equal to the total amount of the memory of the
1278
Rolf Rabenseifner and Alice E. Koniges
system. Thus, on the SX-5 we had to increase the scheduled benchmark time to T =30 minutes.
5
Comparing Systems Using b eff io
In this section, we present a detailed analysis of each run of b eff io on a partition. We test b eff io on three systems, the Cray T3E900-512 and SX-5Be/32M2 at HLRS/RUS in Stuttgart and an RS 6000/SP system at LLNL called “blue.” On the T3E, we use the tmp-filesystem with 10 striped Raid-disks connected via a GigaRing for the benchmark. The peak-performance of the aggregated parallel bandwidth of this hardware configuration is about 300 MB/s. The LLNL results presented here are for an SP system with 336 SMP nodes each with four 332 MHz processors. Since the I/O performance on this system does not increase significantly with the number of processors on a given node performing I/O, all test results assume a single thread on a given node is doing the I/O. Thus, a 64 processor run means 64 nodes assigned to I/O, and no requested computation by the additional 64*3 processors. On the SP system, the data is written to the IBM General Parallel File System (GPFS) called blue.llnl.gov:/g/g1 which has 20 VSD I/O servers. Recent results for this system show a maximum read performance of approximately 950MB/sec for a 128 node job, and a maximum write performance of 690MB/sec for 64 nodes [5].1 Note that these are the maximum values observed, and performance degrades when the access pattern and/or the node number is changed. The NEC SX-5 system has four striped RAID-3 arrays DS 1200, connected by fibre channel. The SFS filesystem parameters are: 4 MB cluster size (=block size), and if the size of an I/O request is less than 1 MB then a 2 GB filesystem-cache is used. On both platforms, MPI-I/O is implemented with ROMIO but with different device drivers. On the T3E, we have modified the MPI Release mpt.1.3.0.2, by substituting the ROMIO/ADIO Unix filesystem driver routines for opening, writing and reading files. The Posix routines were substituted by the asynchronous counter part, directly followed by the the wait routine. This trick enables parallel disk access [13]. On the RS 6000/SP blue machine, GPFS is used underneath the MPICH version of MPI with ROMIO. On the SX-5, we use MPI/SX 10.1. For each run of b eff io, the I/O bandwidth for each chunk size and pattern is reported in a table that can be plotted in the pictures shown in each row in Fig. 2. First, consider the first two rows of Fig. 2. They show the results of one benchmark on the SP and T3E systems, both scheduled to run T = 10 minutes, during which time other applications were running on the other processors of the systems. They demonstrate the main differences between both MPI and filesystem implementations. Based on the results in Fig. 3, which we discuss later, we decided to run the benchmark on the T3E on 32 processors and on the SP 1
Upgrades to the AIX operating system and underlying GPFS software may have altered these performance numbers slightly between measurements in [5] and in the current work.
Effective File-I/O Bandwidth Benchmark PE 0
PE 1
PE 2
PE 0
PE 3
PE 1
PE 2
L
PE 0
PE 3
memory
PE 1
PE 2
PE 0
PE 3
memory
memory
file
files
PE 1
PE 2
1279
PE 3
memory
l
file
file LSEG
pattern type 0
pattern type 1
pattern type 2
pattern type 3/4
Fig. 1. Data transfer patterns used in b eff io. Each diagram shows the data transferred by one MPI-I/O write call.
blue RS 6000/SP 1000 rewrite [MB/s]
first write [MB/s]
1000
10000 type 0 type 1 type 2 type 3 type 4
100
10
1
0.1
10000 type 0 type 1 type 2 type 3 type 4
blue RS 6000/SP
blue RS 6000/SP 1000
read [MB/s]
10000
100
10
1
0.1
1k +8 32k +8 1M +8 8M contiguous chunks on disk [bytes]
100
10 type 0 type 1 type 2 type 3 type 4
1
0.1
1k +8 32k +8 1M +8 8M contiguous chunks on disk [bytes]
1k +8 32k +8 1M +8 8M contiguous chunks on disk [bytes]
(a) 128 PEs on the “blue” RS 6000/SP at LLNL, T = 10 min, b eff io = 311 MB/s hww T3E900-512 1000 rewrite [MB/s]
first write [MB/s]
1000
10000 type 0 type 1 type 2 type 3 type 4
100
10
1
0.1
10000 type 0 type 1 type 2 type 3 type 4
hww T3E900-512 1000
read [MB/s]
10000
100
10
1
0.1
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
type 0 type 1 type 2 type 3 type 4
hww T3E900-512
100
10
1
0.1
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
1000 100 10 1 0.1
SUPER-UX hwwsx5 10.1 Rev1 SX-5 100000 scatter/coll - type 0 b_eff_io strided/coll - type 1 rel. 0.6 separate - type 2 T=30.0min 10000 segmented - type 3 segm./coll - type 4 1000 100 10 1
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
0.1
bandwidth - read [MB/s]
SUPER-UX hwwsx5 10.1 Rev1 SX-5 100000 scatter/coll - type 0 b_eff_io strided/coll - type 1 rel. 0.6 separate - type 2 T=30.0min 10000 segmented - type 3 segm./coll - type 4
bandwidth - rewrite [MB/s]
bandwidth - first write [MB/s]
(b) 32 PEs on the T3E900-512 at HLRS, T = 10 min, b eff io = 71 MB/s SUPER-UX hwwsx5 10.1 Rev1 SX-5 100000 scatter/coll - type 0 b_eff_io strided/coll - type 1 rel. 0.6 separate - type 2 T=30.0min 10000 segmented - type 3 segm./coll - type 4 1000 100 10 1
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
0.1
1k +8 32k +8 1M +8 2M contiguous chunks on disk [bytes]
(c) 4 PEs on the SX-5Be/32M2 at HLRS, T = 30 min, b eff io = 60 MB/s
Fig. 2. Comparison of the results on T3E, SP and SX-5
1280
Rolf Rabenseifner and Alice E. Koniges 100
400
80
60
40 T=1800s T=600s T=300s T=150s T=120s
20
0
2
4
8 16 32 64 128 256 512 number of MPI processes
blue RS 6000/SP
350 b_eff_io of one partition [MB/s]
b_eff_io of one partition [MB/s]
hww T3E900-512
300 250 200 150 T=600s T=300s
100 50 0
2
4
8 16 32 64 128 256 512 number of MPI processes
Fig. 3. Comparison b eff io of different numbers of processes on SP and T3E, measured partially without pattern type 3.
on 128 processors. The three diagrams in each row of Fig. 2 show the bandwidth achieved for the three different access methods: writing the file the first time, rewriting the same file, and reading it. On each diagram, the bandwidth is plotted on a logarithmic scale, separately for each pattern type and as a function of the chunk size. The chunk size on disk is shown on a pseudo-logarithmic scale. The points labeled “+8” are the non-wellformed counterparts of the power of two values. The maximum chunk size is different on both systems because the maximum chunk size was chosen proportional to the memory size per node to reflect the scaling up of applications on larger systems. On the SX-5, a reduced maximum chunk size was used. Type 0 is a strided access, but the buffer used in each I/O-call is at least 1 MB. In the case of a chunk length less than 1 MB, the buffer contents must be scattered to different places in the file. On the T3E, this pattern type is optimal, except for chunks larger than 1 MB, where the initial write of segmented files is faster. When non-wellformed chunk sizes are used, there is a substantial drop in performance. Additional measurements show that this problem increases with the total amount of data written to disk. On the RS 6000/SP, other pattern types show higher bandwidth. Type 1 writes the same data to disk, i.e., each process has the same logical fileview, but MPI-IO is called for each chunk separately. In the current benchmark, this test is done with individual filepointers, because the MPI-I/O ROMIO implementation on both systems does not have shared filepointers. By default, b eff io measures this pattern type with shared pointers when available. On both platforms, this pattern type results in essentially the worst bandwidth for most access methods and chunk sizes. Type 2 is the writing winner on RS 6000/SP. Each process writes a separate file at the same time, i.e., parallel and independently. (We note that optimized vendor supplied MPI-IO implementations may do a better job with other pattern types.) Type 3 writes in the same pattern, but the files of all processes are
Effective File-I/O Bandwidth Benchmark
1281
concatenated. To guarantee wellformed starting points for each process, the filesize of each process is rounded up to the next MByte. Type 4 writes in the same way as type 3, but the access is done collectively. On the T3E, we see that these three pattern types are consistently slow for small buffer sizes and consistently fast for large buffer sizes. In contrast on the RS 6000/SP, type 3 and 4 are about a factor2 of 10–20 slower than type 2 for writing files. For reading files, the diagram cannot show the real speed for type 3 and 4 due to three effects: The repetition factor is only one for chunk sizes of 1 MB and more, the reading of the 8 MB chunk fills internal buffers, and currently, the b eff io does not perform a file sync operation before reading a pattern. Looking at the (non-weighted) average, we see that on the RS 6000/SP, reading the segmented files is a factor of 2.5 slower than reading individual files. Finally on both systems, the read access is clearly faster than the write access. On the T3E, the read access is 5 times faster than “first write” and 2.7 faster than “rewrite”. On the RS 6000/SP blue machine, the read access is 10 times faster than both types of write access. The measurements were done with b eff io Release 0.5 [10]. The last row of Fig. 2 shows the measurement on the SX-5. It had to be done with the longer schedule time of T = 30 minutes to assure that most of the I/O operations are done on real disks and not only in the filesystem’s internal buffer space. The curves show still some hot spots that may be caused by pure memory copying. One can see that the scattering-pattern type 0 and the separate-filepattern type 2 perform the best. There is little difference between wellformed and non-wellformed I/O. Write and read bandwidth are similar. For long chunk sizes, reading from separate files (pattern type 2) is faster than the gathering/strided accesses (type 0 and 1) and the segmented accesses (type 3 and 4). Figure 3 shows the b eff io values for different partition sizes and different values of T , the time scheduled for benchmarking one partition. All measurements were taken in a non-dedicated mode. For the T3E, the maximum is reached at 32 application processes, with little variation from 8 to 128 processors. In general, an application only makes I/O requests for a small fraction of the compute time. On large systems, such as those at the High-Performance Computing Center at Stuttgart and the Computing Center at Lawrence Livermore National Laboratory, several applications are sharing the nodes, especially during prime time usage. In this situation, I/O capabilities would not be requested by a significant proportion of the CPU’s at the same time. “Hero” runs, where one application ties up the entire machine for a single calculation are rarer and generally run during non-prime time. Such hero runs can require the full I/O performance by all processors at the same time. The right-most diagram shows that the RS 6000/SP fits more to this latter usage model. Note that GPFS on the SP’s is configurable, i.e., number of I/O servers and other tunables, and the performance on any given SP/GPFS system depends on the configuration of that system. 2
All factors in this section are computed, based on weighted averages using the time units U , if not stated otherwise.
1282
Rolf Rabenseifner and Alice E. Koniges
Figure 3 also shows that on both systems, the results depend more on the I/O usage of the other concurrently running applications on the system than on the requested time T for each benchmark. Comparison of measurements with T = 10 and 30 minutes shows that the analysis reported in Fig. 2 may vary in details. For instance, the differences between wellformed and non-wellformed I/O is more notable with T = 30 minutes on the T3E. Finally, we compare these results with other measurements. On the same RS 6000/SP, Posix read and write measurements ranging between 500 and 900 MB/s are measured [5]. 3 The b eff io result is 311 MB/s in the presented measurement. This means that the MPI application programmer has a real chance to get a significant part of the I/O capabilities of that system. On the T3E studied, the peak I/O-performance is about 300 MB/s. Thus the b eff io value of 71 MB/s shows that on average, only a quarter of the peak can be attained with normal MPI programming. We also note that the ROMIO implementation on the RS 6000/SP has not been optimized for the GPFS filesystem. Vendor implementations and future versions of ROMIO should show performance closer to peak. In general, our results show that the b eff io benchmark is a very fast method to analyze the parallel I/O capabilities available for applications using the standardized MPI-I/O programming interface. The resulting b eff io value summarizes I/O capabilities of a system in one significant I/O bandwidth value.
6
Outlook
It is planned to use this benchmark to compare several additional systems. More investigation is necessary to solve problems arising from 32 bit integer limits and handling read buffers in combination with file sync operations. Although [1] stated, that “the majority of the request patterns are sequential”, we should examine whether random access patterns can be included into the b eff io benchmark.
Acknowledgments The authors would like to acknowledge their colleagues and all the people that supported this project with suggestions and helpful discussions. At HLRS, they would especially like to thank Karl Solchenbach and Rolf Hempel for productive discussions for the redesign of b eff. At LLNL, they thank Kim Yates and Dave Fox. Work at LLNL was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48. 3
Again we note that upgrades to the AIX operating system and underlying GPFS software may have slightly altered these performance numbers between measurements.
Effective File-I/O Bandwidth Benchmark
1283
References 1. P. Crandall, R. Aydt, A. Chien, D. Reed, Input-Output Characteristics of Scalable Parallel Applications, In Proceedings of Supercomputing ’95, ACM Press, Dec. 1995, www.supercomp.org/sc95/proceedings/. 2. Ulrich Detert, High-Performance I/O on Cray T3E, 40th Cray User Group Conference, June 1998. 3. Philip M. Dickens, A Performance Study of Two-Phase I/O, in D. Pritchard, J. Reeve (eds.), Proceedings of the 4th Internatinal Euro-Par Conference, EuroPar’98, Parallel Processing, LNCS–1470, pages 959–965, Southampton, UK, 1998. 4. Peter W. Haas, Scalability and Performance of Distributed I/O on Massively Parallel Processors, 40th Cray User Group Conference, June 1998. 5. Terry Jones, Alice Koniges, R. Kim Yates, Performance of the IBM General Parallel File System, to be published in Proceedings of the International Parallel and Distributed Processing Symposium, May 2000. Also available as UCRL JC135828. 6. Kent Koeninger, Performance Tips for GigaRing Disk I/O, 40th Cray User Group Conference, June 1998. 7. Message Passing Interface Forum. MPI-2: Extensions to the Message-Passing Interface, July 1997, www.mpi-forum.org. 8. J.P. Prost, R. Treumann, R. Blackmore, C. Harman, R. Hedges, B. Jia, A. Koniges, A. White, Towards a High-Performance and Robust Implementation of MPI-IO on top of GPFS, Internal report. 9. Rolf Rabenseifner, Effective Bandwidth (b eff ) Benchmark, www.hlrs.de/mpi/b eff/. 10. Rolf Rabenseifner, Effective I/O Bandwidth (b eff io) Benchmark, www.hlrs.de/mpi/b eff io/. 11. Rajeev Thakur, William Gropp, and Ewing Lusk, On Implementing MPI-IO Portably and with High Performance, in Proc. of the Sixth Workshop on I/O in Parallel and Distributed Systems, pages 23-32, May 1999. 12. Rajeev Thakur, Rusty Lusk, Bill Gropp, ROMIO: A High-Performance, Portable MPI-IO Implementation, www.mcs.anl.gov/romio/. 13. Rolf Rabenseifner, Striped MPI-I/O with mpt.1.3.0.1, www.hlrs.de/mpi/mpi t3e.html#StripedIO. 14. Karl Solchenbach, Benchmarking the Balance of Parallel Computers, SPEC Workshop on Benchmarking Parallel and High-Performance Computing Systems, Wuppertal, Germany, Sept. 13, 1999. 15. Karl Solchenbach, Hans-Joachim Plum and Gero Ritzenhoefer, Pallas Effective Bandwidth Benchmark – source code and sample results, ftp://ftp.pallas.de/pub/PALLAS/PMB/EFF BW.tar.gz. 16. Universities of Mannheim and Tennessee, TOP500 Supercomputer Sites, www.top500.org.
,QVWDQW,PDJH 7UDQVLWLYHDQG&\FOLFDO6QDSVKRWVLQ 'LVWULEXWHG6WRUDJH9ROXPHV 3UDVHQMLW6DUNDU 6WRUDJH6\VWHPVDQG6HUYHUV'LYLVLRQ ,%0$OPDGHQ5HVHDUFK&HQWHU 6DQ-RVH&$ [email protected] $EVWUDFW 6QDSVKRWV DUH D XVHIXO PHFKDQLVP WR PDQDJH GDWD SDUWLFXODUO\ IRU EDFNXSV LQ KLJKDYDLODELOLW\ VWRUDJH V\VWHPV 7KLV SDSHU SUHVHQWV WKH ,QVWDQW ,PDJH DOJRULWKP IRU KDQGOLQJ VQDSVKRWV RI VWRUDJH YROXPHV LQ D GLVWULEXWHG VWRUDJH V\VWHP 7KH DOJRULWKP SODFHV QR UHVWULFWLRQV RQ WKH FKRLFH RI VWRUDJH YROXPHVDOORZLQJVQDSVKRWUHODWLRQVKLSVEHWZHHQVWRUDJHYROXPHVWKDWFDQEH WUDQVLWLYHDQGF\FOLFDO,QDGGLWLRQLQDGLVWULEXWHGHQYLURQPHQWZLWKVQDSVKRW UHODWLRQVKLSV LQYROYLQJ Q VWRUDJH VXEV\VWHPV UHDGV DQG ZULWHV WR VWRUDJH YROXPHV LQYROYH RQO\ 2 VWRUDJH VXEV\VWHPV SHU UHDG RU ZULWH WKHUHE\ UHGXFLQJ PHVVDJLQJ FRVWV )LQDOO\ WKH DOJRULWKP LV QRW VSHFLILF WR VWRUDJH V\VWHPV DQG FDQ EH DSSOLHG WR RWKHU FRQWH[WV UHTXLULQJ VQDSVKRWV $ SHUIRUPDQFHDQDO\VLVLQGLFDWHVGRXEOLQJRIDYHUDJHUHDGDQGZULWHSHUIRUPDQFH RQDFOXVWHURIVWRUDJHVXEV\VWHPV
,QWURGXFWLRQ 6QDSVKRWV DUH D YHU\ XVHIXO PHFKDQLVP WR PDQDJH VWRUHG GDWD $ VQDSVKRW RI D FRQWLQXDOO\ FKDQJLQJ VWRUDJH YROXPH LV WDNHQ DW D SDUWLFXODU WLPH DQG FDSWXUHV WKH VWDWHRIWKHVWRUDJHYROXPHDWWKDWWLPH6QDSVKRWVKDYHEHHQZLGHO\XVHGLQVWRUDJH ILOHV\VWHP DQG GDWDEDVH DUFKLWHFWXUHV DV WKH\ DUH YHU\ XVHIXO LQ YDULRXV DSSOLFDWLRQV VXFKDVEDFNXSDQGWHVWLQJ>@)RUH[DPSOHRQHSUREOHPRIWDNLQJDEDFNXSRID VWRUDJH YROXPH LV WKDW WKH GDWD LQ WKH YROXPH FKDQJHV ZKLOH WKH YROXPH LV EHLQJ EDFNHG XS PDNLQJ LW GLIILFXOW WR JHW D FRQVLVWHQW SLFWXUH RI WKH GDWD 7KLV LV SDUWLFXODUO\ WUXH RI WHUDE\WH VWRUDJH YROXPHV WKDW PD\ WDNH KRXUV WR EDFN XS WRWDOO\ DQGUHTXLUHVDGPLQLVWUDWRUVWREORFNDFFHVVWRWKHVWRUDJH YROXPHVEHLQJEDFNHGXS 6QDSVKRWVREYLDWHWKLVSUREOHPE\FUHDWLQJDSUHFLVHLPDJHRIDVWRUDJHYROXPHDWD SDUWLFXODUWLPHLQVWDQWDQGWKHQWDNLQJDEDFNXSRIWKHVQDSVKRW7KHYROXPHUHPDLQV DFFHVVLEOH WR FOLHQWV RI WKH VWRUDJH V\VWHP DQG WKH DGPLQLVWUDWRU GRHV QRW KDYH WR ZRUU\ DERXW WKH FKDQJHV WR WKH YROXPH 7KLV IHDWXUH RI VQDSVKRWV HQGHDUV LW WR WKH ZRUOGRIHOHFWURQLFFRPPHUFHVHUYHUVZKHUHGRZQWLPHLVDOX[XU\ $SDUDOOHOGHYHORSPHQWIXHOHGE\WKHGHYHORSPHQWRIKLJKEDQGZLGWKQHWZRUNVLVWKH GHYHORSPHQWRIGLVWULEXWHGVWRUDJHV\VWHPVVXFKDV3HWDO>@DQG[)6>@$GLVWULEXWHG VWRUDJHV\VWHPFRQVLVWVRIVHYHUDOVWRUDJHVXEV\VWHPV LHGULYHDUUD\V FRQQHFWHGE\ DKLJKEDQGZLGWKQHWZRUNDQGSUHVHQWVDXQLILHGVWRUDJHDGGUHVVVSDFHWRWKHFOLHQWV RI WKH VWRUDJH V\VWHP 'LVWULEXWHG VWRUDJH V\VWHPV DOVR DOORZ IRU HDV\ DGGLWLRQ DQG UHPRYDO RI VWRUDJH VXEV\VWHPV ZLWKRXW DQ\ GLVUXSWLRQ RI VHUYLFH /DVW EXW $%RGHHWDO (GV (XUR3DU/1&6 SS ©6SULQJHU9HUODJ%HUOLQ+HLGHOEHUJ
,QVWDQW,PDJH 7UDQVLWLYHDQG&\FOLFDO6QDSVKRWVLQ'LVWULEXWHG6WRUDJH9ROXPHV
LPSRUWDQWO\GLVWULEXWHGVWRUDJHV\VWHPVDOORZIRUDVSHFLDOL]HGVWRUDJHV\VWHPFOLHQW WRGREDFNXSVDQGUHGXFHWKH&38XWLOL]DWLRQVRIWKHUHPDLQLQJFOLHQWVRIWKHVWRUDJH V\VWHP,QDFRQYHQWLRQDOVWRUDJHV\VWHPSHUIRUPDQFHLVDIIHFWHGZKHQDFOLHQWEDFNV XSDVWRUDJHYROXPHZKLOHVHUYLQJDSSOLFDWLRQV 7KH LQVWDQWDQHRXV QDWXUH RI VQDSVKRWV JLYHV ULVH WR P\ULDG UHODWLRQVKLSV EHWZHHQ VWRUDJH YROXPHV )RU H[DPSOH LI DQ DGPLQLVWUDWRU WDNHV D VQDSVKRW RI D VWRUDJH YROXPH RQWR D VHFRQG VWRUDJH YROXPH DQG WKH VQDSVKRW FRPSOHWHV LQVWDQWDQHRXVO\ WKHQQRWKLQJSUHYHQWVWKHDGPLQLVWUDWRUIURPWDNLQJDVQDSVKRWRIWKHVHFRQGVWRUDJH YROXPH)XUWKHUPRUHWKHUHLVQRUHVWULFWLRQRQWKHGHVWLQDWLRQRIWKHVHFRQGVQDSVKRW LQRWKHUZRUGVWKHDGPLQLVWUDWRUPD\DVZHOOGHFLGHWRWDNHDVQDSVKRWRIWKHVHFRQG VWRUDJH YROXPH RQWR WKH ILUVW 7KLV OHDGV WR ERWK WUDQVLWLYH DQG F\FOLFDO VQDSVKRW UHODWLRQVKLSV EHWZHHQ VWRUDJH YROXPHV ([LVWLQJ V\VWHPV OLNH GDWDEDVHV GLVDOORZ WUDQVLWLYH DQG F\FOLFDO UHODWLRQVKLSV EHWZHHQ VQDSVKRWV WR PLQLPL]H WKH HIIHFW RQ WUDQVDFWLRQSURFHVVLQJ+RZHYHULQW\SLFDOVWRUDJHV\VWHPVVXFKUHVWULFWLRQVSUHYHQW XVHIXO DSSOLFDWLRQV IURP UXQQLQJ RQ WKH VQDSVKRW )RU H[DPSOH D :HE GHYHORSHU PLJKWZDQWWRWHVWKLVDSSOLFDWLRQRQDVQDSVKRWRIDQRQOLQHGDWDEDVH 7KHLPSOLFDWLRQRIWKHHPHUJHQFHRIGLVWULEXWHGVWRUDJHV\VWHPVLVWKDWVQDSVKRWVDUH QRORQJHUUHVWULFWHGWRWKHVDPHVWRUDJHVXEV\VWHP)RUH[DPSOHDGLVWULEXWHGVWRUDJH V\VWHP PD\ GHFLGH WR DOORFDWH D JLYHQ VWRUDJH YROXPH LQ DQ\ RQH RI LWV FRPSRQHQW VWRUDJH VXEV\VWHPV DV D UHVXOW RI VWRUDJH DYDLODELOLW\ FRQVWUDLQWV 6R DQ\ VQDSVKRW DOJRULWKPPXVWWDNHLQWRWKHGLVWULEXWHG QDWXUHRIWKHSUREOHP DQGHQVXUHWKDWUHDGV DQGZULWHVWRVWRUDJHYROXPHVGRQRWVXIIHUDVDFRQVHTXHQFHRIH[FHVVLYHPHVVDJHV EHLQJH[FKDQJHGEHWZHHQGLIIHUHQWVWRUDJHVXEV\VWHPV 7KLV SDSHU SUHVHQWV WKH ,QVWDQW ,PDJH DOJRULWKP IRU HQDEOLQJ VQDSVKRWV LQ D VWRUDJH V\VWHP ZLWK WKH DELOLW\ WR KDQGOH WUDQVLWLYH DQG F\FOLFDO VQDSVKRW UHODWLRQVKLSV EHWZHHQVWRUDJHYROXPHV7KH,QVWDQW,PDJHDOJRULWKPSODFHVQROLPLWRQWKHQXPEHU RI VQDSVKRW UHODWLRQVKLSV LQ WKH VWRUDJH V\VWHP DQG LV LQGHSHQGHQW RI WKH DFWXDO SK\VLFDO OD\RXW RI D VWRUDJH YROXPH )XUWKHUPRUH WKH DOJRULWKP HQVXUHV WKDW LQ D GLVWULEXWHGHQYLURQPHQWZLWKQVWRUDJHVXEV\VWHPVHYHU\UHDGRUZULWHWRDQ\VWRUDJH YROXPH GRHV QRW LQYROYH PRUH WKDQ 2 VWRUDJH VXEV\VWHPV LQGHSHQGHQW RI WKH QXPEHURIVQDSVKRWVLQWKHVWRUDJHV\VWHP,QIDFWSHUIRUPDQFHDQDO\VLVLQGLFDWHVD GRXEOLQJ RI DYHUDJH UHDG DQG ZULWH SHUIRUPDQFH LQ D VWRUDJH VXEV\VWHP FOXVWHU )LQDOO\ WKH DOJRULWKP LV QRW VSHFLILF WR VWRUDJH V\VWHPV DQG FDQ EH DSSOLHG ZLWKRXW PRGLILFDWLRQWRRWKHUFRQWH[WVWKDWUHTXLUHVQDSVKRWV 7KHUHVWRIWKHSDSHULVRUJDQL]HGDVIROORZV 6HFWLRQLQWURGXFHVGHILQLWLRQVUHOHYDQW WR WKH ,QVWDQW ,PDJH DOJRULWKP DQG LV IROORZHG E\ 6HFWLRQ WKDW GHVFULEHV WKH DOJRULWKP LQ GHWDLO 5HOHYDQW OLWHUDWXUH LV GHVFULEHG LQ 6HFWLRQ IROORZHG E\ FRQFOXVLRQVLQ6HFWLRQ
'HILQLWLRQV $VQDSVKRWFDQEHLQWXLWLYHO\GHVFULEHGDVD³SRLQWLQWLPH´FRS\ ZKHUHDVQDSVKRW RIDVWRUDJHYROXPHEHKDYHVOLNHDFRS\RIWKHVWRUDJHYROXPHDWWKHSRLQWRIWLPHWKH VQDSVKRW LV WDNHQ 7KXV RQH NH\ SDUDPHWHU RI WKH VQDSVKRW LV WKH WLPH RI RSHUDWLRQ
3UDVHQMLW6DUNDU
WKDWLVXVHGWRSUHFLVHO\GHILQHWKHDFWXDOFRQWHQWVRIWKHVQDSVKRW)ROORZLQJWKLVDOO UHDGVDQGZULWHVWRWKHVWRUDJHYROXPHDVZHOODVWKHVQDSVKRWPXVWKRQRUWKHGHILQHG FRQWHQWRIWKHVQDSVKRW)RUH[DPSOHLIDEORFNLQDVWRUDJHYROXPHLVZULWWHQWRDIWHU DVQDSVKRWLVWDNHQWKHQWKHRULJLQDOFRQWHQWVRIWKHEORFNPXVWEHWUDQVIHUUHGWRWKH VQDSVKRW )RUPDOO\ WKH ,QVWDQW ,PDJH VQDSVKRW DOJRULWKP WDNHV WKUHH SDUDPHWHUV D VRXUFH VWRUDJHYROXPHVDWDUJHWVWRUDJHYROXPHWDQGDVSHFLILHGWLPHLQVWDQW7*LYHQWKH VRXUFHVWRUDJHYROXPHVDQGWDUJHWVWRUDJHYROXPHWWKH,QVWDQW,PDJHFRPPDQGDW WLPH 7 FDXVHV WKH WDUJHW VWRUDJH YROXPH W WR EHKDYH DV D ORJLFDO FRS\ RI WKH VRXUFH VWRUDJHYROXPHVDWWLPH7)ROORZLQJWKLVWKHVWRUDJHYROXPHVVDQGWDUHVDLGWREH SDUWRIWKHVDPHVQDSVKRWUHODWLRQVKLS 7RPDLQWDLQWKHVHPDQWLFVRIDVQDSVKRWUHDGVDQGZULWHVWRVDQGWDIWHUWLPH7DUH DIIHFWHGLQWKHIROORZLQJPDQQHU •
$UHDGWRVSURFHHGVXQKLQGHUHG
•
$ZULWHWRWDOVRSURFHHGVXQKLQGHUHG
•
$ZULWHWRVUHVXOWVLQWKHRULJLQDOGDWDEHLQJFRSLHGRYHUIURPVWRWRQO\LIWKH GDWDZDVQRWSUHYLRXVO\FRSLHGRYHUIURPVWRWDQGWKHGDWDZDVQRWZULWWHQWRLQ W)ROORZLQJWKLVWKHZULWHLVDOORZHGWRSURFHHG
•
$UHDGWRWLVUHGLUHFWHGWRVRQO\LIWKHGDWDZDVQRWSUHYLRXVO\FRSLHGRYHUIURP V WR W DQG WKH GDWD ZDV QRW ZULWWHQ WR LQ W 2WKHUZLVH WKH UHDG SURFHHGV XQKLQGHUHG
7KHUHDUHQRUHVWULFWLRQVRQWKHFKRLFHRIVDQGW,QRWKHUZRUGVWKHVRXUFHDQGWDUJHW VWRUDJH YROXPHV FRXOG EH SDUW RI SUHYLRXV VQDSVKRW UHODWLRQVKLSV ZLWK DQ\ RWKHU VWRUDJH YROXPH 7KH JRDO RI WKH ,QVWDQW ,PDJH DOJRULWKP LV WR WDNH LQWR DFFRXQW WKH HIIHFW RI WKHVH SUHYLRXV VQDSVKRW UHODWLRQVKLSV )XUWKHUPRUH WKH ,QVWDQW ,PDJH DOJRULWKP LV DOVR LQGHSHQGHQW RI WKH DFWXDO SK\VLFDO OD\RXW RI ERWK WKH VWRUDJH YROXPHVLQYROYHGLQWKHVQDSVKRWUHODWLRQVKLS 6QDSVKRWVUHTXLUHWKDWWKHWDUJHWVWRUDJHYROXPHEHDEOHWRVWRUHGDWDHTXDOWRWKHVL]H RIWKHVRXUFHVWRUDJHYROXPH,IWKHHQWLUHVRXUFHVWRUDJHYROXPHLVZULWWHQWRDOOWKH RULJLQDOGDWDLQWKHVRXUFHVWRUDJH YROXPH PXVWEHFRSLHGRYHU WRWKHWDUJHWVWRUDJH YROXPH+RZHYHUWKH,QVWDQW,PDJHDOJRULWKPLVLQGHSHQGHQWRIWKHH[DFWPHFKDQLVP E\ ZKLFK WKLV LV DFKLHYHG ,Q FHUWDLQ LPSOHPHQWDWLRQV WKH VRXUFH DQG WDUJHW VWRUDJH YROXPHV VKDUH WKH XQGHUO\LQJ SK\VLFDO VWRUDJH XQWLO ZULWHV WR HLWKHU RI WKH YROXPHV QHFHVVLWDWHV GLVWLQFW SK\VLFDO VWRUDJH>+LW] @ ,Q RWKHU LPSOHPHQWDWLRQV WKH VRXUFH DQGWDUJHWVWRUDJHYROXPHVGRQRWVKDUHDQ\SK\VLFDOVWRUDJH 7KH,QVWDQW,PDJHDOJRULWKPDLPVWRDFKLHYHWKHIROORZLQJSURSHUWLHV •
•
7UDQVLWLYH 5HODWLRQVKLSV *LYHQ WKUHH VWRUDJH YROXPHV $ % & LW LV SRVVLEOH WR VSHFLI\ D VQDSVKRW UHODWLRQVKLS ZLWK $ DV WKH VRXUFH DQG % DV WKH WDUJHW DQG DIWHUZDUGV VSHFLI\ DQRWKHU VQDSVKRW UHODWLRQVKLS ZLWK % DV WKH VRXUFH DQG & DV WKHWDUJHW 0XOWLSOH 7DUJHWV ,W LV SRVVLEOH IRU D VWRUDJH YROXPH WR EH WKH VRXUFH IRU PDQ\ VQDSVKRW UHODWLRQVKLSV )RU H[DPSOH JLYHQ VWRUDJH YROXPHV $ % & ' WKH
,QVWDQW,PDJH 7UDQVLWLYHDQG&\FOLFDO6QDSVKRWVLQ'LVWULEXWHG6WRUDJH9ROXPHV
IROORZLQJFKURQRORJLFDOVHTXHQFHRIVQDSVKRWUHODWLRQVKLSVDUHYDOLG L $DVWKH VRXUFHDQG%DVWKHWDUJHW LL $DVWKHVRXUFHDQG&DVWKHWDUJHW LLL $DVWKH VRXUFHDQG'DVWKHWDUJHW • &\FOLFDO 5HODWLRQVKLSV ,W LV SRVVLEOH WR VSHFLI\ PXOWLSOH VQDSVKRW UHODWLRQVKLSV WKDWDUHF\FOLFDOLQQDWXUH)RUH[DPSOHJLYHQWKUHHVWRUDJHYROXPHV$%&LWLV SRVVLEOH WR VSHFLI\ WKH IROORZLQJ VQDSVKRW UHODWLRQVKLSV LQ FKURQRORJLFDO VHTXHQFH L $DVWKHVRXUFHDQG%DVWKHWDUJHW LL %DVWKHVRXUFHDQG&DVWKH WDUJHW LLL &DVWKHVRXUFHDQG$DVWKHWDUJHW • 'LVWULEXWHG5HODWLRQVKLSV 7KH,QVWDQW,PDJHDOJRULWKPKDVHIILFLHQWVXSSRUWIRU GLVWULEXWHG UHODWLRQVKLSV ,Q SDUWLFXODU JLYHQ VWRUDJH YROXPHV LQYROYHG LQ VQDSVKRW UHODWLRQVKLSV QR UHVWULFWLRQ RQ HLWKHU WKH QDWXUH RU QXPEHU RYHU Q VWRUDJH VXEV\VWHPV WKH DOJRULWKP HQVXUHV WKDW UHDGV DQG ZULWHV WR VWRUDJH YROXPHVLQYROYHRQO\2 VWRUDJHVXEV\VWHPVSHUUHDGRUZULWH7KLVPLQLPL]HV WKHQXPEHURIUHTXHVWUHGLUHFWLRQVDQGGDWDFRSLHVWKDWKDYHWREHSHUIRUPHGLQ WKHVWRUDJHV\VWHP )LQDOO\WKHUHLVQROLPLWWRWKHQXPEHURIFRQFXUUHQWVQDSVKRWUHODWLRQVKLSVDVORQJDV WKHUH DUH HQRXJK SK\VLFDO UHVRXUFHV LH PHPRU\ GLVN VSDFH WR VXSSRUW WKH UHODWLRQVKLSV
$OJRULWKP 7KH,QVWDQW,PDJHDOJRULWKPDVVXPHVDQDUUD\RIVWRUDJHYROXPHVLQDVWRUDJHV\VWHP FRQVLVWLQJ RI Q VWRUDJH VXEV\VWHPV ZLWK QR UHVWULFWLRQV RQ WKH GLVWULEXWLRQV RI WKH VWRUDJHYROXPHVRYHUWKHVWRUDJHVXEV\VWHPV7KHVWRUDJHYROXPHVPD\EHSDUWRIDQ\ QXPEHURISUHYLRXVVQDSVKRWUHODWLRQVKLSV7KH,QVWDQW,PDJHRSHUDWLRQLVGLUHFWHGWR DVRXUFHGDWDYROXPHVDQGDWDUJHWGDWDYROXPHWDWWLPH7 ,Q WKH UHPDLQGHU RI WKH VHFWLRQ WKH PHWDGDWD UHTXLUHG IRU WKH ,QVWDQW ,PDJH LV GHVFULEHG 0HWDGDWD (DFK VWRUDJH YROXPH KDV PHWDGDWD DVVRFLDWHG ZLWK WKH YROXPH WR DVVLVW LQ WKH LPSOHPHQWDWLRQ RI WKH ,QVWDQW ,PDJH DOJRULWKP (DFK VWRUDJH YROXPH LV ORJLFDOO\ GLYLGHGLQWRPHTXDOVL]HGVHJPHQWVZKHUHPLVSURSRUWLRQDOWRWKHVL]HRIWKHVWRUDJH YROXPH7KHVL]HRIDVHJPHQWLVIL[HGE\WKHGHVLJQHURIWKH,QVWDQW,PDJHDOJRULWKP DQGDVVXPHGWREHFRQVWDQWDFURVVWKHVWRUDJHV\VWHP,WPXVWEHQRWHGKHUHWKDWWKH DOJRULWKPLVLQGHSHQGHQWRIWKHFKRVHQVHJPHQWVL]H )RUHYHU\VWRUDJHYROXPHWKHUHDUHWZRPDLQPHWDGDWDVWUXFWXUHVDVVRFLDWHGZLWKWKH YROXPH)RUDVWRUDJHYROXPHLWKHVHDUH6RXUFH L >P@DQG7DUJHW L >P@ZKHUH PDVGHILQHGEHIRUHLVPD[LPXPQXPEHURIVHJPHQWVLQWKHYROXPH7KHPHDQLQJRI WKHVWUXFWXUHVLVDVIROORZV
y 6RXUFH L >M@ N LPSOLHV WKDW WKH MWK VHJPHQW RI VWRUDJH YROXPH L LV SK\VLFDOO\ ORFDWHG LQ WKH MWK VHJPHQW RI VWRUDJH YROXPH N 7KXV WKH VRXUFH RI D VWRUDJH
3UDVHQMLW6DUNDU
YROXPHVHJPHQWSURYLGHVDSRLQWHUWRWKHDFWXDOSK\VLFDOORFDWLRQRIWKHVHJPHQW ,IN LLWPHDQVWKDWWKHVHJPHQWLVORFDWHGZLWKLQWKHYROXPHLWVHOI
y 7DUJHW L >M@ ^NV`PHDQVWKDWIRUHYHU\VWRUDJHYROXPHNVLQWKHVHWGHILQHGE\WKH
7DUJHWILHOGWKHMWKVHJPHQWRINVLVORFDWHGLQWKHMWKVHJPHQWRIVWRUDJHYROXPH L 7KLV PHDQV WKDW ZKHQ D VWRUDJH YROXPH VHJPHQW LV ZULWWHQ WKH RULJLQDO FRQWHQWVRIWKHVHJPHQWPXVWEHZULWWHQWRWKHORFDWLRQVLQGLFDWHGE\WKH7DUJHW ILHOG,IWKH7DUJHWILHOGVHWLVHPSW\LWLQGLFDWHVWKDWWKHRULJLQDOFRQWHQWVRIWKH VHJPHQWFDQEHGLVFDUGHG
:KHQ D VWRUDJH YROXPH L LV FUHDWHG WKHQ IRU HDFK VHJPHQWV M LQ WKH YROXPH 6RXUFH L >M@ LDQG7DUJHW L >M@ ^` 7KHUHODWLRQEHWZHHQWKH7DUJHWDQG6RXUFHPHWDGDWDVWUXFWXUHV LVQRWV\PPHWULF,Q RWKHUZRUGVLI6RXUFH L >M@ NLWGRHVQRWQHFHVVDULO\PHDQWKDWLLVLQ7DUJHW N >M@ )RUDOJRULWKPLFQHHGVDWKLUGDX[LOLDU\PHWDGDWDVWUXFWXUHWRPLUURUWKH7DUJHWILHOGLV GHILQHG 7KLV PHWDGDWD VWUXFWXUH LV GHILQHG DV 3UHYLRXV L >P@ ZKHUH P LV WKH PD[LPXPQXPEHURIVHJPHQWVLQVWRUDJHYROXPHL7KHPHDQLQJRIWKLVVWUXFWXUHLV VLPSO\ 3UHYLRXV L >M@ N !LLVLQ7DUJHW N >M@ )LQDOO\HYHU\VWRUDJH VXEV\VWHP NHHSV D WDEOH WKDW PDSV D JLYHQ VWRUDJH YROXPH WR WKHFRUUHVSRQGLQJVWRUDJHVXEV\VWHPZKHUHWKHYROXPHLVORFDWHG $OJRULWKP (VWDEOLVKPHQW:KHQDQ,QVWDQW,PDJHRSHUDWLRQLVLQYRNHGZLWKVDVWKHVRXUFHGDWD YROXPH DQG W DV WKH WDUJHW GDWD YROXPH D VQDSVKRW UHODWLRQVKLS LV VDLG WR EH HVWDEOLVKHG EHWZHHQ V DQG W 7KHUH DUH PDQ\ FKRLFHV LQ FUHDWLQJ D VQDSVKRW UHODWLRQVKLS )RU H[DPSOH DV LV VKRZQ EHORZ VRPH FKRLFHV GHJUDGH UHDG SHUIRUPDQFH ZKLOH RWKHUV GHJUDGH ZULWH SHUIRUPDQFH 7KH ,QVWDQW ,PDJH DOJRULWKP HVWDEOLVKHV VQDSVKRW UHODWLRQVKLSV LQ D PDQQHU VXFK WKDW ERWK UHDG DQG ZULWH SHUIRUPDQFHLVRSWLPL]HG 7KH ILUVW SDUW RI WKH DOJRULWKP XSGDWHV WKH 6RXUFH PHWDGDWD VWUXFWXUH IRU WKH WDUJHW VWRUDJH YROXPH IRU WKH VQDSVKRW 2QH DOWHUQDWLYH LV WR VHW 6RXUFH W >M@ V IRU HDFK VHJPHQWMLQWVXFKWKDWUHTXHVWVIRUGDWDLQWJHWUHGLUHFWHGWRV7KHGLVDGYDQWDJHRI WKH VFKHPH LV WKDW LW GHJUDGHV UHDG SHUIRUPDQFH LQ VLWXDWLRQV ZKHUH D VHJPHQW UHDG UHTXHVW WR W JHWV UHGLUHFWHG PXOWLSOH WLPHV RYHU WKH QHWZRUN EHIRUH UHDFKLQJ WKH VWRUDJHYROXPHZKHUHWKHVHJPHQWLVSK\VLFDOO\ORFDWHG ,Q WKH ,QVWDQW ,PDJH DOJRULWKP WKH 6RXUFH PHWDGDWD VWUXFWXUH LQ WKH WDUJHW VWRUDJH YROXPH LV XSGDWHG VXFK WKDW 6RXUFH W >M@ SRLQWV WR WKH VWRUDJH YROXPH ZKLFK DFWXDO ORFDWLRQ RI WKH GDWD VR WKDW DW PRVW RQH UHGLUHFWLRQ LV UHTXLUHG 7KXV LQ DQ HQYLURQPHQW ZLWK PXOWLSOH VQDSVKRW UHODWLRQVKLSV LQYROYLQJ Q VWRUDJH VXEV\VWHPV D UHDG IRU D VHJPHQW ZRXOG LQYROYH DW PRVW 2 VWRUDJH VXEV\VWHPV ,Q FRQWUDVW WKH DOWHUQDWLYHDSSURDFKFRXOGLQYROYH2 Q VWRUDJHVXEV\VWHPVLQDVLPLODUVLWXDWLRQ &RQVHTXHQWO\IRUDOOVHJPHQWVMLQWKHWDUJHWGDWDYROXPHWZHVHW
6RXUFH W >M@ 6RXUFH V >M@
$
,QVWDQW,PDJH 7UDQVLWLYHDQG&\FOLFDO6QDSVKRWVLQ'LVWULEXWHG6WRUDJH9ROXPHV
7KHVHFRQGSDUWRIWKHHVWDEOLVKPHQWRSHUDWLRQXSGDWHVWKH7DUJHWPHWDGDWDVWUXFWXUH 2QHDOWHUQDWLYHLVWRPLUURUWKH6RXUFHPHWDGDWDVWUXFWXUHXSGDWHLQWKHSUHYLRXVVWHS +HUH LI DIWHU WKH 6RXUFH PHWDGDWD VWUXFWXUH XSGDWH 6RXUFH W >M@ X IRU D SDUWLFXODU VHJPHQWMWKHQZHZRXOGDGGWWRWKHVHWGHILQHGE\7DUJHW X >M@7KHGLVDGYDQWDJHRI WKLV PHFKDQLVP LV WKDW WKH FDUGLQDOLW\ RI WKH VHW GHILQHG E\ WKH 7DUJHW ILHOG IRU D VHJPHQWFDQEHKLJK&RQVHTXHQWO\LWDIIHFWV ZULWHSHUIRUPDQFHLQVLWXDWLRQV ZKHUH WKHRULJLQDOGDWDIRUDVHJPHQWPXVWEHZULWWHQRYHUWKHQHWZRUNWRPXOWLSOHVWRUDJH YROXPHVEHIRUHDZULWHIRUWKHVHJPHQWLVDOORZHGWRSURFHHG ,QWKH,QVWDQW,PDJHDOJRULWKPWKHFDUGLQDOLW\RIWKHVHWGHILQHGE\WKH7DUJHWILHOGLV UHVWULFWHGWRRQHVRDVWRLPSURYHZULWHSHUIRUPDQFH7KHDLPLVWRPDNHVXUHWKDWWKH RULJLQDO VHJPHQW QHHGV WR EH ZULWWHQ RQO\ RQFH EHIRUH WKH ZULWH IRU WKH VHJPHQW LV DOORZHGWRSURFHHG7KXVLQVWHDGRIDGGLQJDVWRUDJHYROXPHWRWKHVHWGHILQHGE\WKH 7DUJHWILHOGIRUDVHJPHQWZHFUHDWHDOLQHDUFKDLQRIVWRUDJHYROXPHVZKHUHWKHVHW GHILQHG E\ WKH 7DUJHW ILHOG LQ D VWRUDJH YROXPH FRQWDLQV RQO\ WKH LGHQWLW\ RI WKH VXFFHHGLQJVWRUDJHYROXPHLQWKHFKDLQ)LQDOO\WKH 3UHYLRXV PHWDGDWD ILHOG IRU WKH VHJPHQWLVDOVRXSGDWHGWRUHIOHFWWKHFKDQJHWRWKH7DUJHWILHOG7KLVLVDFKLHYHGDV IROORZV YDUU V ZKLOH 7DUJHW U >M@ ^` U VLQJOHWRQHOHPHQWRI7DUJHW U >M@ $GGWWR7DUJHW U >M@ 3UHYLRXV W >M@ U %
2QFHWKHOLQHDUFKDLQLVHVWDEOLVKHGDZULWHWRDVWRUDJHYROXPHVHJPHQWLVDOORZHGWR SURFHHGLPPHGLDWHO\DIWHUWKHRULJLQDOFRQWHQWVRIWKHVHJPHQW KDYHEHHQ ZULWWHQWR WKHVXFFHHGLQJVWRUDJHYROXPHLQWKHFKDLQ7KLVPHDQVWKDWWKHZULWHWRWKHRULJLQDO VWRUDJH YROXPH VHJPHQW LV GHOD\HG E\ DW PRVW RQH RWKHU ZULWH WR WKH VXFFHHGLQJ VWRUDJH YROXPH 7KH VXFFHHGLQJ VWRUDJH YROXPH WKHQ ZULWHV WKH RULJLQDO GDWD WR LWV VXFFHVVRU ZLWKRXWKROGLQJ XSWKHRULJLQDO ZULWH WR WKH VWRUDJH YROXPH VHJPHQW 7KH VHTXHQFH RI ZULWLQJ WR WKH VXFFHVVRU FRQWLQXHV WLOO WKH HQG RI WKH FKDLQ LV UHDFKHG 7KXV LQ DQ HQYLURQPHQW RI Q VWRUDJH VXEV\VWHPV D ZULWH WR D VHJPHQW LQYROYHV DW PRVW2 VWRUDJHVXEV\VWHPV,QFRQWUDVWWKHDOWHUQDWLYHFRXOGLQYROYH2 Q VWRUDJH VXEV\VWHPV LQ WKH ZRUVW FDVH ZKHUH D ZULWH WR D VWRUDJH YROXPH VHJPHQW ZRXOG EH GHOD\HGE\2 Q FRQVHFXWLYHZULWHV 2QHFRUROODU\DFWLRQWKDWLVGRQHGXULQJWKHFUHDWLRQRIWKHFKDLQRIVWRUDJHYROXPHV LV WKH GHWHFWLRQ RI F\FOHV LQ WKH FKDLQ 6XFK F\FOHV DUH HOLPLQDWHG E\ XQGRLQJ WKH XSGDWH WR WKH 7DUJHW DQG 3UHYLRXV ILHOGV WR JLYH SUHFHGHQFH WR H[LVWLQJ VQDSVKRW UHODWLRQVKLSV $VLVHYLGHQWIURPWKHDERYHWKH,QVWDQW,PDJHDOJRULWKPLQFUHDVHVWKHFRPSOH[LW\RI HVWDEOLVKPHQWWRRSWLPL]HUHDGDQGZULWHSHUIRUPDQFH7KLVLVMXVWLILHGEHFDXVHUHDGV DQG ZULWHV WR VWRUDJH YROXPHV DUH H[SHFWHG WR EH PRUH FRPPRQ WKDQ WKH HVWDEOLVKPHQWRSHUDWLRQV &RQVLVWHQF\ 7KH ,QVWDQW ,PDJH DOJRULWKP PXVW DOVR XSGDWH WKH 6RXUFH DQG 7DUJHW PHWDGDWDVWUXFWXUHVZKHQDVHJPHQWLVFRSLHGRYHUIURPDVRXUFHVWRUDJH YROXPHWR WDUJHW VWRUDJH YROXPH RU ZKHQ D VHJPHQW LV ZULWWHQ WR LQ D WDUJHW VWRUDJH YROXPH
3UDVHQMLW6DUNDU
7KLVXSGDWHLVQHFHVVDU\WRPDLQWDLQWKHVHPDQWLFVRIDVQDSVKRWUHODWLRQVKLS GHILQHG LQ6HFWLRQ $VVXPHWKDWDQDFFHVVWRDVWRUDJHYROXPHFDXVHVHLWKHUDVHJPHQWMWREHFRSLHGRYHU WRVWRUDJHYROXPHWRUEHRYHUZULWWHQLQVWRUDJHYROXPHW)LUVWWKH6RXUFHPHWDGDWD VWUXFWXUHLVXSGDWHGLQWWRLQGLFDWHWKDWWKHVHJPHQWMLVORFDWHGLQWKHYROXPHLWVHOI 6RXUFH W >M@ W & 7KH QH[W VWHS LV WR XSGDWH WKH 7DUJHW DQG 3UHYLRXV PHWDGDWD VWUXFWXUHV WR UHPRYH W IURPWKHOLQHDUFKDLQRIVWRUDJHYROXPHV YDUS 3UHYLRXV W >M@J 7DUJHW W >M@ 3UHYLRXV J >M@ 3UHYLRXV W >M@ 7DUJHW S >M@ 7DUJHW W >M@ ' $SURWRW\SHRIWKH,QVWDQW,PDJHDOJRULWKP ZDV LPSOHPHQWHG LQ DQ ,%0 6HUY5$,' ,,,VWRUDJHFRQWUROOHUZLWKD0+]*&;SURFHVVRUDQG0%RIPHPRU\
5HODWHG:RUN 6WRUDJH V\VWHP VQDSVKRWV DUH ³GXPE´ VQDSVKRWV RI D VWRUDJH YROXPH LQ WKDW WKH FRQWHQWV RI WKH VWRUDJH YROXPH DUH QRW LQWHUSUHWHG>@ ,Q FRQWUDVW WR VWRUDJH V\VWHP VQDSVKRWV ILOH V\VWHP DQG GDWDEDVH VQDSVKRWV DUH PRUH LQWHOOLJHQW DQG LQWHUSUHW WKH FRQWHQWV RI WKH VWRUDJH V\VWHP WR DOORZ ILQHUJUDLQHG VQDSVKRWV ([DPSOHV RI ILOH V\VWHPV ZLWK JRRG VQDSVKRW IHDWXUHV DUH WKH (FKR ILOH V\VWHP>@ DQG WKH :$)/ V\VWHP>@ &KHUYHQDN HW DO SURYLGH D JRRG VXUYH\ RI OLWHUDWXUH SHUWDLQLQJ WR WKLV ILHOG>@'DWDEDVHVDOVRSURYLGHVQDSVKRWIHDWXUHVIRUXVHUVWRTXHU\YDULRXVHSRFKVRI WKH VDPH GDWDEDVH DQG XVH YDULRXV VWRUDJH DQG DOJRULWKPLF WHFKQLTXHV WR RSWLPL]H TXHU\SURFHVVLQJWLPHDJDLQVWWKHVHVQDSVKRWV> @+RZHYHUDFRPPRQIHDWXUHWKDW VHSDUDWHV WKH VQDSVKRW VWUDWHJ\ SUHVHQWHG KHUH LV WKDW WKH ,QVWDQW ,PDJH VQDSVKRW DOJRULWKP DOORZV QRUPDO DFFHVV LQFOXGLQJ UHDGV DQG ZULWHV WR VQDSVKRW YROXPHV DOORZLQJVQDSVKRWUHODWLRQVKLSVWKDWDUHERWKWUDQVLWLYHDQGF\FOLFDO
&RQFOXVLRQVDQG)XWXUH:RUN 7KLVSDSHUSUHVHQWVWKH,QVWDQW,PDJHDOJRULWKP IRUSURYLGLQJ VQDSVKRW VXSSRUW LQ D VWRUDJHHQYLURQPHQW7KH,QVWDQW,PDJHDOJRULWKPFDQQRWRQO\KDQGOHWUDQVLWLYHDQG F\FOLFDOVQDSVKRWUHODWLRQVKLSVEXWDOVRRSWLPL]HVUHDGDQGZULWHPHVVDJHWUDIILFLQD GLVWULEXWHG HQYLURQPHQW 7KLV ZDV DFKLHYHG E\ FRQVLGHULQJ YDULRXV DOWHUQDWLYHV WR HVWDEOLVK D VQDSVKRW UHODWLRQVKLS DQG WKHQ VHOHFWLQJ WKH EHVW DSSURDFK WKDW LQYROYHG RQO\2 VWRUDJHVXEV\VWHPVSHUUHDGRUZULWHLQDQHQYLURQPHQWRIVWRUDJHYROXPHV VSUHDGRYHUQVWRUDJHVXEV\VWHPV
,QVWDQW,PDJH 7UDQVLWLYHDQG&\FOLFDO6QDSVKRWVLQ'LVWULEXWHG6WRUDJH9ROXPHV
5HIHUHQFHV $GLED0/LQGVD\% 'DWDEDVH6QDSVKRWV3URFHHGLQJVRI9/'% $QGHUVRQ 7 'DKOLQ 0 1HHIH - 3DWWHUVRQ ' 5RVHOOL ' :DQJ 5 6HUYHUOHVV 1HWZRUN)LOH6\VWHPV$&072&69RO &KDQG\ .0 /DPSRUW / 'LVWULEXWHG 6QDSVKRWV 'HWHUPLQLQJ *OREDO 6WDWHV RI 'LVWULEXWHG6\VWHPV$&072&69RO &KHUYHQDN $ 9HODQNL 9 .XUPDV = 3URWHFWLQJ )LOH 6\VWHPV $ VXUYH\ RI EDFNXS WHFKQLTXHV3URFHHGLQJVRIWKH-RLQW1$6$,(((0DVV6WRUDJH&RQIHUHQFH &KXWDQL6$QGHUVRQ2.D]DU0/HYHUHWW%0DVRQ$6LGHERWKDP5 7KH(SLVRGH )LOH6\VWHP3URFHHGLQJVRIWKH:LQWHU8VHQL[7HFKQLFDO&RQIHUHQFH +LW]'/DX-0DOFRP0 )LOHV\VWHPGHVLJQIRUD)LOH6HUYHU$SSOLDQFH3URFHHGLQJV RI:LQWHU8VHQL[7HFKQLFDO&RQIHUHQFH +XWFKLQVRQ10DQOH\6)HGHUZLVFK0+DUULV*+LW]'.OHLQPDQ62¶0DOOH\ 6 3URFHHGLQJVRI26',6\PSRVLXP /HH(7KHNNDWK& 3HWDO 'LVWULEXWHG9LUWXDO'LVNV3URFHHGLQJVRI$63/269 /LQGVD\%+DDV/0RKDQ&3LUDKHVK+:LOPV3 $6QDSVKRW'LIIHUHQWLDO5HIHUHVK $OJRULWKP3URFHHGLQJVRI6,*02' 6DWR < ,QRXH 0 0DVX]DZD 7 )XMLZDUD + $ 6QDSVKRW $OJRULWKP IRU 'LVWULEXWHG 0RELOH6\VWHPV3URFHHGLQJVRI,&'&6 6SH]LDOHWWL0.HDUQV3 (IILFLHQW'LVWULEXWHG 6QDSVKRWV3URFHHGLQJV RI ,&'&6
9HQNDWHVDQ 6 0HVVDJHRSWLPDO ,QFUHPHQWDO 6QDSVKRWV 3URFHHGLQJV RI ,&'&6
Scheduling Queries for Tape-Resident Data Sachin More and Alok Choudhary Department of Electrical and Computer Engineering Northwestern University
Abstract. Tertiary storage systems are used when secondary storage can not satisfy the data storage requirements and/or it is a more cost effective option. The new application domains require on-demand retrieval of data from these devices. This paper investigates issues in optimizing I/O time for a query whose data resides on automated tertiary storage containing multiple storage devices.
1
Introduction
Tertiary storage systems are employed in cases where secondary storage can not satisfy the data storage requirements or tertiary storage is a more cost effective option [8]. NASA’s Earth Observing System (EOS) Data and Information System (EOSDIS) is an example of the former [4, 13]. The latter case, can be found in data warehousing applications. Inmon [10] shows that substantial monetary savings can be achieved using a hierarchical data storage containing comparatively small amount of secondary storage and vast amounts of tertiary storage without sacrificing performance. Tertiary storage devices have traditionally been used as archival storage. The new application domains require on-demand retrieval of data from these devices [9]. While data archiving applications access large chunks of contiguous data, these new applications access data that is scattered on multiple media. Hence correct scheduling of data retrieval requests becomes important. For example, I/O time for a query that accesses data from two different media using a single robotic arm and two tape drives is minimized when the tape that needs more time to read is loaded first. Many of the application domains that use tertiary storage access multidimensional datasets [2]. In a multidimensional dataset, each data item occupies a unique position in a n-dimensional hyper-space. A query selects a subset of the data items by selecting a subset of the domain in each dimension. Given the wide variety of expected queries [18], it is not possible to store data accessed by each query contiguously without high amount of data replication. Hence, a query on a multidimensional dataset stored on tertiary storage, accesses data from multiple media [15]. Time needed to read the query data from a media depends on the amount of data and location of the data inside the media. When using
This work is supported by DOE ASCI Alliance program under a contract from Lawrence Livermore National Labs B347875.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1292–1301, 2000. c Springer-Verlag Berlin Heidelberg 2000
Scheduling Queries for Tape-Resident Data
1293
automated tertiary storage, the total I/O time for the query is also influenced by the order in which media are accessed. Various issues in tertiary storage management have been addressed by the database community. Carey et. al. [1] evaluate issues in extending database technology for storing/accessing data on tertiary storage. Stonebraker [23] proposes a database architecture that uses hierarchical storage. Livny et. al. [17] and Sarawagi [20, 21, 22] examine issues in query processing when data resides on tertiary storage. Data striping on tertiary storage has been evaluated in [3, 5]. Tertiary storage space organization issues are addressed in [2, 6]. This paper investigates issues in optimizing I/O time for a query whose data resides on automated tertiary storage containing multiple storage devices. We model the problem as a limited storage parallel two-machine flow-shop scheduling problem with additional constraints. Given a query, we establish an upper bound on the number of storage devices for an optimal I/O schedule. For queries that access small amounts of data from multiple media, we derive an optimal schedule analytically. For queries that access large amount of data we derive a heuristics-based scheduling algorithm using analytically proven results. The rest of this paper is organized as follows. Section 2 introduces the problem. Sections 3, 4 and 5 analyze the problem and provide theoretical results for the problem. Section 6 discusses important practical considerations and presents performance evaluation of our approach. Conclusions are presented in Section 7.
2
Background
The system model consists of A ≥ 1 robotic arms and T > 1 tape drives. A query needs data from n tapes. Reading data from a tape consists of the following set of operations: rewinding currently loaded tape; ejecting the tape; putting it back; fetching the tape to be read; loading the tape; searching and reading data inside the tape. Putting back a tape and fetching a new one are handled by the robotic arm and the rest of the actions are carried out by the tape drive. The time to do the arm operations is denoted by tA (which we assume to be same for all tapes [6]) and time to do the drive operations is denoted by tD . Given a set of tapes and blocks from each tape that need to be accessed by a query, find the order in which the tapes should be read to minimize the total I/O time. The problem is cast as a special case of two-machine flow-shop scheduling problem [19]. The arm operations denote the first machine and the drive operations denote the second machine. There are n jobs to be scheduled. The optimality criteria is makespan, total execution time of the schedule. The distinctive features of our problem (in contrast to the traditional two-machine flow-shop scheduling problem) are: – Multiple instances of machines More common system configurations have a single robotic arms servicing a number of tape drives. In this paper, we consider the case where there is one instance of the first machine and multiple instances of the second machine.
1294
Sachin More and Alok Choudhary
– Buffer bound = T At the most T jobs can be in the shop simultaneously. The robotic arm can not load more tapes while all drives are busy accessing tapes loaded in them and must remain idle. – If job i starts at time si on the first machine then it must be scheduled so that it finishes by si + tA + tDi on the second machine. The second machine is idle for time tA before a job can be scheduled on it. This accommodates for the behavior of a tertiary storage system where a drive is empty while the robotic arm is loading the next tape. The case where A = T = 1 (a single tape drive serviced by a single robotic arm) is uninteresting under this condition. In the rest of this paper we assume T > 1. – Practical considerations prevent use of scheduling algorithms that compare tA and tDi values. The value of tDi can not be predicted correctly unless a very accurate analytical model of the tape drive is available. For example, Johnson’s algorithm [12], which is optimal for traditional two-machine flowshop scheduling, performs such a comparison.
3
Workload Characterization
A job is characterized by a k value of the job. For any job i, its k value is governed by the inequality ((k − 1) × tA ) < tDi ≤ (k × tA ). The k value of a workload is defined as ((k − 1) × tA ) < maxi (tDi ) ≤ (k × tA ) where 0 ≤ i < n. Theorem 1. [14] For A = 1, if (k − 1) × tA < maxi (tDi ) ≤ k × tA , then increasing number of instances of the second machine beyond min(n, k + 1) does not improve the makespan of any schedule. The above result provides an interesting insight to the problem. Given a workload, it tells us when the first machine is the bottleneck and when it is not. Given the system configuration, makespans of workloads with k values less than or equal to T will be constrained by the first machine, that is idle times can be introduced on the second machine because the first machine is always busy. The jobs in these kind of workloads have their execution time on the second machine bounded above by T × tA . Jobs in these workloads are small jobs. Execution time on the second machine for a large job is more than T × tA .
4
Workloads Consisting of Small Jobs
A workload containing small jobs represents a situation where after loading a media in a drive, the drive finishes reading data off that media before the robotic arm can finish loading media in other drives. In such a situation, we find that the robotic arm is busy all the time (except at the end when there are no more media to access) irrespective of the order in which the media are loaded. Theorem 2. [14] ∀i, if tDi ≤ (T − 1) × tA and A = 1, then longest-tD -first (LtF) schedule is optimal.
Scheduling Queries for Tape-Resident Data
5
1295
Workloads Consisting of Big Jobs
For queries that access large amount of data from each tape (where tDi > (T − 1) × tA ), the first machine is not a bottleneck. This leads one to believe that eliminating idle times on the second machine will lead to an optimal schedule. Proposition 3. [14] ∀i, if tDi > (T − 1) × tA and A = 1, then shortest-tD -first (StF) schedule is optimal in terms of idle time for the second machine. However, StF schedule does not necessarily produce the optimal schedule. Apart from idle times of machines, the length of the head and tail of the schedule determine the optimality of a schedule. For the problem under consideration, the length of the head is independent of the scheduling algorithm. The StF schedule puts the job with largest second machine time last. This results in bigger tail, producing a suboptimal makespan. Schedules generated by longest-tD -first (LtF) algorithm on the other hand can produce idle time on the second machine but are successful in reducing the length of the tail. This is because the LtF algorithm puts smallest jobs at the end of the schedule. [14] shows that when number of jobs is less than or equal to number of instances of the second machine LtF produces optimal makespan. But LtF is not necessarily optimal when number of job exceeds number of instances of the second machine. Proposition 4. [14] ∀i, if tDi > (T − 1) × tA and A = 1 and n > T , then longest-tD -first (LtF) schedule can be suboptimal. We propose a new heuristic that combines properties of StF and LtF: 1. Sort the jobs using StF strategy. 2. Pick the last T (number of instances of the second machine) jobs and reverse their order. If there are n jobs, the last T jobs are numbered n − T, n − T + 1, . . . , n − 2, n − 1 at the end of previous step. And tDn−T ≤ tDn−T +1 ≤ . . . ≤ tDn−2 ≤ tDn−1 . We reverse their order so that tDn−T ≥ tDn−T +1 ≥ . . . ≥ tDn−2 ≥ tDn−1 3. Repeat the above step for jobs n − 2T, n − 2T + 1, . . . , n − T − 2, n − T − 1. Keep repeating step 3 moving towards the start of the schedule. Below is the illustration explaining working of our heuristic algorithm: Jobs a b c d e f g h i j k l m n tD 6 10 7 13 8 5 4 1 9 2 0 12 11 3 Applying StF (Step 1) k h j n g f a c e i b m l d Reversing order of the k h j n g f a c e i d l m b last 4 jobs (Step 2) Repeated application k h j n g f i e c a d l m b of step 3 k h f g j n i e c a d l m b h k f g j n i ecad l mb
1296
Sachin More and Alok Choudhary
The workload consists of 14 jobs (a, b, . . . , n). The configuration of the flowshop is A = 1, T = 4. The jobs are first sorted using StF algorithm. Then the order of the last 4 jobs is reversed. The algorithm then works its way towards the start of the schedule, reversing orders of 4 consecutive jobs. In the final step, only two jobs remain, jobs h and k. Their order is reversed too.
6
Performance Evaluation
So far we assumed that the values of tA and tDi for each job (tape to be read) are known. In general, its hard if not impossible to calculate tDi accurately given the set of blocks on the tape that are to be read, since it requires accurate modeling of the tape drive(s). On the other hand, tape drive manufacturers do provide peak/average search (seek) rate and peak/average read rate. These values can . be used to estimate tDi . The estimated value of tDi is denoted by testimated Di estimated < t should hold. We evaluated three Ideally, if tDj < tDk then testimated Dj Dk different schemes to compute testimated : Di 1. Maximum Offset Estimate For each tape find the offset of the farthest block = maximum offset. to be read inside that tape. For each tape i, testimated Di This value is approximately proportional to the time it will take to rewind this tape under the tape drive model we use. 2. Data Volume Estimate For each tape i, testimated = number of blocks read Di from the tape. This value is approximately proportional to the time it will take to read the blocks from this tape. 3. Full Estimate This estimation method combines the above two estimation = seek rate×maximum offset+read rate×blocks read+ methods. testimated Di seek rate × (maximum offset − blocks read). Our experiments revealed that data volume estimates and full estimates help scheduling algorithms perform better than using maximum offset estimates. We also found that scheduling algorithms perform equally well whether data volume estimates are used or full estimates are used. This is because read times dominate seek times for the workloads we considered. We use data volume estimates for all scheduling algorithms since it has lower computing requirements. We use a tape library simulator to execute the schedules created by various scheduling algorithm. Most of the literature [3, 17, 21, 22] uses a linear approximation of the locate time for tape drives. [7] found that such linear approximation is inaccurate. We use the analytical models of Exabyte’s EXB-8505XL tape drive and EXB-210 tape library described in [6] in our tape library simulator. We use the SORT algorithm described in [9] for I/O scheduling when fetching data from the same tape. Random Workload Fig. 1 shows the performance of various scheduling algorithms over a set of randomly generated workloads. The set contains 1000
Scheduling Queries for Tape-Resident Data 140
UNOPTIMIZED LtF StF FoldLtF Heu
135 130
% of OPT
1297
125 120 115 110 105 2
4
8
16 Number of drives
32
Fig. 1. Performance of various scheduling algorithms for random workloads.
workloads1. For each workload, we determine how many blocks to read from each tape by generating a random number2 between 0 and the total number of blocks on the tape. Then for each tape, we generate N distinct block numbers randomly, where N is the number of blocks to be read from this tape. The UNOPTIMIZED algorithm loads the tapes in random order. The FoldLtF algorithm is a heuristic proposed in [16] for job scheduling in a limited floor-space flow-shop environment. The algorithm first generates a list using LtF algorithm and then schedules jobs from both the ends of the list. The figure plots the makespan of each scheduling algorithm as percent of an OPT value. The OPT value is a lower bound on the makespan of the optimal schedule. The OPT value is then sum of the times to access each tape divided by the total number of drives in the system. Note that we use the same set of workloads for all the data points in the figure. Hence the OPT value is inversely proportional to the number of drives in the system. We find that performance of the scheduling algorithms is not within a constant factor of OPT (for the expected range of value of number of drives in the system), its a function of the number of drives in the system. Since LtF always outperforms StF, we conclude that the length of the tail of the schedule is more important than amount of idle time in the schedule for reducing makespan, The FoldLtF algorithm performs only slightly better than the UNOPTIMIZED case that too when number of drives in the system is comparatively higher. Our heuristic based algorithm always performs well due to a careful balance between idle times and length of tail achieved by our algorithm. Performance of LtF algorithm approaches performance of our algorithm when number of drives in the system is very low or very high. The reason LtF (and the UNOPTIMIZED case too) perform on par with our algorithm when number of drives is small is because there is very little scope for optimization in that case due to limited choice available to scheduling algorithms when fewer tape 1 2
We found that varying number of instances of the workload does not change the results qualitatively. We use an inversive congruential generator for generating random numbers.
1298
Sachin More and Alok Choudhary
drives are used. As we saw earlier, LtF is optimal for large jobs when number of instances of the second machine are equal to or greater than number of jobs. When the number of drives is high, number of tapes to be loaded is close to/equal to number of drives available, making LtF optimal. But our algorithm clearly outperforms LtF when number of drives in the system is moderate (between 4 and 16). Note that this range of values of number of drives, is commonly found in a typical data management system handling large amounts of data. Since the performance of UNOPTIMIZED, StF and FoldLtF is considerably worse than LtF and our heuristic algorithm, we did not consider these algorithms for further performance studies. Experimental Verification Using Sequoia 2000 Storage Benchmark We use the national dataset from the benchmark over a period of four years (200 weeks) and is about 64GB in size. The schema for the tables used in these queries is: create RASTER(location=box, time=int4, band=int4, data=int2[][]); time is a four byte integer and denotes the half-month over which the raster image was captured. The location attribute is the bounding box for the raster data. band is the wavelength band at which the data was captured. data is a two dimensional array of size 10240X6400 of two byte integers at a spatial resolution of 0.5kmX0.5km. All the raster images are stored chronologically sorted, since they were captured such. Raster images for a half-month are not sorted in any particular order. A query type represents an access pattern on the dataset. A query is an instance of a query type. In general multiple access patterns are observed on a typical dataset. Access patterns are executed with different frequencies [2, 11]. In order to capture this phenomenon, we first define variety of access patterns (query types) on the dataset. Then we create different query mixes using these query types by manipulating number of different queries for each query type in the mix. We use two types of queries. Query Type 1 selects all images belonging to a band. The data of interest is spread over the entire set of tapes that store the dataset. Query Type 2 selects all images belonging to a half-month. The data of interest is localized in a few tapes of the set of tapes that store the dataset. A query mix is generated using two parameters: Number of queries denotes the total number of queries that this mix will consist of. Query type percentages represent the percentage of query instances that belong to each query type. The number of queries determines the accuracy of the query mix generation process. For all query types, if the number of distinct query instances that belong to a query type is n and the query type percentage is p, the mix should contain at least np queries. This assures that the expected number of occurrences for any query instance for a query type is at least 1. We evaluate two different query mixes: Query mix 1 consists of majority (90%) of queries from query type 1. Query mix 2 has equal mix of queries from query type 1 and query type 2.
Heuristic LtF
100
Makespan (% of unoptimized schedule)
Makespan (% of unoptimized schedule)
Scheduling Queries for Tape-Resident Data
90
80
1 2
4
8
16 Number of drives
(a) Query mix 1
32
1299
Heuristic LtF
100
90
80
1 2
4
8
16 Number of drives
32
(b) Query mix 2
Fig. 2. Experimental results for Sequoia 2000 storage benchmark
Fig. 2 shows the performance results of our heuristic algorithm against LtF algorithm. The time taken to execute a query mix by an algorithm is plotted as a percent of the time taken by a naive scheme that does not do any scheduling. The results show that our algorithm performs consistently well. Note that for 16 and 32 drives case, the number of tapes from which data is read for a query is less than the number of drives in the system. Since LtF has already been proved to be optimal in that case, LtF performs equally well when compared to our algorithm. When number of drives in the system is moderate, our algorithm clearly outperforms LtF. The gains in performance are due to a balanced optimization of both drive idle times and the size of the tail of the schedule.
7
Conclusions
This paper investigated issues in optimizing I/O time for a query whose data resides on automated tertiary storage containing multiple storage devices. We modeled the problem as a limited storage parallel two-machine flow-shop scheduling problem with additional constraints. The paper presented analytical results that provide insight to the problem. We presented a heuristic algorithm for scheduling data from a tape library. Our performance results show impressive gains for synthetic as well as real workloads.
References [1] Carey, M. J., Haas, L. M., and Livny, M. Tapes hold data, too: Challenges of tuples on tertiary storage. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data (Washington, D.C., 1993), ACM Press, pp. 413–417.
1300
Sachin More and Alok Choudhary
[2] Chen, L. T., Drach, R., Keating, M., Louis, S., Rotem, D., and Shoshani, A. Efficient organization and access of multi-dimensional datasets on tertiary storage systems. Information Systems 20, 2 (April 1995), 155–183. [3] Drapeau, A. L., and Katz, R. H. Striping in large tape libraries. In Proceedings of the 1995 ACM/IEEE Supercomputing Conference (San Diego, CA, 1993), IEEE Computer Society Press. [4] Fox, S., Prasad, N., and Szezur, M. NASA’s EOSDIS: an integrated system for processing, archiving, and disseminating high-volume earth science imagery and associated products, July 1996. [5] Golubchik, L., Muntz, R. R., and Watson, R. W. Analysis of striping techniques in robotic storage libraries. In Proceedings of the Fourteenth IEEE Symposium on Mass Storage Systems (Monterey, CA, 1995), IEEE Computer Society Press, pp. 225–238. [6] Hillyer, B. K., Rastogi, R., and Silberschatz, A. Scheduling and data replication to improve tape jukebox performance. In Proceedings of the 15th International Conference on Data Engineering (Sydney, Australia, 1999), IEEE Computer Society Press, pp. 532–541. [7] Hillyer, B. K., and Silberschatz, A. On the modeling and performance characteristics of a serpentine tape drive. In Proceedings of 1996 ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems (Philadelphia, Pennsylvania, 1996), ACM Press, pp. 170–179. [8] Hillyer, B. K., and Silberschatz, A. Random I/O scheduling in online tertiary storage systems. In Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data (Montreal, Canada, 1996), ACM Press, pp. 195–204. [9] Hillyer, B. K., and Silberschatz, A. Scheduling non-contiguous tape retrievals. In Proceedings of Sixth NASA Goddard Conference on Mass Storage Systems and Technologies and Fifteenth IEEE Mass Storage Systems Symposium (University of Maryland, College Park, MD, March, 1998), IEEE Computer Society Press. [10] Inmon, B. The Role of Nearline Storage in the Data Warehouse: Extending your Growing Warehouse to Infinity. Technical white paper, 1999. Provided by StorageTek. http://billinmon.com/library/whiteprs/st nls.pdf. [11] Jagadish, H. V., Lakshmanan, L. V. S., and Srivastava, D. Snakes and sandwiches: Optimal clustering strategies for a data warehouse. In Proceedings ACM SIGMOD International Conference on Management of Data (Philadephia, Pennsylvania, 1999), ACM Press, pp. 37–48. [12] Johnson, S. M. Optimal two- and three-stage production schedules with setup times included. Naval Research Logistics Quarterly 1, 1 (March 1954), 61–68. [13] Kobler, B., Berbert, J., Caulk, P., and Hariharan, P. C. Architecture and design of storage and data management for the nasa earth observing system data and information system (eosdis). In Proceedings of the Fourteenth IEEE Symposium on Mass Storage Systems (Monterey, CA, 1995), IEEE Computer Society Press, pp. 65–76. [14] More, S., and Choudhary, A. Scheduling Queries on Taperesident Data. Tech. Rep. CPDC-TR-2000-01-001, Center for Parallel and Distributed Computing, Northwestern University, January 2000. http://www.ece.nwu.edu/cpdc/TechReport/1999/11/CPDC-TR-2000-01001.html.
Scheduling Queries for Tape-Resident Data
1301
[15] More, S., and Choudhary, A. Tertiary storage organization for large multidimensional datasets. In 8th NASA Goddard Space Flight Center Conference on Mass Storage Systems and Technologies and 17th IEEE Symposium on Mass Storage Systems (College Park, MD, March 2000), IEEE Computer Society Press, pp. 203–209. [16] More, S., Muthukrishnan, S., and Shriver, E. Efficiently sequencing taperesident jobs. In Proceedings of the Eighteenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (Philadelphia, Pennsylvania, June 1999), ACM Press, pp. 33–43. [17] Myllymaki, J., and Livny, M. Disk-tape joins: synchronizing disk and tape accesses. In Proceedings of ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems (Ottawa, Canada, 1995), ACM Press, pp. 279–290. [18] APB-1 OLAP Benchmark, Release II, November 1998. OLAP Council. [19] Pinedo, M. Scheduling Theory, Algorithms and Systems. Prentice-Hall, Englewood Cliffs, NJ, 1995. [20] Sarawagi, S. Database systems for efficient access to tertiary memory. In Proceedings of the Fourteenth IEEE Symposium on Mass Storage Systems (Monterey, CA, 1995), IEEE Computer Society Press, pp. 120–126. [21] Sarawagi, S. Query processing in tertiary memory databases. In Proceedings of 21th International Conference on Very Large Data Bases (Zurich, Switzerland, 1995), Morgan Kaufmann, pp. 585–596. [22] Sarawagi, S., and Stonebraker, M. Reordering query execution in tertiary memory databases. In Proceedings of 22th International Conference on Very Large Data Bases (Mumbai (Bombay), India, 1996), Morgan Kaufmann, pp. 156–167. [23] Stonebraker, M. Managing persistent objects in a multi-level storage. In Proceedings of the 1991 ACM SIGMOD International Conference on Management of Data (Denver, Colorado., 1991), ACM Press, pp. 2–11.
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays Ying Chen, Windsor W. Hsu, and Honesty C. Young IBM Almaden Research Center {ying, windsor, young}@almaden.ibm.com
Abstract. Parity-based disk arrays provide high reliability and high performance for read and large write accesses at low storage cost. However, small writes are notoriously slow due to the well-known read-modifywrite problem. This paper presents logging RAID, a disk array architecture that adopts data logging techniques to overcome the small-write problem in parity-based disk arrays. Logging RAID achieves high performance for a wide variety of I/O access patterns with very small disk space overhead. We show this through trace-driven simulations.
1
Introduction
Several key characteristics of a redundant array of independent disks (RAID), e.g., high I/O bandwidth, high system reliability, and low cost, have made them attractive to many different types of applications. Traditionally, RAID systems are classified into a number of RAID levels, each corresponding to a different arrangement of data and parity on disk. The common RAID levels include RAID level 0, 1, 3, 4, and 5 (“RAID-x” for short). [1] gives a detailed description of different RAID levels. We describe these RAID levels briefly here. RAID-0 stripes data across a set of disk drives in a round-robin fashion. The unit of striping is typically a multiple of disk sector size. This unit is commonly referred to as “stripe unit”. Each row of stripe units across all the drives in the array is called a “stripe”. RAID-0 allows for parallel disk accesses, but it is unreliable. RAID-1 deals with the reliability issue by duplicating each data block on two separate drives, i.e., “data mirroring”. RAID-1 tolerates any single drive failure, but it also doubles the storage capacity cost. RAID-3, 4, and 5 are parity-based RAID architectures, where redundant data information is stored in the form of parity, which is computed as an XOR of all the data it protects. Parity-based RAIDs provide single-drive failure protection at low storage capacity cost1 . They perform well for workloads with a large number of reads and/or large, stripe-aligned writes, i.e., writes that completely overwrite one or multiple RAID stripes, but they suffer the well-known readmodify-write problem for small writes. This is because for each small write, the parity-based RAID must read the old data block and the old parity information, 1
1 Only D th of the total storage is needed to store parity information. Here D is the number of disks in a RAID system excluding parity disk.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1302–1311, 2000. c Springer-Verlag Berlin Heidelberg 2000
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays
1303
compute the new parity, and write the new data and the parity blocks. Four disk I/O operations per small write is common in parity-base RAID systems. The problem does not exist for large, stripe-aligned writes, since the entire stripe will be overwritten, there is no need to read the old data and parity information. RAID-3, 4, and 5 have relatively small differences as highlighted in [1]. We do not discuss them due to space limitations. In this paper, we use RAID-5 to represent parity-based RAIDs. Our techniques are applicable to RAID-4 as well. The logging RAID architecture is designed to overcome the RAID-5 smallwrite problem while guaranteeing high performance for many other read and write access patterns. The cornerstone of this architecture is the data logging technique that has been used in several different types of systems [2,3,4]. Logging RAID bundles small writes into large RAID-5 stripes using a small Non-VolatileMemory (NVM) buffer. We flush the NVM buffer to the end of a small temporary log on disk based on a predetermined threshold. Writes to the log are always RAID-5 stripe-aligned, and hence efficient. Data blocks stored in the log are moved to their permanent disk locations later on during system idle time to ensure subsequent efficient read accesses. In the remainder of the paper, we begin with a description of the logging RAID scheme in Section 2. Section 3 presents a set of trace-driven simulation results. Due to space limitations, we do not discuss related work in this paper. Interested readers can find a relatively detailed description of the related work in [5]. Finally, we draw conclusions in Section 4.
2
The Logging RAID Architecture
To improve write performance, a number of systems have chosen to write data in the form of a log. However, if care is not taken, data logging can slow down reads, since the layout suitable for writes may not be appropriate for reads. For instance, the workloads generated by data-mining and decision support systems [6] often contain a sweep of random writes followed by large sequential reads/scans. If the randomly updated data blocks are grouped together on disk, sequential scans may perform poorly. Logging RAID exploits the data logging idea to optimize writes, while working hard to optimize reads. To optimize writes, logging RAID always writes data in large, RAID-5 stripe-aligned fashion. To optimize reads, an off-line data destaging and reorganization process is performed when the system is idle. In the rest of the discussion, we assume that the smallest unit addressable by the host system is one disk block or sector (512 bytes). 2.1
The Logging RAID Storage Layout
Conceptually, the disk space in logging RAID consists of a permanent storage area (“PA” for short) and a temporary log area (“LA” or “log” for short). The host system only sees the PA. Reads and writes are performed as if the LA does not exist. The LA is a small fraction of the total disk space in the disk array.
1304
Ying Chen, Windsor W. Hsu, and Honesty C. Young
Both the PA and the LA are organized as RAID-5 and may reside on the same or different sets of disks in a given array. They can also use different stripe units and different numbers and types of disks. The log serves as a “fast write cache” in logging RAID. Writes to the log are always appended to the end. For in-line small writes, logging RAID first accumulates them in a small NVM buffer, which contains one or more RAID-5 stripes in the log. Each RAID-5 stripe in the NVM buffer is called “a stripe buffer”. One or more stripe buffers are flushed to the log whenever the NVM buffer is full. So the unit of flushing is always one or more stripe buffers. This guarantees that writes to the LA are always RAID-5 stripe-aligned, hence fast. Normal writes can return as soon as the data is written to the NVM buffer, so the I/O response time is short. To simplify our discussion, we assume that the NVM buffer contains one RAID-5 stripe in the rest of the paper. Storing data in this log-structured layout is good for writes. However, as discussed earlier, some read patterns (e.g. seqeutial reads) may suffer poor performance. To avoid subsequent poor sequential read performance, logging RAID moves data stored in the LA to the PA during system idle time. Note that the underlying assumption is that the data layout in the PA is efficient for sequential reads. We believe that this is reasonable since the data layout in the PA is typically determined by host file system or applications. Most of today’s file systems and applications such as databases employ a fair amount of optimizations on data placement, such as allocating contiguous disk blocks for data in the same file. Currently, we do not exploit other reorganization optimizations, such as optimizing data layout based on runtime request access patterns [7]. 2.2
The Mapping Structures
In logging RAID, a disk block may reside in the LA or the PA. The data blocks in the LA are always the most up-to-date, since the new updates are always written to the LA first (with one exception as discussed in Section 2.3.). Logging RAID uses a log map, implemented using a hash table, as an internal mapping structure to keep track of the blocks that are in the LA. On receiving a read request for block x, logging RAID first checks if there is a hash entry corresponding to x. If so, x is read from the disk block address indicated by the hash entry. Otherwise, x must be in the PA and its location is x. Writes are somewhat tricky since a disk block can be moved from PA to LA or from one location to another location within LA. Section 2.3 explains in detail how writes are handled. Note that the typical request size between the host system and the disk subsystem is often not as small as one disk sector. In fact, most file systems format disk space using a block size of several KBs. This suggests that the unit of relocation between the PA and the LA or within the LA itself can be much larger than a disk sector. Using large relocation units also reduces the mapping size, since logging RAID only needs to maintain one hash entry per relocation unit rather than per disk sector. The hash table size is c × la r , where c is the number of bytes per hash table entry, la is the log size, and r is the relocation
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays
1305
unit size. For example, with a 2 GB log space and 2 KB relocation unit size, the total hash table size is around 16 MB (assuming c = 16B). If the host system only updates a subset of the disk blocks in a relocation unit, logging RAID will first read the entire relocation unit from either the LA or the PA (depending on where the disk blocks are) into the NVM buffer before the update can take place. Clearly, using a small relocation unit will reduce disk bandwidth usage but may significantly increase the memory requirement for the hash table. Logging RAID must choose an appropriate relocation unit to balance such tradeoffs. We evaluate the effect of relocation unit size in section 3. Logging RAID keeps the hash table in memory at all times. The hash table is also written to a designated disk location in the disk array during system idle time and normal system shutdown. To ensure that logging RAID can reconstruct the in-memory hash table after a system crash, the hash table itself is also logged and written to disk in a similar fashion to data block updates. In other words, logging RAID uses an NVM buffer to accumulate hash table changes and a dedicated log space to store the hash table on disk. Whenever there is an update to the hash table, the update is first written to the NVM buffer. When the buffer is full, it is flushed to the end of the hash table log on disk. This NVM buffer is separate from the NVM buffer used to accumulate the RAID-5 data writes. Logging RAID also periodically writes the entire in-memory hash table to disk (called “checkpointing” in logging RAID). Checkpoints are infrequent, so they rarely cause visible performance impact. After the hash table is checkpointed on disk, logging RAID can recycle the log disk space for the hash table before that checkpoint. Between two consecutive checkpoints, only changes to the hash table need to be written. During system recovery, logging RAID retrieves the mapping information from the most recent checkpoint, then uses the hash table log and NVM to perform a roll-forward to reconstruct the system states. 2.3
Logging RAID Operations
In-line writes Upon receiving a small write to a set of disk blocks, logging RAID first checks if the relocation units that the disk blocks reside in are in the NVM. If so, logging RAID updates the corresponding relocation units directly in the NVM. Otherwise, logging RAID consults the hash table and reads the disk relocation units either from the LA or the PA into the NVM (this read is needed only if the updates are not full relocation unit updates) and updates them. The writes can return to the host once they are written to the NVM. One exception to the scheme above is that for large, RAID-5 stripe-aligned writes, instead of relocating the data blocks in the stripe from the PA to the LA or within the LA, logging RAID writes the stripe directly to the PA. This avoids moving data around later without sacrificing write performance. For each write, if the disk blocks are written to a new location, the hash table is updated accordingly. In-line reads Upon receiving a host read request, logging RAID first checks if the data is in the NVM. If so, the read can be satisfied from the NVM buffer. Otherwise, logging RAID checks the hash table to see if the data has been
1306
Ying Chen, Windsor W. Hsu, and Honesty C. Young
relocated to LA. If so, logging RAID fetches the data from the LA locations indicated by the hash entry. Otherwise, the requested data is read from the PA. Background operation: Idle time data destaging Logging RAID destages the relocation units from LA to PA whenever there is a detected system idle time. Logging RAID destages one stripe at a time until the amount of free space reaches a predetermined threshold. Currently, logging RAID selects the destaging RAID-5 stripe from the head of the log. Since new writes are always appended to the end of the log, the head of the log contains the “oldest” data. [5] discusses several more sophisticated schemes. We skip them here due to space limitations. System idle time detection is also an interesting problem [8]. Currently, logging RAID starts the idle time activities whenever the system is idle for D seconds. In-line data destaging In-line data destaging may be necessary when the log becomes full during NVM flushing. Logging RAID chooses to perform one of the following two actions based on the system state at the time of NVM flushing: 1. write new data directly to PA as in a RAID-5 system. 2. queue the new writes and move some RAID-5 stripes from the LA to the PA to make room for the new writes. After moving some RAID-5 stripes from the LA to the PA, logging RAID writes the NVM buffer to the LA and the queued requests can proceed. In theory, if moving data from the LA to the PA is fast, e.g., when the RAID-5 stripe to be moved contains a lot of “holes” (the holes are created due to multiple updates on the same stripe), queuing the new requests could be more benefitial than direct updates to the PA. On the contrary, if the selected RAID-5 stripe is mostly full, moving it back to the PA could generate a lot of read-modify-write operations. So writing new data directly to the PA could be a win. Currently, logging RAID chooses to queue writes and perform in-line data destaging only if the selected RAID-5 stripe from the LA has at least half of the stripe empty. Otherwise, logging RAID writes the new data to the PA directly.
3
A Trace-Driven Simulation Study
To evaluate the effectiveness of logging RAID, we carried out a trace-driven simulation study. We built a disk array simulator based on DiskSim [9], a configurable disk system simulator. In our evaluation, we studied the behavior of logging RAID under different situations and compared the performance of logging RAID with RAID-1 and RAID-5. Due to space limitations, we do not present all the results in this paper. More detailed results can be found in [5]. 3.1
Experimental Setups and Traces
We used several traces from different systems to drive our simulations. CELLO is a three-month trace gathered on a time-sharing HP-UX system in a real world engineering environment. SNAKE is a two-month trace collected on a HP-UX cluster file server. In addition, we have two TPC-C [10]2 traces gathered on 2
Because our TPC-C benchmark setup has not been audited per the benchmark specification, our workload is technically not TPC-C benchmark workload and should
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays
1307
a multiprocessor PC server running DB2. TPC-C benchmark is an industry standard benchmark used to measure the database system performance for online transaction processing (OLTP) type of workloads. The difference between the two TPC-C traces is on the database buffer pool size configurations in DB2. One trace used a 32 MB buffer pool, and the other used 128 MB. Different DB2 buffer pool configurations can result in different request sequences at the disk array controller due to DB2 caching effects. We are interested in how logging RAID performs with these different TPC-C request streams. Due to long simulation times and numerous combinations of simulations tests, we only used a fraction of the Cello and TPC-C traces in our simulations. Specifically, we used the first one-third of the TPC-C traces and the first half of the Cello trace. Table 1 summarizes the key characteristics of our traces. We especially focused on the write characteristics of these traces. In both the Cello and Snake traces, almost all the writes are less than or equal to 8 KB. In the TPC-C traces, the request size is always 4 KB. Trace characteristics # of requests (in millions) # of reads (in millions) # of write (in millions) Ave read request size Ave write request size Write footprint
Snake 12.0 5.3 6.7 6 KB 7 KB 530742
Cello TPCC-32 TPCC-128 25.4 7.2 5.5 11.0 4.7 3.6 14.4 2.5 1.9 6 KB 4 KB 4 KB 6.5 KB 4 KB 4 KB 2022709 1862556 1828328
Table 1. The key characteristics of the traces. Here, TPCC-X is the TPC-C trace with X MB buffer pool size. The write footprint is the number of distinct 1 KB disk blocks (the smallest write unit in all the traces).
To study the logging RAID behavior, we varied the individual logging RAID tuning parameters one at a time, i.e., the relocation unit sizes and the log space. We also varied the RAID-5 stripe unit size. We found that logging RAID is insensitive to the stripe unit size as long as the unit is larger than 8 KB. In the results shown in this paper, we used 16 KB as the RAID-5 stripe unit. We used smaller number of disks for Snake since it has a smaller footprint. Table 2 lists the parameters used in our sensitivity study. The bold items are the baseline settings for the results shown in this paper. We focus on the relative performance of logging RAID under different situations rather than the absolute performance. Our performance metrics is the average I/O response time.
only be referred to as TPC-C-like. In the rest of this paper, when TPC-C is used to refer to our workload, it should be taken to mean TPC-C-like. Details about the TPC-C traces and the collection methodology can be found in [11].
1308
Ying Chen, Windsor W. Hsu, and Honesty C. Young Simulation parameter Settings Relocation unit size (KB) 2, 4, 8, 16, 32 Fraction of disk space for log 1%, 5%, 10%, 20%, 40% NVM size (KB) 64, 128, 256, 512, 1024, 2048, 4096 Stripe unit (KB) 4, 8, 16, 32, 64 Number of disks 8 for Snake, 16 for others Disk model HP C2249A Request scheduling policy FCFS, SSTF, ELEVATOR, VSCAN Disk RPM 5400, 7200, 10000 SCSI bus transfer rate 10 MB/s, 20 MB/s, 40 MB/s
Table 2. The parameter settings used in the sensitivity study. The bold items are the settings used in the results shown in this paper.
3.2
Simulation Results
Sensitivity to Relocation Unit Sizes Figure 1 shows the performance of logging RAID with different relocation unit sizes for Snake and Cello with a 256 KB NVM stripe buffer. A general trend in these results is that relatively small relocation units give better performance. This is because almost all writes are less than or equal to 8 KB in both of these traces. In logging RAID, if a write only updates a portion of a relocation unit, a pre-read of the entire relocation unit is required. So using relocation unit sizes larger than 8 KB may waste disk bandwidth due to pre-reads and writes of unchanged data. Since our TPC-C traces only contain 4 KB requests, we use 4 KB as the relocation unit size.
Average I/O response time (milisec)
Relocation unit size effect on the performance of logging RAID 30 25 20 15 10 5 0 2 KB
4
8
16
32
Relocation unit sizes
Cello
Snake
Fig. 1. Sensitivity of logging RAID performance to relocation unit sizes.
Sensitivity to Log Disk Space We varied the log space between 1% and 40% of the total disk space in the system. As discussed in Section 2, as long as the log is large enough to buffer the typical bursty writes, logging RAID will perform well. With a small log space, logging RAID has to destage some of the relocated units in-line, lengthening queuing delays for the host requests. Once the log space is
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays
1309
above 5%, the in-line destaging can be largely avoided as shown in Figure 2. It is clear that with a log space that is 1% of the total disk space, both Cello and Snake perform a little worse than the other cases. This is because some number of in-line destaging occured with this small log, causing host requests to wait whenever the in-line destaging occurs. For TPCC-32, 1% of the total disk space is sufficient to store the small writes. We did not repeat the tests with TPCC-128, since TPCC-128 contains fewer writes than TPCC-32.
Average I/O response time (ms)
Effects of the log space on the performance of logging RAID 20 15 10 5 0 Cello 1%
5%
Snake 10%
20%
TPCC-32 40%
Fig. 2. Sensitivity of logging RAID performance to log space.
Note that shortage of log space can potentially affect the performance of logging RAID, but not to the point of being worse than a normal RAID-5 system. This is because in the worst case, logging RAID will simply write data directly to the PA, as does a normal RAID-5 system. Comparison with RAID-1 and RAID-5 In this section, we compare the performance of logging RAID with RAID-1 and RAID-5 with a 512 KB NVM buffer size. For RAID-1 and RAID-5, we applied some optimizations on writes as detailed in [5]. We also selected the best stripe unit size and the number of disks for RAID-1 and RAID-5. We chose 64 KB stripe unit for RAID-1, and 4 KB and 8 KB for RAID-5 for TPC-C traces and HP traces respectively. In logging RAID, we used 5% of the total disk space for the log in the Cello and Snake tests, and 1% in the TPC-C tests. Figure 3 compares the overall performance of different disk array systems, as well as break-downs by read and write performance. Overall, logging RAID gives the best performance in all the tests. It is encouraging to see that in many cases logging RAID performs much better than RAID-1. This is not a surprise since logging RAID can bundle multiple small writes into one large write, while this is not possible with RAID-1. When compared with RAID-5, logging RAID doubles (or even triples) the RAID-5 performance for both Snake and Cello. Logging RAID did not improve TPC-C performance as much as it did the other two workloads. This is because our TPC-C traces are read-intensive, only 34% of the total requests are writes, while Cello and Snake are both write in-
Ying Chen, Windsor W. Hsu, and Honesty C. Young Performance comparison of different disk array architectures for reads only 20 16 12 8 4 0 Snake
RAID-5
TPCC-32
TPCC-128
Performance comparison of different disk array architectures for writes only 60 50 40 30 20 10 0 Snake
Logging RAID
Average response time (ms)
RAID-1
Cello
Average response time (ms)
Average response time (ms)
1310
RAID-1
Cello
RAID-5
TPCC-32
TPCC-128
Logging RAID
Performance comparison of different disk array architectures 35 30 25 20 15 10 5 0 Snake Cello TPCC-32 TPCC-128
RAID-1
RAID-5
Logging RAID
Fig. 3. Performance comparison of RAID-1, RAID-5 and logging RAID for readonly (top-left), write-only (top-right), and overall (bottom) situations.
tensive (more than 60% of requests are writes). Nevertheless, logging RAID is still more than 20% better than RAID-5 for TPC-C. Since logging RAID relocates small data blocks between PA and LA, when reads (especially sequential reads) occur before the data are moved back from LA to PA, the read performance may suffer. To simulate such scenario, we examined the read performance alone in our gathered simulation results. We expect a degraded read performance in our results. Indeed, Figure 3 shows such effects through a breakdown of read performance and write performance. Clearly, logging RAID degraded read performance by a small percentage (from 10% to 24%) when compared to the RAID-5 read performance. In an ideal case, if there is enough idle time for logging RAID to destage data from LA to PA before reads, both reads and writes can be done efficiently. When there is not enough idle time to move data from LA to PA, but the writes dominate the workloads, logging RAID also can perform very well. The problematic situation in logging RAID is when there is not enough idle time and yet reads dominate the workload. Moreover, many data blocks read have been randomly updated, and hence moved from the PA to the LA. In such cases, logging RAID could perform worse than a normal RAID-5. There are several potential solutions to this problem. One is to adopt some dynamic access pattern detection mechanism to detect such workload patterns, and instruct logging RAID not to perform logging optimizations at all. Another possible solution is to select a larger relocation unit size. However, the relocation
Logging RAID – An Approach to Fast, Reliable, and Low-Cost Disk Arrays
1311
unit size must strike a balance between wasting disk bandwidth on writes and maintaining data sequentiality for reads. Our relocation unit size sensitivity tests did not include the worst case workloads. Such tests may be done in the future.
4
Conclusion
Logging RAID employs the logging technique to provide superior performance under a wide range of workloads. It tolerates any single disk failure and incurs low storage hardware cost. The key techniques in logging RAID include: 1. Dynamic relocation of data blocks between a permanent storage area and a log area. 2. Batching small writes into large, sequential writes. Our trace-driven simulation results showed that logging RAID performs well with diverse workloads. The extra storage space required is small. Logging RAID outperforms all the disk array schemes that we simulated. Depending on the workload characteristics, logging RAID can more than double RAID-5 performance.
References 1. D. Patterson, G. Gibson, and Randy Katz. A case for redundant arrays of inexpensive disks (RAID). In Proceedings of the ACM SIGMOD International Conference on Management of Data, pages 109–116, Chicago, IL, June 1988. ACM Press. 2. M. Rosenblum and J. Ousterhout. The design and implementation of a LogStructured file system. ACM TOCS, 10(1):26–52, Februray 1992. 3. J. Wilkes, Richard Golding, Carl Staelin, and Tim Sullivan. The HP AutoRAID hierarchical storage system. ACM TOCS, 14(1):108–136, February 1996. 4. J. Menon. A performance comparison of RAID-5 and log-structured arrays. In Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing, pages 167–178, August 1995. 5. Y. Chen, W. Hsu, and H. Young. Logging RAID – an approach to fast, reliable, and low-cost disk arrays. RJ 10161, IBM Almaden Research Center, October 1999. 6. Transaction Processing Performance Council. TPC benchmark D standard specification. TR, Waterside Associates, Fremont, CA. 7. J. Matthews, D. Roselli, A. Costello, R. Wang, and T. Anderson. Improving the performance of Log-Structured file systems with adaptive methods. In Proceedings of the sixteenth ACM symposium on Operating system principles, Saint Malo, France, 1997. ACM Press. 8. R. Golding, P. Bosch, C. Staelin, T. Sullivan, and J. Wilkes. Idleness is not sloth. In Proceedings of Winter 1995 USENIX, pages 201–222, New Orleans, LA, 1995. 9. G. Ganger, B. Worthington, and Y. Patt. The DiskSim simulation environment. Technical Report http://www.ece.cmu.edu/ ganger/disksim, University of Michigan, February 1998. 10. Transaction Processing Performance Council. TPC benchmark C standard specification. Technical report, Waterside Associates, Fremont, CA. 11. W. W. Hsu, A. J. Smith, and H. C. Young. Analysis of the Characteristics of Production Database Workloads and Comparison with the TPC Benchmarks. TR CSD-99-1070, Computer Science Division, UC Berkeley, November 1999.
Topic 21 Problem Solving Environments Jos´e C. Cunha, David W. Walker, Thierry Priol, and Wolfgang Gentzsch Topic Chairmen
Introduction This workshop encompasses several aspects of current research on Problem Solving Environments (PSE). A PSE is an integrated computing environment for supporting the complete life cycle of design, development, and execution within a specific application domain. A PSE must assist the user in the design and evaluation of the more adequate solutions and problem-solving strategies. It must also support the development of rapid and efficient prototypes to ease the experimentation. This idea of a PSE has been among us for several decades. There are already several fully developed PSEs in distinct areas, such as the automotive and aerospace industries, and PSEs are being developed in many research projects. Modern PSEs increasingly depend on an adequate integration of a diversity of heterogeneous components such as sequential, parallel or distributed problem solvers, tools for data processing, advanced visualization, computational steering, and access to large databases and scientific instruments. Recently there has been an increasing awareness to this topic due to the need to exploit the enabling technologies that will allow the handling of more complex simulation models and larger volumes of generated data, higher degrees of human computer interaction, and more effective forms of cooperation among users in collaborative distributed environments. The main goal of this workshop is to promote a discussion of the main issues involved in the design, implementation and application of PSEs, and to contribute towards future developments concerning PSEs.
The Papers in This Workshop The papers in this workshop provide a global perspective of current work on PSE by addressing the following relevant issues: – – – – – –
Coupling of multi-disciplinary simulation codes Integration of sequential and parallel programs The role of visualization systems in PSEs Interaction and computational steering Large scale Web-based environments Architectures of generic and domain independent PSEs
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1313–1314, 2000. c Springer-Verlag Berlin Heidelberg 2000
1314
Jos´e C. Cunha et al.
Overall, these five papers discuss approaches which are representative of the state-of-the-art and help identifying the trends in the development of future PSEs. The first paper ’AMANDA - A Distributed System for Aircraft Design’ by Kersken et al. discusses how a component-based framework was designed and how it is being used for the integration of coupled sequential and parallel programs. The paper identifies the main application requirements concerning the integration, the efficient massive data exchange between integrated programs, and the hierarchical structuring of a complex system. The paper describes the architecture of the TENT framework and the use of the MpCCI support library for code coupling. Experimental work on the development of two pilot applications in airplane and turbine design is described. The second paper ’Problem Solving Environments: Extending the Role of Visualization Systems’ by Wright et al. presents a discussion on how visualization systems can evolve concerning the aspects of collaborative working, data management and the usability of dataflow visualization environments. The role of Modular Visualization is emphasized as an approach to provide more generic and open architectures for PSEs. The above aspects are discussed and illustrated through examples using the IRIS Explorer system. An architecture is described for Web-based visualization using IRIS Explorer, HTML and VRML, aiming at high flexibility in visualization and user interaction. The third paper ’An Architecture for Web-based Interaction and Steering of Adaptive Parallel/Distributed Applications’ by Muralidhar et al. describes an ongoing effort towards the development of a Web-based collaborative environment supporting remote monitoring and control of adaptive scientific applications. The paper discusses the design of the distributed architecture of the environment and illustrates how its internal layers support the required interaction and steering capabilities. The fourth paper ’Computational Steering in Problem Solving Environments’ by Lancaster focusses on the design and implementation issues of computational steering in PSEs. The paper discusses the requirements posed by computational steering upon the architecture of a PSE. The paper describes experimental work on the implementation of a prototype PSE based on JavaBeans and CORBA and discusses the degree of steering achieved. Finally, the paper ’Implementing Problem Solving Environments for Computational Science’ by Rana et al. discusses the requirements to build a PSE that is easy to use, enables the development of new applications, or the uniform integration of existing application codes. The paper identifies the main functionalities of a PSE, concerning support for component composition and for resource management, and then it presents an overview of current efforts on PSEs. The authors propose a generic, domain independent, infrastructure for a PSE and then show how it is used to build two application dependent PSEs.
AMANDA - A Distributed System for Aircraft Design Hans-Peter Kersken1, Andreas Schreiber1 , Martin Strietzel1 , Michael Faden1 , Regine Ahrem6 , Peter Post6 , Klaus Wolf6 , Armin Beckert4 , Thomas Gerholt2 , Ralf Heinrich5 , and Edmund K¨ ugeler3 1
6
DLR, Simulation- and Softwaretechnology, 51170 Cologne, Germany, 2 DLR, Institute of Fluid Dynamics, 37073 G¨ ottingen, Germany, 3 DLR, Institute of Propulsion Technology, 51170 Cologne, Germany, 4 DLR, Institute of Aeroelasticity, 37073 G¨ ottingen, Germany, 5 DLR, Institute of Design Aerodynamics, 37073 G¨ ottingen, Germany,
Abstract. In the AMANDA project a component-based framework for the integration of coupled technical applications, running distributed in a network, is developed. It is designed to deal with parallel and sequential programs and massive data exchange between the integrated programs. Two pilot applications will be implemented to show the feasibility of the chosen approach: a trimmed, freely flying, elastic airplane and an aircooled turbine. Beside using the integration systems, the MpCCI1 library is used for the coupling of the codes, each simulating single physical processes.
1
Introduction
An efficient design process for airplanes and propulsion systems requires the calculation of numerous variants for which, in the context of the product development, usually only a limited time window is available. A prerequisite for this task is the availability of high-quality procedures for the simulation of the physical processes involved, as well as their coupling, in order to be able to make predictions about physical and functional properties of the final product. In addition, a powerful soft- and hardware infrastructure must be available in order to be able to use these procedures fast and efficiently. Here a software integration system acts as an agent which manages the interaction of software tools. Otherwise the coupling of these tools would require time consuming human intervention. Additionally, it provides a uniform user interface to the integrated applications. The AMANDA project has two main focuses: The first is to extent and enhance the software integration system TENT [9] which is a joint development of 1
MpCCI is a trademark owned by GMD.
A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1315–1322, 2000. c Springer-Verlag Berlin Heidelberg 2000
1316
Hans-Peter Kersken et al.
GMD and DLR. It was already used successfully in the project SUPEA [10] and is available as a prototype system [11]. The second focus is the implementation of pilot applications using this integration environment. Two applications have been selected: – a trimmed, elastic, freely flying airplane and – an air-cooled turbine. The development of the integration framework is driven by requirements posed by these applications, e. g. the integration of parallel programs, the use of a hierarchical structure for setting up a process chain, or a control module to steer the execution of the workflow depending on internally calculated parameters. The applications will be described in more detail in the next section, Sec. 3 gives an overview about the current state of the integration system, and Sec. 4 describes some new features of the integration system necessary for the implementation of the two AMANDA applications.
2 2.1
The AMANDA–Applications Airplane Design
For the simulation of a trimmed, freely flying, elastic airplane the following programs have to be integrated into TENT: – a CFD code, TAU [8] and FLOWer [4], – a structural mechanics code (NASTRAN [7]) and a multi-body program (SIMPACK [5]), to calculate the deformation of the structure, – a coupling module, to control the coupled fluid-structure computation, – a grid deformation module, to calculate a new grid for the CFD simulation, – a flight mechanics/controls module (build using an Object-Oriented modelling approach [6]), to set the aerodynamic control surface positions, – visualization tools. Figure 1 shows a possible scenario of the coupling. The process chain consisting of these codes is hierarchically structured. The lowest level contains the parallel CFD solver only. This CFD-subsystem consists of the flow solver itself and auxiliary programs to handle mesh adaptation. The next level, the aeroelasticsubsystem, comprises the CFD solver, the structural mechanics or multi-body code, and the grid deformation module. The data transfer from the CFD code to the structural mechanics code is performed via the MpCCI (former CoColib) [1] library which will be extended in order to deal with the interpolation used in fluid-structure interaction problems. For the solution of the coupled non-linear aeroelastic equations a staggered algorithm is implemented in the control process [2]. The highest level consists of the flight mechanics module coupled to the aeroelastic-subsystem. Each single code is additionally accompanied by its own visualization tool. The computation of a stable flight state typically proceeds as follows:
AMANDA - A Distributed System for Aircraft Design
1317
aeroelastic 3
8
2
TAU subsystem
9
Flight Mechanics
Grid Deformation
TAU PreProcessor
6
9
TAU SimEngine
Gather
6 mpi-start
Control Control
3
Scatter
4 mpi-start
TP NT
7 5
7
NASTRAN
TAU PostProcessor
7
4
8
5
Control
TAU Adaption
flight mechanics 3 Grid Generator
PATRAN
1
1
base geometry
flight attitude
NT
Control
wrapper (base component)
2
1
5
4
Visualizer for NASTRAN
7 TP
Visualizer for TAU
coupled simulation data transfer (inside the subsystem)
flap position
subsystem (container)
data transfer (across subsystem boundaries)
control unit
implicit control (event sender)
[controls all components inside a subsystem (uses explicit control mechanisms)]
implicit control (event receiver) control attribute
Fig. 1. Data flow for coupled CFD/structural mechanics/flight control system. The order of execution of the components is controled by the components labeled Control by means of attributes provided by the simulation components. Starting by calculating the flow around the airplane, the pressure forces on the wings and the nacelle are derived. Theses forces are transfered to the structural mechanics code and interpolated to the structural mechanics grid using the MpCCI library. This code calculates the deformation of the structure which in turn influences the flow field around the airplane. At a final stage it is planned to extend the system and feed the calculated state into a flight mechanics/controls module to set the control surfaces accordingly and to obtain a stable flight position. This changes the geometry of the wings and requires therefore the recalculation of the flow field and the deformation. As a prototype the coupled CFD/structural mechanics computation for a flexible wing has been realized without the integration system using file transfer for data exchange. The MPCCI library and the integration system will be used in near future. 2.2
Turbine Design
A new aspect in virtual turbine design is the simulation of flow inside the turbine in consideration of the heat load on the blades and the cooling. The numerical modeling of the position and size of cooling channels and holes in the blades are essential for the layout of an air-cooled turbine.
1318
Hans-Peter Kersken et al.
Fig. 2. Screenshot of the TENT gui for the turbine simulation described in Sec. 2.2 The CFD code TRACE [13], a 3D-Navier-Stokes-Solver for the simulation of steady and unsteady multistage turbomachinery applications, and a heat conduction problem solver (NASTRAN) are coupled to simulate the airflow through the turbine together with the temperature distribution inside the blades. For the coupling of the flow simulation and the heat conduction a stable coupling algorithm as been developed where TRACE delivers the temperatures of the air surrounding the blade and the heat transfer coefficients as boundary conditions to NASTRAN which in turn calculates the temperature distribution inside the blade. The temperature at the surface of the blade is returned to TRACE as boundary condition for the airflow. A coupled simulation has been already realized using the MpCCI library for data exchange.
3
The Software Integration System
TENT is a framework for building and controlling complex technical workflows where the embedded application may be run on arbitrary machines in a network, i. e. the machines can be chosen depending on the application. It was developed in view of the integration of HPC applications and provides a graphical user interface to build process chains and control the integrated applications. Figure 2 show a screenshot for the simualtion scenario described in Sec. 2.2. The
AMANDA - A Distributed System for Aircraft Design
1319
TENT software system consists of four different packages as shown in Fig. 3. The software development kit (SDK) summarizes all interface definitions and libraries for the software development with TENT. The base system includes all basic services needed to run a system integrated with TENT. The facilities are a collection of high-level services, and the components consist of wrappers and special services for the integration of applications as TENT-components. 3.1
The Software Development Kit
TENT defines a component architecture and an application component interface on top of the CORBA object model. The TENT component architecture is inspired by the JavaBeans specification [3]. The data exchange interface as part of the software development kit allows the transparent data exchange between two TENT-components. It can invoke a data converter, which automatically converts data between the data formats supported by the integrated tools. For efficiency reasons CORBA is used only to coordinate the data transfer, i. e. only references to the data are exchanged using CORBA itself. These references are for example path names in a file system or port numbers and IP-addresses, if socket communication is employed. The transfer of the data themselves is performed in a more efficient way, depending on the situation at hand, e. g. by socket communication or file transfer. 3.2
TENT - Base System
The TENT base system consists of the components necessary to run the integrated system of applications. For portability reasons, these components are currently implemented in Java. The Master Control Process (MCP) is the main control instance in TENT. It realizes and controls system tasks such as process chain management, construction and destruction of components, or monitoring the state of the system.
TENT Basesystem
TENT - SDK Component Architecture IDLInterfaces
Factories
GUI
Coupling
Data Converter
Name Server
TENT Facilities
Data Exchange Interface Development Support Libraries
Master Control Process
Scheduling System
Data Server
TENT Components CFDWrapper
FEMWrapper
VRWrapper
VisualizerWrapper
Fig. 3. Packages of the software integration system TENT.
1320
Hans-Peter Kersken et al. Component Factory
Component Factory
Compute Server
Simulation Engine
Partitioner
Postprozessor
Visualization Tool
Frontend Machine
Name Server MCP JavaBeans like Component stubs
GUI encapsulated Component GUIs
Key:
Java Impl.
other
Corba. References
Fig. 4. Architectural overview of TENT.
The factories run on every machine in the TENT framework. They start, control, and terminate the applications on their machine. The naming service is an enhanced CORBA naming service. The relations between the base components are shown in Fig. 4. 3.3
TENT Facilities
In order to support higher-level workflows, the system offers more sophisticated facilities, e. g. data and job management(scheduling system) servers are planed. An important facility is the data converter which acts as a filter when integrated programs with incompatible data formats exchange information. 3.4
TENT Components
All applications must be encapsulated by wrappers to access their functionality in TENT. Therefore these wrappers have to be equipped with a CORBA interface to comply to the TENT component architecture. Depending on the level of access to the program, source code, API, or file I/O, the wrapper can be tightly coupled with the application, e. g. linked together with it, or must be a stand alone tool that starts the application via system services. Depending on the wrapping mechanism, the communication between the wrapper and the application is implemented by direct memory access, IPC mechanisms, file exchange, or any other suitable communication method
4
Impacts of the AMANDA–Applications on TENT
In this section new features which have been added to the integration system due to requirements from the AMANDA-project are discussed.
AMANDA - A Distributed System for Aircraft Design
4.1
1321
NASTRAN-Co-process
Every application not especially written for the TENT-systems has to be wrapped to be accessible by the system. For codes with source code access it is now a standard task. Some codes have already been integrated by changing the main routine only. The source code had to be modified to include control and communication commands. For NASTRAN just the opposite was true, neither the source code was accessible nor does an API exist. A NASTRAN run is controlled completely by an input file. Therefore, an additional program had to be written, a NASTRAN-co-process, to hook up NASTRAN to the integration system. The co-process is responsible for setting up NASTRAN input files, parsing NASTRAN output files to extract the results, and connecting to the integration system. Because this module was developed with its integration into TENT in mind, it was implemented as a library which methods can be called from a Python script. If NASTRAN is a part of an application coupled by the MpCCI the script can handle the control of the coupled application as well. 4.2
Strongly Coupled Multi-disciplinary Subsystems
In the context of AMANDA the coupling of multi-disciplinary simulation codes is a major issue. Therefore an interface to the coupling library MpCCI is provided. MpCCI coordinates the exchange of boundary informations between simulation applications working on adjacent grids. The communication between applications inside the integration system is usually controlled by CORBA mechanisms. In case of applications coupled by the MPI-based MpCCI, data are exchanged using the MPI-library without any interference of the integration system. The coupling algorithm itself, i. e. the number of time or iteration steps a code has to perform or the decision when to terminate the computation is not part of the MpCCI library. Hence, a control process is introduced into TENT to handle this task. A scripting language (Python) is used to simplify the development and the maintenance of this module. The flexibility offered by a scripting language is more important here than in other modules because setting up the coupling of two independent programs based on algorithms originally not intended for the use in coupled computation may require a lot of experimentation to find a good, i. e. converging and stable, coupling algorithm. 4.3
Hierarchical Structure
As can be easily seen in Fig. 1, without a structuring strategy it soon becomes impossible to handle the complexity of coupled problems. A design decision motivated by scenarios like this was to allow the encapsulation of existing workflows to become a single TENT-component which in turn may consist of other composite components. Each component has its own control process and by defining appropriate interfaces the complexity of a module is completely invisible to other components. The master control process is hence replaced by a set of independently working control processes in each module.
1322
5
Hans-Peter Kersken et al.
Conclusions
The TENT integration system allows to link software tools originally intended for stand alone use. By embedding them into the TENT framework, on the on hand, a uniform access to the tools is provided, on the other hand, data are transfered transparently between different tools without intervention from the user. The applications described show the flexibility and extensibility of the integration system. It will be used and extended in other, scientific and industrial, projects, e. g. virtual prototyping in the automotive industry [12], the simulation of a re-entry system, and combustion chamber modeling.
Acknowledgments The work described in this paper is partially funded by the Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren (HGF) and the german ministry for education and research under grant 01SF9822.
References [1] R. Ahrem, P. Post, and K. Wolf. A communication library to couple simulation codes on distributed systems for multiphysics computations. In Proceedings of ParCo99, Delft, August 1999. [2] A. Beckert. Ein Beitrag zur Str¨ omung-Struktur-Kopplung f¨ ur die Berechnung des aeroelastischen Gleichgewichtszustandes. ISRN DLR-FB 97-42, DLR, G¨ ottingen, 1997. [3] G. Hamilton. JavaBeans API Specification. Sun Microsystems Inc., July 1997. [4] N. Kroll. The National CFD Project Megaflow - status report. In H. K¨orner and R. Hilbig, editors, Notes on numerical fluid mechanics, volume 60. Braunschweig, Wiesbaden, 1999. [5] W. Kr¨ uger and W. Kort¨ um. Multibody simulation in the integrated design of semi-active landing gears. In Proceedings of the AIAA Modeling and Simulation Technologies Conference. Boston, 1998. [6] D. Moormann, P. J. Mosterman, and G. Looye. Object-oriented computational model building of aircraft flight dynamics and systems. Aerospace Science and Technology, 3:115–126, April 1999. [7] Nastran. http://www.macsch.com. [8] D. Schwamborn, T. Gerholt, and R. Kessler. DLR TAU code. In Procedings of the ODAS-Symposium, June 1999. [9] M. Strietzel, T. Breitfeld, T. Forkert, A. Schreiber, and K. Wolf. The distributed engineering framework TENT. to be published in: Vector and Parallel Processing – VECPAR2000. Lecture Notes in Computer Science, 2000. [10] Supea. http://www.sistec.dlr.de/de/projects/supea. [11] Tent. http://www.sistec.dlr.de/tent. [12] C.-A. Thole, S. Kolibal, and K. Wolf. AUTOBENCH: Environment for the Development of Virtual Automotive Prototypes. In Proceedings of 2nd CME-Congress (to appear), Bremen, September 1999. [13] D. T. Vogel and E. K¨ ugler. The generation of artificial counter rotating vortices and the application for fan-shaped film-cooling holes. In Proceedings of the 14th ISABE, ISABE-Paper 99-7144, 1999.
Problem Solving Environments: Extending the Rˆ ole of Visualization Systems Helen Wright1 , Ken Brodlie2 , Jason Wood2 , and Jim Procter3 1
2 3
Deptartment of Computer Science, University of Hull, Hull HU6 7RX, UK [email protected] School of Computer Studies, University of Leeds, Leeds LS2 9JT, UK {kwb, jason}@scs.leeds.ac.uk now at Research School of Chemistry, Australian National University, Canberra, ACT 0200, Australia [email protected]
Abstract. Visualization systems based on the dataflow paradigm are enjoying increasing popularity in the field of scientific computation. Not only do they permit rapid construction of a display application, but they also allow the simulation to be incorporated, giving the scientist the opportunity for computational steering as well. However, if these systems are to realise their full potential as problem solving frameworks, then three key requirements of support for group working, data persistence and usability must be addressed. This paper reviews our prior work on collaborative visualization and data management and reports new developments to improve user interface flexibility. These extensions are then assessed in the context of a unifying, augmented architecture, which in turn indicates scope for future work.
1
Introduction
Advances in computing in the last ten years have brought about a significant change in the modelling and simulation of complex phenomena. Increases in computer processing power, memory, disk and graphics capacity have not only brought about a corresponding increase in the size of problem that can be attempted, but have also brought about a fundamental change in how such problems are tackled. Calculations which were formerly run in batch mode with their output scrutinised afterwards can now be monitored whilst in progress using graphical means, or even ‘steered’ by altering their input parameters according to the current visual results. One approach, exemplified in the SCIRun system [1], is to develop a purpose-built computational steering system from scratch, giving greater flexibility at the expense of more development effort. Another is to provide the tools to instrument an existing code and visualize the results (e.g. [2]). Such work has computational steering as its main focus, whilst other projects such as [3] have addressed the interworking of component-like simulation and visualization codes across heterogenous networks. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1323–1331, 2000. c Springer-Verlag Berlin Heidelberg 2000
1324
Helen Wright et al.
The application of computational simulation to real world problems thus increasingly depends on the integration of a collection of different tools, utilised by a number of investigators having a variety of different skills. Furthermore, as projects grow in size and complexity, collaboration amongst co-workers who may be geographically separated must also be considered an issue. This, too, is recognised in [2] and has received attention in other forums [4], [5]. In this paper we review the rˆ ole that commercially-available visualization systems may play in computational simulation, concentrating in particular on Modular Visualization Environments (MVEs). We begin by capturing an architectural model of these systems, which is then extended in three key ways.
2
Visualization Architecture and Extensions
MVEs, usefully summarised in [6], offer a variety of techniques for graphical output but without the need to program. Instead, the user selects code blocks, or modules, from a system-provided repository and joins them together using the mouse. Although widely used for post-processing simulation data, these systems are also interesting for computational steering because they allow the scientist to interact with the simulation code itself, either by incorporating it into the environment using an application programmer’s interface, or by loosely-coupling the simulation and MVE. These systems implement the Filter, Map and Render pipeline model of visualization proposed by Haber and McNabb [7]. At the Filter stage, data input from disk or direct from some simulation code is sampled if dense or interpolated if sparse. In the Mapping stage, a geometrical representation is constructed, whilst at the Render stage this object is drawn and lit in order to produce the image. Modules only execute when the user varies one of their parameters or new data arrives. In practice a number of individual modules may contribute to each of these stages and may comprise both serial and parallel codes. Coarse-grained parallelism can be exploited by distributing modules across a number of heterogeneous machines, with the environment handling user authentication and synchronisation of data flow. Alternatively, a computationally intensive code may be run on some remote supercomputer, with the results returned transparently for visualization at the workstation. Although often the user interacts with the dataflow pipeline directly, systems also provide an end-user abstraction, or ‘shrink-wrap’ mode. Here, selected parameters of the simulation and visualization can be exported to a separate user interface, whilst the pipeline itself and other parameters remain private to the application developer. Figure 1 depicts a typical scenario, with a simulation code and filter elements running on a remote machine. In spite of their flexibility, MVEs also have their weaknesses. Firstly, although the dataflow pipeline may be distributed across several computers, logically it remains a single system for one user, with its interface, parameters and data flows intrinsically bound to it. Secondly, as the user changes the parameters of the calculation, so the data that flows changes. Data is ephemeral in a standard dataflow system and
Problem Solving Environments: Extending the Rˆ ole of Visualization Systems
1325
S - Simulation
S
F
F - Filter Remote Host Data
Network Local Host
M
R
Simplified User Interface
I
Parameter M - Map R - Render
I - Image
Fig. 1. MVE being used to steer a simulation running remotely only the current state of the simulation/visualization program is ever accessible. Thirdly, complex sequences of actions, as may arise in computational steering, are difficult to achieve in the dataflow paradigm. Nonetheless, the general applicability of these systems and their extensibility, coupled with the infrastructure they provide for process management and distribution, warrants their consideration as generic problem solving frameworks, provided these deficiencies can be addressed. 2.1
Collaborative Working
One approach to collaborative visualization is the COVISA system described by Wood et al [8]. This system supports collaboration by allowing the selective sharing of the data, visualization parameters and pipeline building processes of an MVE. Because the MVE architecture is open, data and parameters can be captured where they flow between modules and distributed to other users. By exploiting the extensibility of the MVE we have created a set of modules, denoted by ‘C’ in Figure 2, that can pass data and parameters into and out of the environment. These are supported by an external server process, which is invisible to the users, for routing data between collaborators. Sharing data via modules provides a familiar paradigm and avoids the need to learn a new interface. Other modules provided allow a user to join and leave a collaborative session and to participate in collaborative map building. This flexible approach means that users are free to choose at which points in the pipeline data and parameters are shared. Each team member works with their own copy of the visualization system, sharing data and control parameters as resource limitations and security considerations dictate. For example, in a compute-intensive application which generates a lot of data, only the geometrical representation coming from the mapping stage need be shared in order for co-workers to see the visualization (Figure 2). This configuration also means the simulation code and raw data remain private to the simulation owner, though co-workers can contribute to steering and visualization choices by exporting parameter values to their colleague’s environment.
1326
Helen Wright et al. Principal Investigator
S
F
C
M C
R
I C - Collaboratively Aware Module
C
Network
C
C C
R
I
Co-worker
Fig. 2. Typical collaborative configuration, with shared elements reflecting resource and security constraints COVISA is by no means the only collaborative visualization system available, for example, COVISE [9] offers collaboration by running a single instance of the base visualization system, but with multiple user interfaces, each accessing the whole pipeline. This can be achieved in COVISA by one user running all of the modules in their environment, with the others sharing control of the visualization parameters and receiving the visualized result. Another, HIGHEND [10], has multiple instances of a visualization system, one per user, each with its own user interface. The systems share synchronisation information so consistency is maintained. The equivalent in COVISA would be for each user to run the same set of modules and share control parameters. The architecture adopted here thus allows COVISA to emulate the style of collaboration offered by these other approaches, but with the key difference that users can work independently on some parts of the pipeline and collaborate over others. Whilst our implementation is for IRIS Explorer, a similar architecture has also been used to extend AVS [11]. 2.2
Data Persistence
Interactive systems can bring improved insight to a problem, but the significance of any particular result is rarely appreciated at the time it is observed. More usually, hindsight plays a large part in understanding what has gone before. The need to record the progress of an investigation is thus important and has been addressed previously: the GRASPARC project [12] captured both data and parameters in the form of a tree, whose branch points signified a return to some previous simulation attempt; Mulder and van Wijk [13] have linked simulation parameters to graphical objects in the visualization, in order to see how data changes over time. Data persistence is difficult to achieve in the dataflow model, however. Abram and Treinish [14] have proposed caching data, whilst van Liere et al [15] cite this problem as a motivating factor towards a completely new visualization architecture. Another approach by Wright et al [16] aims to be more flexible than a simple cache, but nonetheless uses a dataflow approach. Based on the GRASPARC tree idea, it includes this into the pipeline by providing an additional
Problem Solving Environments: Extending the Rˆ ole of Visualization Systems
1327
module called HyperScribe. Figure 3 shows the module being used to capture simulation data for a gas turbine study, resulting from steering the ambient temperature, T. The first series of results comprises twelve distinct runs carried out at 600K to 1150K in 50K increments. As each new data set is computed the user stores it on disk by specifying an ‘Add’ event on the module’s graphical user interface (GUI), in the process creating the circles (simulation ‘snapshots’) which build up the central portion of the tree from left to right. Visualization shows the optimum operating conditions to lie somewhere between 700K and 800K, so additional runs are made to fill in data between these temperatures and the extra snapshots are added to form the side branches. The entire set of results is then retrieved for visualization using ‘Recall’ events, again on the HyperScribe GUI. Data and Parameter Storage Network
S
F
H
M
R
T
600K Data Set
Graphical User Interface to Parameter and Data Organisation
I
H - HyperScribe Data and Parameter Management Module
1150K Data Set
Fig. 3. HyperScribe data management
Figure 4 shows the results of tests in which users were asked to read in 12, 16 or 20 files of results and visualize them, either direct from disk or using Recall events on a tree. On average, using HyperScribe gave a 63% reduction in the time taken but, more importantly, during the 42 task instances observed, 6 errors were noted and all occurred when working with the results files directly. One reason could be a progressive loss of concentration by the user during time-consuming, repetitive tasks, in which case we might expect HyperScribe to increase nearly three-fold the size of problem that can reasonably be tackled. With planned enhancements to the HyperScribe interface, this figure could be improved still further. 2.3
Pipeline Management
Interactive systems often exhibit a repetitive element [17] and computational steering is no exception. For example, consider the following study of a chemical reaction which varies over time:
Helen Wright et al.
Task Completion Time (s)
1328
100
Without HyperScribe With HyperScribe 50
0
0
5
10
15
20
Size of Problem
Fig. 4. Time taken to recall and visualize a 12-, 16- and 20-snapshot tree, compared with reading in the equivalent results stored as a set of files 1. Calculate new data starting from t = 0 and view the whole time series 2. Crop the viewed data to the portion of interest between t = t1 and t = t2 3. Restart the calculation from t = t1 , changing parameters. In the dataflow model this requires three separate pipeline configurations which may be used repeatedly as parameters such as ambient temperature, computational tolerance and timestep are steered. Connecting and disconnecting modules manually becomes laborious and mistakes may result in data being lost. Additionally the end-user abstraction, or shrink-wrap mode, cannot be used since this hides the pipeline. Our most recent MVE extension, in the form of topology definition and management modules, seeks to support such composite activites by mapping whole configurations to a single menu item or button (Figure 5). Using the standard MVE GUI, the user constructs each pipeline just once and
Fig. 5. Separate pipeline configurations map to a single button each
Problem Solving Environments: Extending the Rˆ ole of Visualization Systems
1329
gives it a meaningful name, which is stored in a file by the topology definition module, along with the configuration. Pipeline configurations held in the file can be edited, replaced and added to for flexibility. Thereafter, pressing a particular configuration button causes the topology management module to input to the environment a script of IRIS Explorer command line instructions which changes the pipeline. The particular sequence of instructions needed is determined automatically given the module’s knowledge of the current and target configurations. An application developer can choose whether to leave the pipeline available for additional manual interactions, or whether to export the configuration menu to a simplified user interface using shrink-wrap mode.
2.4
Augmented Architecture
Combining all these extensions, we can now propose an augmented architecture, Figure 6, whereby all the elements can work together. For example, HyperScribe’s GUI is realised using the geometry data type, and it is a simple matter
Co-worker(s)/ Resumed Session
Data and Parameter Storage
F
t
Network
S
0
M
F
M
Remotely Hosted Module
R
Locally Hosted Module
M S
User Interface - Parameter and Pipeline Abstraction
I
Data and Parameter Import/ Export
Command Script
S
Fig. 6. Augmented MVE, with a simplified user interface giving access both to selected module parameters and different pipeline configurations. Data and parameter import and export allow different steering runs to be recorded and/or passed to co-workers, either synchronously or asynchronously
1330
Helen Wright et al.
to export this to one’s co-workers using a collaboratively aware module, in order to cooperate over a single database of steering results. If distant members of a team work asynchronously, anyone can pick up a database of results made earlier by a colleague by running HyperScribe remotely on that person’s machine – the only collective decision needed is which particular machine to save the results on. Our aim throughout is to work within the existing capabilities of the MVE, and this is especially important at the user interface. Here, by providing additional modules to perform the various operations, we avoid placing an additional learning burden on the user. This strategy also means our additional features can become part of a simplified interface constructed for an end-user. For example, a collaborative working application can be set up which hides the communicating modules, or different pipeline configurations can be presented simply as buttons, in a manner analogous to the abstraction of selected parameters already described.
3
Conclusions
In this paper we have reviewed past and current efforts to extend existing Modular Visualization Environments, drawing these together into an augmented architecture which demonstrates the potential for such systems to become Problem Solving Environments. A key principle of our approach has been to utilise existing MVE featues, which allows the different enhancements to interwork as required. Improving the functionality of existing systems holds twofold benefits: firstly, we can build on work already completed to improve these; secondly, extending current systems with an established user base allows early exploitation of the ideas – IRIS Explorer alone, for example, has several hundred users. Perhaps more importantly, improving the usability of current systems allows experience gained on large-scale projects to benefit a range of other, smaller-scale endeavours. This class of applications includes those where the demand for megaflops may not be so great, but where, if a solution is to be found efficiently, the requirement for cooperative work, data and process management tools is just as real as in any large Problem Solving Environment. Thus we envisage in the coming years a growing recognition that the concept of high performance includes not only the processing power applied to some problem, but also the effectiveness of the user in solving it.
Acknowledgements We particularly thank NAG Ltd and the UK EPSRC for funding; JP thanks Mr and Mrs G Procter for their support whilst at Leeds. Case studies, simulation code and application expertise were generously provided by BG plc, the School of Chemistry and the Department of Fuel and Energy at the University of Leeds,
Problem Solving Environments: Extending the Rˆ ole of Visualization Systems
1331
UK, and Sandia National Laboratories, Livermore, USA. Finally, our thanks to the anonymous reviewers for their valuable suggestions for improvement.
References 1. C Johnson, S G Parker, C Hansen, G L Kindlmann and Y Livnat. Interactive simulation and visualization. IEEE Computer, 32(12):59–65, IEEE Computer Society (1999) 2. J A Kohl and P M Papadopoulos. Computational steering and interactive visualization in distributed applications. ORNL/TM-13299, Oak Ridge National Laboratory, Oak Ridge, USA. (1999) 3. Efficient coupling of parallel applications using parallel application workspace. http://www/acl/lanl.gov/PAWS/, Los Alamos National Laboratory, California, USA. (2000) 4. K W Brodlie, D A Duce, J R Gallop and J D Wood. Distributed co-operative visualization, in Eurographics State of the Art Reports, A Augusto de Sousa and R Hopgood (editors), Eurographics98 Conference, pp27 - 50. (1998) 5. K W Brodlie and J D Wood. Volume graphics and the Internet, in M Chen, A E Kaufman and R Yagel (editors), Volume Graphics, pp317–331. Springer Verlag, (2000) 6. G Cameron. Modular visualization environments: Past, present and future. Computer Graphics, 29(2):3–4, (1995) 7. R B Haber and D A McNabb. Visualization idioms : A conceptual model for scientific visualization systems. In B Shriver G M Nielson and L J Rosenblum, editors, Visualization in Scientific Computing, pages 74–93. IEEE, (1990) 8. J D Wood, H Wright and K W Brodlie, Collaborative visualization, Proceedings of Visualization ‘97, pages 253–259. IEEE, (1997) 9. A Wierse, U Lang and R Ruhle, Architectures of Distributed Visualization Systems and their Enhancements, Eurographics Workshop on Visualization in Scientific Computing, Abingdon (1993) 10. H G Pagendarm and B Walter, A prototype of a cooperative visualization workplace for the aerodynamicist, Computer Graphics Forum, 12(3):C485–C496. Eurographics, (1993) 11. D A Duce, J R Gallop, I J Johnson, K Robinson, C D Seelig and C S Cooper, Distributed Cooperative Visualization - The MANICORAL Approach, EurographicsUK Chapter Conference, Leeds (1998) 12. K W Brodlie, A Poon, H Wright, L A Brankin, G A Banecki and A M Gay, GRASPARC: A Problem Solving Environment Integrating Computation and Visualization, Proceedings of Visualization ‘93. IEEE, (1993) 13. J D Mulder and J J van Wijk, 3D computational steering with parametrized graphical objects, Proceedings of Visualization ‘95, pages 304–311. IEEE, (1995) 14. Greg Abram and Lloyd Treinish. An extended data-flow architecture for data analysis and visualization. Computer Graphics, 29(2):17–21, (1995) 15. R van Liere, J Harkes and W de Leeuw, A distributed blackboard architecture for interactive data visualization, Proceedings of Visualization ‘98, pages 225–231. IEEE, (1998) 16. H Wright, K W Brodlie and M Brown, The dataflow visualization pipeline as a problem solving environment, Eurographics Workshop on Scientific Visualization ‘96, pages 267–276. Springer-Verlag, (1996) 17. J Nielsen, Usability engineering. Academic Press, (1993)
An Architecture for Web-Based Interaction and Steering of Adaptive Parallel/Distributed Applications Rajeev Muralidhar, Samian Kaur, and Manish Parashar Department of Electrical and Computer Engineering and CAIP Center, Rutgers University, 94 Brett Road, Piscataway, NJ 08854. Tel: (732) 445-5388 Fax: (732) 445-0593 {rajeevdm,samian,parashar}@caip.rutgers.edu
Abstract. This paper presents an architecture for web-based interaction and steering of parallel/distributed scientific applications. The architecture is composed of detachable thin-clients at the front-end, a network of web servers in the middle, and a control network of sensors, actuators and interaction agents at the back-end. The interaction servers enable clients to connect to, and collaboratively interact with registered applications using a conventional browser. The application control network enables sensors and actuators to be encapsulated within, and directly deployed with the computational objects. Interaction agents resident at each computational node register the interaction objects and export their interaction interfaces. An application interaction gateway manages the overall interaction through the control network of interaction agents and objects. It uses Java proxy objects that mirror computational objects to enable them to be directly accessed by the interaction web-server. The presented architecture is part of an ongoing effort to develop and deploy a web-based computational collaboratory that enables geographically distributed scientists and engineers to collaboratively monitor and control distributed applications.
1 Introduction Simulations are playing an increasingly critical role in all areas of science and engineering. As the complexity and computational costs of these simulations grows, it has become important for the scientists and engineers to be able to monitor the progress of these simulations, and to control or steer them at runtime. The utility and cost-effectiveness of these simulations can be greatly increased by transforming the traditional batch simulations into more interactive ones. Closing the loop between the user and the simulations enables the experts to drive the discovery process by observing intermediate results, by changing parameters to lead the simulation to more interesting domains, play what-if games, detect and correct unstable situations, and terminate uninteresting runs early. Furthermore, the increased complexity and multidisciplinary nature of these simulations necessitates a collaborative effort among multiple, usually geographically distributed scientists/engineers. As a result, collaboration-enabling tools have become critical for transforming simulations into true research modalities. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1332-1339, 2000. Springer-Verlag Berlin Heidelberg 2000
An Architecture for Web-Based Interaction
1333
Enabling seamless interaction and steering high-performance parallel/distributed applications presents many challenges. A key issue is the definition and deployment of interaction objects with sensors and actuators [16] that will be used to monitor and control the applications. These sensors and actuators must be co-located with the computational data-structures in order to be able to control individual application data structures. Defining these interfaces in a generic manner and deploying them in distributed environments can be non-trivial, as computational objects can span multiple processors and address spaces. The problem is further compounded in the case of adaptive applications (e.g. simulations on adaptive meshes) where computational objects can be created, deleted, modified and redistributed on the fly. Another issue is the deployment of a control network that interconnects these sensor and actuators so that commands and requests can be routed to the appropriate set of computational objects, and information returned can be collated and coherently presented. Finally, the interaction and steering interfaces presented by the application need to be exported so that they can be easily accessed by a group of collaborating users to monitor, analyze, and control the application. The objective of this paper is to present the design of a web-based collaborative interaction and steering environment that addresses each of these issues. The system supports a 3-tier architecture composed of detachable thin-clients at the front-end, a network of Java interaction servers in the middle, and a control network of sensors, actuators, and interaction agents superimposed on the application data-network at the back-end. The interaction web server enables clients to connect to and collaboratively interact with registered applications using a conventional browser. Furthermore, it provides seamless access to computational and visualization servers, and simulation archives. The application control network enables sensor and actuators to be encapsulated within, and directly deployed with the computational objects. Interaction agents resident at each computational node register the interaction objects and export their interaction interfaces. These agents coordinate interactions with distributed and dynamic computational objects. The application interaction proxy manages the overall interaction through the control network of interaction agents and objects. It uses JNI [2] to create Java proxy objects that mirror the computational objects and allow them to be directly accessed by the interaction web-server. The presented research is part of DISCOVER1, an ongoing research initiative aimed at developing a web-based interactive computational collaboratory. The current implementation enables geographically distributed clients to use the web to simultaneously connect, monitor and steer multiple applications in a collaborative fashion through a set of dedicated interaction Web Servers. The rest of the paper is organized as follows: A brief overview of related research is presented in Section 2. Section 3 outlines the DISCOVER system architecture. Section 4 presents the design, implementation, and operation of the interaction webserver. Section 5 describes the design and implementation of control network, the application interaction substrate and its interface to the interaction server. Section 6 presents conclusions and current and future work.
1
Distributed Interactive Steering and Collaborative (www.caip.rutgers.edu/TASSL/Projects/DISCOVER/)
Visualization
EnviRonment
1334
Rajeev Muralidhar, Samian Kaur, and Manish Parashar
2 Related Work Interactive systems for application run-time steering and control can be classified as follows: (1) Event based steering systems - These systems are oriented towards the processing of “events” that occur when pieces of code are executed. Instrumentation code is placed in the application statically at compile time at such instrumentation points. When the corresponding events occur, steering actions and decisions are taken. Systems that fall under this category include Progress [18], Magellan [16] and the CSE [5]. (2) Systems with high-level abstractions – In order to overcome the shortcomings of event-based steering, systems such as the Mirror Object Steering System (MOSS) [4] provide higher-level abstractions for steering by translating application level data structures and parameters into CORBA [13] - style objects. The DISCOVER system presented in this paper falls under this category. Key contributions of the DISCOVER architecture include (a) support for distributed dynamic interactive objects that can span multiple address spaces and can be dynamically created and destroyed, (b) a scalable control network and (c) support for web-based interaction and steering portals. Other interactive systems include systems for interactive program construction (e.g. SCIRun [6]), systems for interactive performance optimizations (e.g. Autopilot [15]), systems for interactive application configuration and deployment (e.g. WebFlow [17], Gateway [3]). In addition to these systems, there are numerous systems that provide web-based collaborative visualization environments like DOVE [7], the Web Based Collaborative Visualization [8] system, the NCSA Habenaro [9] system, Tango [10], CCASE [11] and CEV [12].
3 DISCOVER: An Interactive Computational Collaboratory Fig. 1 presents an architectural overview of the DISCOVER collaboratory aimed at enabling web-based interaction and steering of high-performance parallel/distributed applications. The system has a 3-tier architecture composed of detachable thin-clients at the front-end, a network of Java interaction servers in the middle, and a control network of sensors, actuators and interaction agents superimposed on the application data-network at the back-end. The front-end consists of a range of web-based client portals supporting palmtops connected through a wireless link as well high-end desktops with high-speed links. Clients can connect to a server at any time using a browser to receive information about active applications. Furthermore, they can form or join collaboration groups and can (collaboratively) interact with one or more applications based on their capabilities. The client interaction portal supports two desktops – a local desktop represents the clients’ private view, while a virtual desktop presents a metaphor for the global virtual space to enable multiple users to collaborate with the aid of tools like whiteboards and chats. Application views (e.g. a plot) can be made collaborative by creating them in the virtual desktop or transferring them from the local to the virtual desktop. Session management and concurrency control is based on capabilities granted by the server. A simple locking mechanism is used to ensure that the application remains in a consistent state. DISCOVER is currently being used
An Architecture for Web-Based Interaction
1335
to provide interaction capabilities to a number of scientific and engineering applications2 including oil reservoir simulations, computational fluid dynamics and numerical relativity. Users
U
Database support
Session Archival
Thin Client
Interaction/Steering
U
Authentication/Security
App 2
U
Policy rule base
Simulation/Interaction/Viz. Broker
Master (Acceptor/Controller) (RMI/sockets/HTTP)
U
Viz./Computation
App 1
Application 2
Application 1
Interaction Agents
Servlets Obj 1 Obj 2 Remote Database
Local database
Fig. 1. Architectural Schematic of the DISCOVER Interactive Computational Collaboratory
4 Interaction and Collaboration Servers The middle tier of the DISCOVER system consists of a network of interaction and collaboration servers aimed at providing a web-based portal to executing highperformance applications. The servers build on Servlet [1] technologies to add a range of specialized capabilities to traditional web-servers. A key innovation of the server architecture is the use of reflection to provide an extensible set of services that can be dynamically invoked in an application specific manner. The overall architecture consists of a master acceptor/controller servlet and a suite of service handler servlets including interaction and steering handler, collaboration handler, security/authentication handler, visualization handler and session archival handler and database handler. The overall architecture of the interaction server is shown in Fig.1 and the key services are described below.
2
For example, see www.caip.rutgers.edu/TASSL/DISCOVER/ipars.html.
1336
Rajeev Muralidhar, Samian Kaur, and Manish Parashar
4.1 Collaborative Interaction and Steering Collaborative interaction is managed by the master servlet along with the interaction and collaboration handler servlets. The master is the central hub for all communication between the client and a server. Furthermore, it coordinates all registered applications and interaction sessions. Each application, on connection, registers with this servlet, which in turn spawns a dedicated application interaction broker. All client requests are classified and forwarded to the corresponding broker by the interaction handler. The broker uses an application proxy to locate application objects, discover object interaction/analysis interfaces, forward queries and commands, and aggregate information from distributed objects. Application responses and updates are multicast to the interested client group by the collaboration handler. Clients can form/join/leave collaboration groups at any time. On a client connection, the master provides the incoming client with a list of registered applications. The master also accepts, parses and validates client requests and redirects them to the relevant utility servlet. 4.2 Security, Authentication, and Access Control Security, client authentication and application access control is managed by a dedicated security and authentication handler. The current implementation supports two-level client authentication at startup; the first level is to authorize access to the server and the second level to permit access to a particular application. On successful validation of the primary authorization, the user is shown a list of the applications for which s/he has access capabilities. A second level authentication is performed for the application s/he chooses. On the client side, digital certificates are used to validate the server identity before the client downloads views. A Secure Socket Layer provides encryption for all communication between the client and the server. To enable access control all applications are required to provide list of users and their access privileges (e.g. read, modify). The application can also provide access privileges (typically read-only) to the “world”. This information is used to create access control lists (ACL). Each interaction request is then validated against the ACL before it is processed. 4.3 Application View Plug-Ins Application information is presented to the client in the form of application Views. Typical views include text strings, plots, contours and iso-surfaces. Associated with each of these views is a view plug-in that can be downloaded from the server at runtime and used to present the requested view to the user. The server supports an extendible plug-in repository and allows users to extend, customize or create new views and associated plug-ins.
An Architecture for Web-Based Interaction
1337
5 Application Control Network for Interaction and Steering The DISCOVER control network is composed of two components: (1) Interaction Objects that encapsulate interaction sensors and actuators, and (2) a Control Network consisting of distributed Interaction Objects and Interaction Agents. These components are described below. 5.1 Sensors/Actuators and Interaction Object Interaction objects extend application computational objects with interaction and steering capabilities, by providing them with co-located sensors and actuators. Computational objects are the data-structures/objects used by the application. Sensors enables the object to be queried whiles actuators allow it to be steered. Efficient abstractions are essential for converting computational objects to interaction objects especially when the computational objects are distributed and dynamic. In the DISCOVER system, this is achieved by deriving the computational objects from a virtual interaction base class of the DISCOVER Distributed Interaction Object library. The derived objects define a set of Views that they can provide and a set of Commands that they can accept. Interaction agents then export these views and commands to the interaction server using a simple Interaction IDL (Interface Definition Language). Interaction objects can be either local to a single computational node, distributed across multiple nodes, or shared between some or all of the nodes. Distributed objects have an additional distribution attribute that describes their layouts. DISCOVER interaction objects can be created or deleted during application execution and can migrate between computational nodes. Furthermore, a distributed interaction object can modify its distribution at any time. In the case of applications written in non-object-oriented languages such as Fortran, application data structures are first converted into computation objects using a C++ wrapper object. These objects are then transformed to interaction objects as described above. 5.2 The Control Network and Interaction Agents The DISCOVER control network (see Fig. 2) has a hierarchical cellular structure and partitions the processing nodes into a number interaction cells. The network is composed of (1) Discover Agents on each node, (2) Base Stations on each interaction cell and (3) an Interaction Gateway that connects to the interaction server and provides a proxy to the entire application. The number of nodes per interaction cell is programmable. The cellular control network is automatically configured at run-time using an underlying messaging environment and the available number of processors. Discover Agents present on each node maintain run-time references to all registered interaction objects on that node. The object references can change dynamically during program execution if data is migrated to handle load balancing. The control network ensures that object references are valid and refer to consistent data. At startup, each Discover Agent exports the registered objects’ interaction
1338
Rajeev Muralidhar, Samian Kaur, and Manish Parashar
information to its corresponding Base Station. Base Stations maintain a Cell Object Registry containing information about interaction objects for its interaction cell. Similarly, the Interaction Gateway maintains a Central Object Registry of interaction objects exported by all Base Stations. It is responsible for interfacing with the interaction server, delegating interaction requests to the appropriate interaction agents (Discover Agents and/or Base Stations), and collecting their responses. In the case of distributed objects, the Gateway additionally performs a gather operation for collating the responses arriving from the corresponding nodes.
Compute Node + Discover Agent
Internal Interactivity System
Interaction messages Interaction Cell
Base Station + Cell Object Registry
Interaction messages Base Station + Cell Object Registry
Interaction Cell Interaction messages Interaction Cell Interaction messages Interaction Cell
Base Station + Cell Object Registry
Interaction Gateway (with Java Virtual Machine)
Interaction Broker (Enhanced Java Web Server
Base Station + Cell Object Registry
Fig. 2. DISCOVER Control Network for Application Interaction and Steering
The Interaction Gateway creates a Java mirror of each registered interaction object interface using the Java Native Interface (JNI) [2]. It uses the proxy object pattern [14] to create a placeholder for the remote (possible distributed) interaction objects (the subject) and controls access to it. This innovative feature enables interaction objects to be easily exported to an interaction server using the Serializable interface of Java to the interaction web server.
An Architecture for Web-Based Interaction
1339
6 Conclusion and Future Work This paper presented the architecture of the DISCOVER web-based computational collaboratory. The current implementation allows thin-clients at the front-end, uses enhanced Java Servlets-based interaction web servers for the middle-tier and a control network of interaction agents and interaction objects at the back-end. The interaction servers enable clients to connect to, and collaboratively interact with registered applications using a conventional browser. Current work includes implementing the Java mirroring capability using JNI at the application’s interaction gateway and extending the middle tier to consist of a network of CORBA connected servers.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
Hunter J.: Java Servlet Programming. 1st edition, O’Reilly, California (1998). Gordon R.: Essential JNI: Java Native Interface. 1st edn. Prentice Hall, New Jersey (1998). Asbury B., Fox G., Haupt T., Flurchick K.: The Gateway Project: An Interoperable Problem Solving Environments Framework for High Performance Computing. http://www.osc.edu/~kenf/theGateway. Eisenhauer G., Schwan K: An Object-Based Infrastructure for Program Monitoring and Steering. 2nd SIGMETRICS Symposium on Parallel and Distributed Tools (1998). van Liere R., Harkes J., de Leeuw W: A Distributed Blackboard Architecture for Interactive Data Visualization. IEEE Viz. Conference (1998). Parker S.G., Johnson S.G.: SCIRun: A scientific Programming Environment for computational steering. Proceedings of Supercomputing (1995). Jain L.K.: A Distributed, Component-Based Solution for Scientific Information Management. MS Report, Oregon State University (1998). Bajaj C., Cutchin S.: Web based Collaborative Visualization of Distributed and Parallel Simulation, IEEE Parallel Symposium on Visualization (1999). NCSA Habenaro Home Page.: http://havefun.ncsa.uiuc.edu/habenaro. NCSA Software Development Division (1998). Tango Interactive.: http://www.webwisdom.com/tangointeractive. Raje R.R., Teal A., Coulson J, Yao S., Winn W., Guy III E.:CCASEE – A Collaborative Computer Assisted Software Engineering Environment. Proceedings of the International Association of Science and Technology for Development (IASTED) Conference (1997). Boyles M., Raje R., Fang S.: CEV: Collaboration Environment for Visualization Using Java RMI. Proceedings of the ACM Workshop on Java for High-Performance Network Computing (1998). Common Object Request Broker Architecture, http://www.omg.org. Gamma E., Helm R., Johnson R., Vlissides J.: Design Patterns, Addison Wesley Professional Computing Series (1994). Ribler R.R., Vetter J., Simitci H., and Reed D.: AutoPilot: Adaptive Control of Distributed Applications. 7th IEEE Symp. On High Performance Distributed Computing (1998). Vetter J., Schwan K: High Performance Computational Steering of Physical Simulations. IEEE International Parallel Processing Symposium (1997). Arkarsu E., Fox G., Furmanski W., Haupt T., Osdemir H., Oademir Z. O.: Building Web/Commodity base Visual Authoring Environments for Distributed Object/Component Applications – A Case Study Using NPAC WebFlow Systems. Vetter J., Schwan K: Progress: A Toolkit for Interactive Program Steering. Proceedings of the 1995 International Conference on Parallel Processing (1995), 139-149.
Computational Steering in Problem Solving Environments David Lancaster and Jeff S. Reeve Electronics & Computer Science, University of Southampton, Southampton SO17 1BJ, U.K. [email protected]
Abstract. A strong motive to build Problem Solving Environments (PSE’s) is the ability to interactively steer computations. We analyse the requirements that steering puts on the architecture of such a PSE and propose a design that does not separate the development and execution parts of the PSE. We describe a prototype implementation of this design based on the standardized software infrastructure of Java Beans and CORBA that enables a high degree of steering.
1
Introduction
The desire to solve ever larger and more complex scientific and engineering problems has been one of the main driving forces in computer development. The improvement in hardware performance has been phenomenal, but the software environment in which solution methods are constructed has also evolved: notably through high level languages and scientific and mathematical libraries. Problem Solving Environments (PSE’s) have been proposed [1] as a further stage in this development. A PSE is a software environment that provides support for all stages in both the development and execution of problem solving code. This paper is concerned with PSE’s that incorporate, and indeed emphasise, computational steering for manipulating the course of problem solving code as it proceeds. Computational steering implicitly puts requirements of interactive responsiveness on the architecture of a PSE. In the next section we analyse some possible PSE designs in terms of these requirements. We argue that a design that does not separate the development and execution parts of the PSE allows computational steering to fit better into the control flow management and is therefore favored.
2
PSE Architecture
The commonly agreed central requirements of a PSE are to support development and execution and can be designed in two parts separately responsible for each requirement. The development part consists of a sophisticated user interface which guides the user in building a solution to his problem. The output of this development stage is a taskgraph that describes the individual tasks that must A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1340–1344, 2000. c Springer-Verlag Berlin Heidelberg 2000
Computational Steering in Problem Solving Environments
1341
be run along with their submission order and data dependencies. Tasks consist of pieces of code and possibly parameters that tune their function. The taskgraph can be described in whatever format is convenient, a modern choice would be XML [2]. The execution part of the design takes the taskgraph and schedules it on whatever platforms are available in such a way as to satisfy all the dependency constraints encoded in the taskgraph. This design, which we shall call the “separated design”, forms the basis of many existing PSE’s. It is quite apparent that this separated design limits the ability to interact with the system at run time because the taskgraph becomes immutable once it has been passed to the execution stage. By contrast, the “combined design” integrates computational steering from the beginning and does not prejudge the kind of steering that may be necessary at run-time. The user may halt the execution, change the set of tasks and their connections, and restart the processes while preserving the state of their data. For example, a visualization module might be attached or a more accurate solver module substituted for a less accurate one. In the language of the separated design: the user is allowed to modify the taskgraph at run time. To understand the distinction between the two designs consider the management of control flow. In general, the control flow in a PSE is determined by two factors: data flow and steering. The data flow model is intuitive in the PSE context and is made manifest in the way that the components are connected together in the graphical composer of the development part of the PSE. In the separated design, the taskgraph is an abstract representation of this data flow, possibly with hooks for a steering controller. The taskgraph is dispatched by a separate execution system and the controller manages the steering part of the control as another separate system. In the combined design, the control flow is never abstracted away from the original form as determined by the user’s selection of modules and connection pattern in the graphical compositor. The components manage their own dispatching; as soon as one component has completed its task and generated its data, control passes to the next component. Computational steering fits in naturally because it is performed at the level of the graphical compositor using the same tools that were used to set up the original problem and no new control structure is needed.
3
Prototype PSE
Software developments for commodity markets over the last few years have provided us with the some basic tools that can be used to construct PSE’s. There are strong arguments to use these software layers in designing software for scientific and engineering purposes, notably: they are cheap (often with free implementations), widely available and most importantly they are standardized. Our prototype PSE uses JavaBeans [3] and CORBA [4]. These are appropriate for a prototype that is intended to be used with medium sized platforms such as clusters [5] small shared memory machines and individual workstations. A more highly featured PSE that was intended to be used with more substantial
1342
David Lancaster and Jeff S. Reeve
computing resources would also need software layers that connect to queuing systems and databases. JavaBeans fulfill the need for a sophisticated user interface that allows graphical programming by connecting computational components. CORBA provides a platform independent way of distributing the computations performed by the PSE so as to take advantage of high performance machines and software implementations. To illustrate the use of the PSE and to make the following discussion more concrete we have implemented some simple components that perform operations that diagonalize a matrix [6]. These components include one to generate the matrix and components Householder and QL that manipulate the form of the matrix. The beans representing these components have interfaces described in the BeanInfo class that prescribe how they can be wired together in the builder tool. Besides listing any parameters, this class lists the type and form of the inputs and outputs that the component expects. The JavaBean code is essentially limited to describing these interfaces and the CORBA client code needed to call the computational module residing on a server. For this reason we call these beans “hollow”. In order to avoid unnecessary data movement and thereby improve performance, the form of the inputs/outputs are not explicit matrices, but are CORBA references to data objects that contain the matrices. The CORBA part of the prototype development requires an IDL for the system including the data objects for the matrices. This IDL is too long and detailed to include in a paper but is available on http://gather.ecs.soton.ac.uk/PSE/Docs. The CORBA servers have evolved since the beginning of this project and now provide facilities to manage the life-cycle of computational components. We employ a performance enhancement that makes non-blocking CORBA calls to hide latency following a standard CORBA technique. Consider for example the Householder component which accepts a symmetric matrix and generates a tridiagonal matrix. When this bean starts to run, it first contacts the scheduler which provides an object reference on some remote machine to the computational class that implements the householder transformation. The bean then uses this reference to submit the object reference of the symmetric matrix data class and control then passes back to the bean which then waits. Meanwhile, the Householder object uses the matrix reference to transfer the matrix data to its local address space and then proceeds with the computation, finally creating a new matrix object in which to store the resulting tridiagonal matrix. The object reference to this tridiagonal matrix object is returned to the bean indicating completion of the job. For the purposes of this work we implemented a scheduler that allows several versions of each component to exist on different machines and selects which one to use on request. This scheduler is dumb in the sense that the algorithm for selection is extremely simple, being either random or based on prior user choices. It does however provide the same kind of interface expected in a more sophisticated version. CORBA initialization employs a nameserver which is used
Computational Steering in Problem Solving Environments
1343
to register servers that are started by hand on whatever remote machines are available. The scheduler contacts the nameserver to obtain all the information about which components are available on which machines. The control flow information has its origin in the act of wiring JavaBeans together and remains encapsulated in the builder tool. It is never abstracted into a taskgraph. As soon as one component has completed its task and generated its data, control passes to the next bean which dispatches a request to the CORBA object that implements the computational work of the component. The flow of control between one component and another takes place using the standard bean event mechanism and PropertyChangeEvents trigger this flow. For example, when the householder bean in the example above, obtains the reference to the tridiagonal matrix object, it fires a PropertyChangeEvent that is picked up by the next bean, in this case the QL bean, that has been wired to listen for such events. The fact that control flow resides entirely in the standard beans on the builder tool is the essential aspect that realizes the combined design and allows the computational steering freedom that we have emphasised throughout.
4
Conclusion
We have analysed the design of a PSE intended to provide good support for computational steering. Then by implementing a prototype incorporating this design we have shown that JavaBeans and CORBA provide an appropriate software basis for small/medium sized PSEs, allowing a high degree of run-time freedom to steer the computation. We have in mind several improvements to the prototype implementation which would make this PSE a more useful and robust tool. To extend the type of steering that can be done, we would like to construct our own builder tool that would allow beans to be deleted or substituted at run time. To allow the freer introduction and sharing of new components they should be described in some language such as XML [7] along with tools that use this description to create most of the beans and BeanInfo classes as well as providing a skeleton for the computational part of the code. The other major area for improvement is in the scheduler, which should monitor load and provide more sophisticated selections on the basis of some knowledge of the intended taskgraph. However, a significant conclusion of this paper is that the scheduler should not be allowed to gain control of dispatching – or the freedoms of computational steering will be forfeited.
Acknowledgments DL would like to acknowledge discussions with Peter Lockey and Matthew Shields. This work was supported by a UK EPSRC grant entitled, Problem Solving Environments for Large Scale Simulations.
1344
David Lancaster and Jeff S. Reeve
References 1. E. Gallopoulos, E. Houstis and J.R. Rice. Problem Solving Environments for Computational Science, IEEE Comput. Sci. Eng., 1, 11-23, 1994. E. Gallopoulos, E. Houstis and J.R. Rice. Workshop on Problem Solving Environments: Findings and Recommendations, ACM Comp. Surv., 27, 277-279, 1995. 2. World Wide Web Consortium, Extensible Markup Language (XML) http://www.w3.org/XML/ 3. http://java.sun.com/beans/ 4. The CORBA specification is controlled by the Object Management Group http://www.omg.org/ 5. D. Ridge, D. Becker, P. Merkey and T. Sterling. Beowulf: Harnessing the Power of Parallelism in a Pile-Of-PC’s, Proc. 1997 IEE Aerospace Conference. See the Beowulf Project page at CESDIS: http://cesdis.gsfc.nasa.gov/linux/beowulf/beowulf.html 6. D. Lancaster and J.S. Reeve. A Problem Solving Environment Based on Commodity Software, Proc. HPCN2000. Lecture Notes in Computer Science 1823, Ed. M. Bubak, H. Afsarmanesh, R. Williams and B. Hertzberger. 7. O.F. Rana, M. Li, D.W. Walker and M.Shields. An XML based Component Model for Generating Scientific Applications and Performing Large Scale Simulations in a Meta-Computing Environment, available from: http://www.cs.cf.ac.uk/PSEweb/
Implementing Problem Solving Environments for Computational Science Omer F. Rana1 , Maozhen Li1 , Matthew S. Shields1 , David W. Walker2 , and David Golby3 1
Department of Computer Science, University of Wales, Cardiff, POBox 916, Cardiff CF24 3XF, UK 2 Computational Sciences Section, Oak Ridge National Laboratory, PO Box 2008, Oak Ridge TN 37831-6367, USA 3 Department of Mathematical Modelling, British Aerospace Systems, Sowerby Research Center, PO Box 5, Filton, Bristol, BS34 7QW, UK
Abstract. A Problem Solving Environment (PSE) should aim to hide implementation and systems details from application developers, to enable a scientist or engineer to concentrate on the science. A PSE is, by definition, problem domain specific, but the infrastructure for a PSE can be problem domain independent. A domain independent infrastructure for a PSE is described, followed by two application dependent PSEs for Molecular Dynamics and Boundary Element codes that make use of our generic PSE infrastructure.
1
Introduction
A Problem Solving Environment (PSE) is a complete, integrated computing environment for composing, compiling, and running applications in a specific area [3]. PSEs have been available for several years for certain specific domains, but most of these have supported different phases of application development, and cannot be used cooperatively to improve a scientists’ productivity, primarily due to the lack of a framework for tool integration and ease-of-use considerations. Extensions to current scientific programs such as Matlab, Maple, and Mathematica are particular pertinent examples of this scenario. Developing extensions to such environments enables the reuse of existing code, but may severely restrict the ability to integrate routines that are developed in other ways or using other applications. Multi-Matlab [4] is an example of one such extension for parallel computing platforms. A PSE must contain: (1) application development tools that enable an end user to construct new applications, or integrate libraries from existing applications, (2) development tools that enable the execution of the application on a set of resources. In this definition, a PSE must include resource management tools, in addition to application construction tools, albeit in an integrated way. Component based implementation technologies provide a useful way of achieving this objective, and have been the focus of research in PSE infrastructure. Based on the types of tools supported within a PSE, we can identify two types A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1345–1349, 2000. c Springer-Verlag Berlin Heidelberg 2000
1346
Omer F. Rana et al.
of users: (1) application scientists/engineers interested primarily in using the PSE to solve a particular problem (or domain of problems), (2) programmers and software vendors who contribute components to help achieve the objectives of the category (1) users. The PSE infrastructure must support both types of users, and enable integration of third party products, in addition to application specific libraries. Many of these requirements are quite ambitious, and existing PSE projects handle them to varying extents. The component paradigm has proven to be a useful abstraction, and has been adopted by many research projects, making use of existing technologies such as CORBA, JavaBeans and DCOM/COM. Although a useful abstraction for integration, performance issues when wrapping legacy codes in Fortran as components have not been addressed adequately. Automatic wrapper generators for legacy codes that can operate at varying degrees of granularity, and can wrap the entire code or sub-routines within codes automatically, are still not available. Part of the problem arises from translating data types between different implementation languages (such as complex numbers in Fortran), whereas other problems are related to software engineering support for translating monolithic codes into a class hierarchy. Existing tools such as Fortran to Java translators cannot adequately handle these specialised data types, and are inadequate for translating large application codes, such as the Lennard-Jones molecular dynamics application discussed in section 2. For PSE infrastructure developers, integrating application codes provides one goal, the other being the resource management infrastructure to execute these codes. The second of these can involve workstation clusters, or tightly coupled parallel machines. We therefore see a distinction between these two tiers of a PSE, (1) a component composition environment, (2) a resource management system. A loose coupling between these two aspects of a PSE will be useful where third party resource managers are being used, whereas a strong coupling is essential for computational steering or interactive simulations.
2
Applications Using the PSE Infrastructure
We describe two applications that makes use of our generic component based PSE infrastructure described in [8]. Molecular Dynamics: The molecular dynamics application models a LennardJones fluid. It is written in C, and was initially wrapped as a single CORBA object. A Java interface component is combined with a component that wraps the executable code, enabling input from the user to be streamed to the executable, and subsequently to display output results to the user. The CORBA object makes use of an MPI runtime internally, and is distributed over a cluster of workstations. The intra-communication within the object is achieved via the MPI runtime, while the interaction between the client and the object is via the ORB. An application developer does not need to know the details of how the code works, or how to distribute it over a workstation cluster. The wrapper therefore provides the abstraction needed to hide implementation details from
Implementing Problem Solving Environments for Computational Science
1347
an application user, requiring the developer to automatically handle exceptions generated from the CORBA and MPI system. The molecular dynamics code was subsequently sub-divided into four CORBA objects, to improve re-usability, and included: (1) an Initialization object to calculate particle velocities and starting positions, (2) a Moveout object to handle communication arising from particle movement and “ghost” regions, (3) a Force object which calculates the force between molecules, and constitutes the main computational part of the code, (4) an Output object that generated simulation results for each time step. These four objects were managed by a Controller object, which coordinated the other four objects. A user now has more control over component interaction, but still does not know how the MPI runtime is initialised and managed within each object. The application developer can construct a molecular dynamics application by connecting components together. Each component is self-documenting based on its XML interface, and component properties can be investigated using an Expert Advisor [7]. BE2D: The BE2D code is a 2D boundary element simulation code for the analysis of electromagnetic wave scattering. The main inputs to the program are a closed 2D contour and a control file defining the characteristics of the incident wave. The original code was written in Fortran. To make the code easier to use within the PSE and also to provide runtime speed comparisons it was decided to convert the code into Java. The use of Fortran to Java converters was considered but abandoned fairly quickly because of the lack of support for complex numbers. Complex numbers caused a number of problems in converting the code, not least because of the fact that they are primitive types in Fortran, but are not directly supported in the standard Java API. A third party implementation of complex numbers for Java was found and used, from Visual Numerics called JNL [5]. Complex number arithmetic proved to be more than just a straight translation from the Fortran code to its Java equivalent. As complex numbers in Java are objects, and Java does not yet support operator overloading, the complex number objects have to use their operator methods, add, multiply, subtract and divide. These methods are either unary or binary, taking a single input parameter which is used to modify the object calling the method or taking two input parameters on a static class method that creates a new result without any modification side effects. Thus calculations involving more than an arbitrary number of parameters become far more complicated in Java than their Fortran equivalents. The converted BE2D was subsequently used as a single component within the PSE. The solver is combined with a graph generator (JChart [6]) as a third party component. The output generated from the code is illustrated in figure 1. If the complete BE2D solver is wrapped as a CORBA object, comprising a single Fortran executable unit, then the interface contains the name of the solver, and a single input file. If the data is to be streamed to the solver, a component that can convert a file (local or at a URL) into a stream is placed before the component.
1348
Omer F. Rana et al.
Fig. 1. Output of the BE2D solver
3
Conclusion and Future Work
Current advances in networking and distributed object technologies provide the infrastructure to support the creation of general purpose PSEs. The emphasis in developing PSEs should be on creating an environment that is easy to use, and intuitive for an application/domain expert. A computational scientist should therefore not have to configure hardware resources in order to undertake molecular dynamics research, for instance. An important lesson learnt from our work is the necessity to understand the impact of specialised data structures on the usage and performance of legacy codes, when these are wrapped as components within a PSE. The Java programming language can be useful in integrating various components, but it still suffers from the absence of data structures that are essential for scientific codes. Current efforts are under way within the JavaGrande forum [1] to address some of these issues, and the result of these efforts will be significant to future PSE infrastructure.
References [1] The JavaGrande Forum. See web site at: http://www.javagrande.org/. [2] I. Foster and C. Kesselman. Globus: A metacomputing infrastructure toolkit. Int. Journal of Supercomputing Applications, 11(2), 1997. [3] E. Gallopoulos, E. N. Houstis, and J. R. Rice. Computer as Thinker/Doer :ProblemSolving Environments for Computational Science. IEEE Computational Science and Engineering, 1(2), 1994. [4] Vijay Menon and Anne E. Trefethen. MultiMATLAB: Integrating MATLAB with High-Performance Parallel Computing. Proceedings of SuperComputing97, 1997. [5] Visual Numerics. JNL: A numerical library for Java. See web site at: http://www.vni.com/products/wpd/jnl/. [6] Roberto Piola. The JChart package. See web site at: http://www.ilpiola.it/roberto/jchart/.
Implementing Problem Solving Environments for Computational Science
1349
[7] M. Shields, O. F. Rana,D. W. Walker, M. Li, and D. Golby, A Java/CORBA based Visual Program Composition Environment for PSEs, Concurrency: Practice and Experience (in press), 2000 [8] D. Walker, M. Li, O. Rana, M. Shields, and Y. Huang. The Software Architecture of a Distributed Problem Solving Environment. Technical report, Oak Ridge National Laboratory, Computer Science and Mathematics Division, PO Box 2008, Oak Ridge, TN 37831, USA, December 1999. Research report no. ORNL/TM-1999/321.
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1 Matthias Brehm, Reinhold Bader, Helmut Heller, and Ralf Ebner Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschaften, Barer Straße 21, 80333 M¨ unchen, Germany {Brehm, Bader, Heller, Ebner}@lrz.de http://www.lrz-muenchen.de/services/compute/hlrb
Abstract. The Leibniz-Rechenzentrum in Munich has started operating a 112-node Hitachi SR8000-F1 with a peak performance of 1.3 Teraflops in the second quarter of 2000, the fastest computer in Europe. In order to make use of the full memory bandwidth and hence to obtain a significant fraction of the peak performance for memory intensive applications, the compilers offer preload and prefetch optimization strategies to pipeline load/store operations, as well as automatic parallelization across the 8 processors contained in every node. The nodes are connected by a conflict-free crossbar, enabling efficient communication via standard message-passing interfaces. An overview of the innovative architectural concepts is given. We demonstrate to which extent the capabilities of the compiler to automatically pseudovectorize/parallelize typical application code are sufficient to produce well-performing code.
1
Aiming for Top Level Computing
In the first quarter of 2000, the Leibniz-Rechenzentrum (LRZ) Munich installed a Hitachi SR8000-F1 intended to serve as Top-Level Compute Server in Bavaria (the German acronym HLRB will be used in the following); this machine will be again enlarged by approximately half its present computing power in a second installation phase in 2Q2002. In installation Phase I the system consists of 112 nodes. Each pseudo-vector node contains 9 CPUs, 8 of which are available for computational tasks, and 8 Gigabytes memory which is accessible from the processors in a shared-memory model. The CPUs are similar to the IBM Power Architecture, with proprietary extensions added by Hitachi (cf. Section 2.1). Since the processors are operated at a frequency of 375 MHz and 4 floating point operations can be executed per cycle, each pseudo-vector node yields 12 GFlops peak performance. Thus the HLRB has a peak performance of 1.344 TFlops. The ninth processor on each node is needed as a service processor. The nodes of the SR8000-F1 are inter-connected via a three-dimensional crossbar with a bi-directional bandwidth of 2x950 MB/s between two nodes and a hardware latency of about 5 microseconds. Further details of the HLRB hardware are given in Tables 1 and 2 below. A. Bode et al. (Eds.): Euro-Par 2000, LNCS 1900, pp. 1351–1361, 2000. c Springer-Verlag Berlin Heidelberg 2000
1352
Matthias Brehm et al.
The LINPACK performance value of the HLRB is 1035 GFlops and a sustained application performance has been measured to 450 GFlops. Hence, LRZ is currently operating the fastest computer within Europe. Usage of the HLRB will be open to German research projects which need high sustained performance and are presently not feasible on any other existing computing platform. Resources will be allocated to individual projects after a peer review process. Vectorizable Codes will be preferred, however the SR8000 architecture is sufficiently flexible that the system may be used in MPP mode as well.
2
The Innovative Architecture of the SR8000-F1
The architecture of the SR8000-F1 allows the usage of the vector programming paradigm and the scalar SMP-Cluster programming paradigm on the same machine. This is achieved by combining the superscalar RISC CPUs into a virtual vector CPU. In a traditional vector CPU the vectorized operations are executed by a vector pipe which delivers one or more memory references per cycle to the CPU. On the Hitachi SR8000-F1 the vectorizable operations are distributed among the 8 effectively usable CPUs of a node (”COMPAS”, COoperative MicroProcessors in single Address Space); furthermore in case of memory-bound computing, specific memory references can be loaded into the registers or the caches some time ahead of actual use (”PVP”, Pseudo-Vector-Processing). These two properties of the SR8000-F1 nodes especially contribute to the high efficiency obtained in comparison to other RISC systems.
Number of SMP-Nodes CPUs per Node Number of Processors Peak Performance per CPU Peak Performance per Node Peak Performance SR8000 LINPACK Performance of the whole System Performance from main memory (most unfavourable case) Memory per Node Memory of total system Aggregated Disk Storage Bidirectional Communication bandwidth using MPI
Phase I Configuration 1Q2000 112 8 (+1 Service) 112*8 = 896 1.5 GFlop/s 12 GFlop/s 1344 GFlop/s 1035 GFlop/s
Phase II Configuration 2Q2002 168 8 (+ 1 Service) 168*8 = 1344 1.5 GFlop/s 12 GFlop/s 2016 GFlop/s to be measured
163.5 GFlop/s
244 GFlop/s
8 GByte 928 GBytes 7.4 TBytes 2x950 MByte/s (Hardware: 1 GByte/s)
8 GByte 1344 GBytes 10 TBytes 2x950 MByte/s (Hardware: 1 GByte/s)
Table 1. Hardware Overview of LRZ’s HLRB.
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1
1353
Processor and Memory Characteristics Frequency and Processor Cycle 375 MHz (2.67 nanoseconds) Maximum Number of Operations per Cycle 4 Number of Floating Point Registers 160 (Global: 32, Slide: 128) Number of Integer Registers 32 Data Cache Size 128 KB (write through, 4way set associative, direct mapped) DCache line size 128 Bytes DCache bandwidth to registers 32 Bytes / cycle Memory Frequency and mem-cycle 250 MHz (4 ns) Maximum Number of Loads from Memory 16 Bytes / mem-cycle Bandwidth to Memory per Processor 4 GB/s (32 GB/s peak for 8-processor node)
Table 2. Properties of the processor and memory system used by Hitachi in the Phase I installation. It has not yet been decided whether a more advanced CPU model will be used in the Phase II upgrade.
2.1
Pseudo-Vector-Processing (PVP)
Hitachi’s extensions to the IBM POWER instruction set, which improve the memory bandwidth, alleviate the memory bottleneck, which is the main deficit of RISC-based High Performance Computing. This property, called Pseudo-Vector Processing by Hitachi, may be used by the compiler to obtain data either directly from memory via preload or via prefetch through the cache, depending on how the memory references are organized (see Fig. 1 below). The concept of PVP may be illustrated by the following example loop: Using prefetch operations, which may be overlapped with the floating point operations, one obtains the sequence shown in the right part of Figure 2. Prefetch is not very efficient when the main memory is accessed non-contiguously because the prefetched cache line may contain unnecessary data. To improve this situation the preload mechanism was implemented. Preload transfers element data directly to the registers as illustrated in Figure 3. The physical registers are mapped to logical addresses via a sliding window technique. A special instruction (sliding window step) is used to update the slide window base value which is held in a special purpose register. DO I=1,N A(I) = B(I) + C(I) ENDDO
2.2
Cooperative Micro Processors in Single Address Space (COMPAS)
COMPAS enables the automatic distribution of computational work of a loop among the 8 CPUs of an SMP node by the compiler (autoparallelization) and the accompanying hardware support for synchronization. COMPAS may also
1354
Matthias Brehm et al.
transfer of cache lines
Memory transfer of element data Prefetch Cache
Preload
Load
160 Floating Point Registers
Slide Window (for program use)
Arithmetic Unit
Fig. 1. Pseudo-vectorization: Prefetch and Preload. Without PVP
With PVP Prefetch B(1) Prefetch C(1) ... Prefetch C(K)
load reg1 B(1) load reg1 C(1) add reg1 reg2 -> reg3 store reg3 A(1)
Prefetch the data to be used in the following iteration
Use the prefetched data to calculate B(*) + C(*)
Loop iteration
Loop iteration
Store the calculation A(*)
Fig. 2. Loop structure without and with PVP Prefetch. Example Loop: do i=1,n s = s + a(i) end do
mapping from logical register numbers to physical register numbers is slided (Slide Window Step)
Software pipelined Code: FR32 = PLD a(1) FR34 = PLD a(2) FR36 = PLD a(3) FR38 = PLD a(4) do i=1,n-5 s = s + FR32 slide 2 end do s = s + FR32 s = s + FR34 s = s + FR36 s = s + FR38
PLD time
PLD PLD PLD ADD slide
PLD ADD ADD
Fig. 3. Loop Structure with PVP Preload.
ADD ADD
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1
1355
be utilized by codes which use the nodes as 8-way SMP-nodes via OpenMP, since version 1.0 of the OpenMP standard is implemented as part of the Hitachi Fortran Compiler.
3
Benchmark Results and Principles for Code Optimization
The most important criterion for evaluation of the offered machines was not the peak performance but the actually obtained ”sustained” performance for a suite of application benchmarks. Furthermore, several additional tests were performed to obtain a measure of how the hardware performs in the least favorable situations or to evaluate scaling of MPI codes to the largest possible problem size. The examples discussed in the following subsections will also provide insights on the principles of optimizing code for the SR8000. 3.1
Memory Throughput
STREAM Benchmark This program (written by John D. McCalpin) is used to evaluate the memory bandwidth of a node. The following loops are performed: Copy
DO J=1,N C(J)=A(J) END DO
Scale
DO J=1,N B(J)=S*C(J) END DO
Add
DO J=1,N C(J)=A(J)+B(J) END DO
Triad
DO J=1,N A(J)=B(J)+S*C(J) END DO
The vector length N used in this case was 19,121,111, corresponding to a memory usage of 437 MB for the program, and ensuring that it is really the transfer from/to memory that is measured. The memory bandwidth is plotted in Figure 4 for all types of loops indicated above. A comparison with other vendor’s hardware is provided in Table 3. The machine balance is obtained as Platform
SR8000-F1 R12000 (SGI) Pentium III Cray C90 NEX SX-5 IBM Nighthawk1
Bandwidth (MB/s) 22311 811 396 103812 583069 3872
Peak Machine Remarks Performance Balance (MFlops) (Flop/Word) 12000 4.3 8 Processors, 375 MHz 2400 23.7 4 Processors, 300 MHz 500 10.1 1 Processor, 500 MHz 15360 1.2 16 Processors 128000 1.8 16 Processors 7104 14.7 8 Processors, 222 MHz
Table 3. Overview of Triad memory bandwidth for various platforms.
the quotient of peak performance and bandwidth, the latter being expressed in 8Byte words; lower numbers are better, as far as memory-intensive (out-of-cache) computing is concerned. Typically, RISC-based systems are at least an order of
1356
Matthias Brehm et al.
magnitude worse than the specialized vector processors. However, Hitachi has nearly managed to bridge this gap by having a better ratio of memory cycle to processor cycle to start with, as well as being able to access memory from all processors in the SMP node simultaneously without too high losses: Of the 32 GBytes/s per node naively calculated from the single-processor bandwidth of at least 22 GByte/s can actually be obtained. In order to see how the memory bandwidth scales with the number of processors used, an OpenMP parallelized version of the STREAM benchmark was run. Figure 4 shows the efficiency as a function of the number of threads for the various loop types. The peak bandwidth for 1 processor is assumed to be 4 GB/s (cf. Table 2). One ob-
measured Bandwidth(n) E(n) = n · peak Bandwidth(1)
STREAM: Memory bandwidth scaling (OpenMP)
0.95
Triad Copy Scale Add
0.9
Efficiency
0.85
0.8
0.75
0.7
0.65
0.6
1
2
3
4
Threads
5
6
7
8
Fig. 4. Memory band-width as measured in STREAM benchmark serves a degradation for increasing numbers of threads. The differences between Triad/Add and the other two tests are accounted for by a difference between Load and Store: Triad/Add involves 2 Loads and 1 Store, while Scale/Copy has only 1 Load and 1 Store. However it is not entirely clear why there is a measurable difference between Scale and Copy. It must be remarked that other RISC SMP machines degrade far more than the SR8000 node: On 8-way systems one can expect at most 40% of peak bandwidth. In the top ten list kept at http://www.cs.virginia.edu/stream/top10/Bandwidth.html the SR8000F1 (as well as its predecessor) would be ranked at position 7. Parallelization and Pseudo-vectorization for Triads Performing Triads for variable vector length yields information not only about the memory throughput, but of the complete register/cache/memory system. Furthermore, some of the compiler’s capabilities to automatically parallelize and pseudovectorize code are investigated. Figure 5 shows the per-formance of the (3 load + 1 store, 2 operation) triad
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1
1357
as a function of vector length for the four execution modes possible on the SR8000: 1) COMPAS-parallel and pseudo-vectorized, 2) COMPAS-parallel without pseudo-vectorization, 3) Non-parallel, but pseudovectorized, 4) Neither parallel nor pseudo-vectorized. For these measurements, the desired effect was obtained within a single program unit by inserting appropriate compiler directives. Looking at case 3) first, one observes uniformly high performance of up to 490 MFlops until the Level-1 Cache is exhausted at approximately n = 4000. After that, performance is governed by the memory throughput, where PVP achieves around 230 MFlops in the n → ∞ limit. Using 8 IPs in parallel allows for eightfold cache usage, hence the vector-like range ends at n=30000, where a performance of 3360 MFlops is reached, the 6.86-fold of what is obtained on a single IP. The vector-like characteristic of the performance for small n is due to the domination of COMPAS startup times for short vector lengths. The size of the cache performance window strongly depends on the particular loop kernel. Very fat loop kernels may lead to register spill and hence may require to be split up, too thin kernels are fused or suitably unrolled. All of this is automatically performed by the compiler. The n → ∞ performance achieved here is 1410 MFlops, which is the 6.13-fold value of a single IP. Hence, due to PVP the triad achieves more than 40% of its maximum performance even when leaving the cache performance window, while without PVP one obtains the RISC-typical value of around 7 %. This is of very high importance for memory intensive computing tasks in technics and science. DO I=1,N A(I) = B(I)*C(I) + D(I) END DO
Contiguous Triads
3500
Parallel, vectorized Parallel, not vectorized Nonparallel, vectorized Nonparallel, not vectorized
3000
MFlops
2500
2000
1500
1000
500 0 100
1000
100000 10000 Vector Length
1e+06
1e+07
Fig. 5. Performance of Contiguous Triads as function of vector length.
3.2
Scalability of MPI Programs
Part of the LRZ benchmark suite was concerned with scalability studies. The programs
1358
Matthias Brehm et al.
– FT (Fast Fourier Transform from NAS Parallel Benchmarks) – MG (basic Multigrid Algorithm from NAS Parallel Benchmarks) – HRD (ScaLAPACK call for Hessenberg transformation) were performed on the SR8000-F1, the results of which are presented in the following. The Class C 512x512x512 three dimensional Fast Fourier Transform and the Class C Multigrid Test were performed in COMPAS mode on an increasing number of nodes. The HRD Test was performed with a fixed number of nodes but varying matrix sizes. The tests generally showed high scalability and very good performance. From the programmers point of view, it is advantageous that one has to deal only with a relatively small number of nodes instead of the eightfold higher number of processors. Benchmark Nodes Processors Performance MPI Efficiency (GFlops ) (relative to 4 nodes) FT 4 32 15.5 1.00 8 64 29.5 0.95 16 128 56.9 0.92 32 256 109.8 0.89 MG 4 32 14.8 1.00 8 64 27.7 0.93 16 128 46.9 0.79 32 256 72.3 0.61 Benchmark Nodes Processors Performance Matrix Size HRD 32 256 11.8 1000 x 1000 32 256 52.8 10000 x 10000 32 256 134.0 30000 x 30000
3.3
Case Studies for the Hybrid (COMPAS/OpenMP + MPI) Programming Paradigm
Parallel Vector Times Matrix As an example for the hybrid programming model, we are going to look at a simple implementation of vector-matrix multiplication. Using MPI for the distributed memory version, there are only few changes from the serial version. The structure of the code is shown in Figure 6 (for the pure MPI version the OpenMP directives are just ignored). The performance for matrix size n=10000 and n=40000 is given in Figure 7. It is obvious that the hybrid version performs much better than the pure MPI version. We first thought that this was due to the reduced amount of MPI communication. However, further examination showed that not only the communication part had shorter execution time, but also the computational part. The latter effect is caused by the increased vector length in the algorithm. For an algorithm with higher ratio of MPI-communication to computation the differences will be even more marked. Parallel Matrix Multiplication Parallelization of Matrix Multiplication via MPI uses a ringcast scheme for matrix blocks, where the block size is chosen
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1
Computation
1359
Communication (MPI_REDUCE_SCATTER)
B m = vector length per node m’ = m/(procs per node) = vector length per processor
Node 0
a
m
Summation of local results
Node 1
+
* = c
Node 2
m’
Node 3
!$OMP PARALLEL !$OMP DO PRIVATE(J,L) DO J=1,N SX(J) = 0.0D0 DO L=1,M SX(J) = SX(J) + A(L)*B(L,J) END DO END DO !$OMP ENDDO NOWAIT !$OMP END PARALLEL CALL MPI_REDUCE_SCATTER(...)
Redistribute (scatter) result
Fig. 6. Code Structure for parallel vector times matrix multiplication. Vector x Matrix (veclength 10K and 40K) 40
Gflop/s
30
20
10
0 0
1
2
3
4
5
6
7
8
9
Num ber of Nodes (1 Node=8 Procs)
hybrid 10K pure mpi 10K
hybrid 40K pure mpi 40K
Fig. 7. Performance for vector by matrix multiplication.
optimally for a given cartesian processor grid. Given a block (IB, JB) of matrix C situated on a particular processor P, the required blocks of matrices A and B are pipelined through this processor in the course of the calculation. Between the communication steps, a normal DGEMM-call is performed. One can expect a nearly linear speed-up provided the block size is chosen large enough
1360
Matthias Brehm et al.
to essentially circumvent MPI latency. For this test a fixed amount of memory per node was used; the matrix dimension correspondingly scales upward with the square root of the number of nodes used. Figure 8 shows the dependence of Parallel Matrix Multiply
900
COMPAS MPP (no changes) MPP (optimized) MATRIX/MPP
800 Performance (GFlops)
700
600 500 400 300 200
Nodes in X
100 90 8 7 6 5 4 3 2 1
0
20
40
Nodes
60
80
100
Fig. 8. Performance of parallel matrix multiplication. performance on the number of nodes in various situations in the upper box, for the COMPAS case the lower box gives information about the node layout in the cartesian grid; e. g., for 21 nodes a 3x7 grid was used. In the COMPAS case, scaling is reasonably linear; for a yet unknown reason, square grids (4x4, 5x5, . . . ) appear to work particularly badly and should be avoided, at least with this particular communication pattern. Memory per node was 1.6 GByte. For the MPP (intra-node MPI) case, where 200 MByte of memory were used to obtain the same memory footprint per node as in the COMPAS case, runs with usually up to 256 processors (corresponding to 32 nodes) were performed. One obtains the very bad performance shown by the lowest curve if one simply recompiles the COMPAS code in scalar mode, using the scalar version of BLAS provided by Hitachi. The reason for this is simply that the latter library was not properly optimized at the time the tests were performed; as the ”matmul gemm” curve in Figure 9 shows, the code apparently did not optimally reuse the cache. Since the LRZ hand-coded routine (”matmul opt2”) works best for matrix sizes around 60-80, a smaller block size was chosen for a further run of the program yielding the second-lowest curve in Figure 10, which shows an improvement by a factor of 2.5. However, Hitachi also provides a proprietary library of highly optimized routines, MATRIX/MPP. Use of these – with a large block size – yields a performance comparable to COMPAS also for the internode case, as shown in the
Pseudovectorization, SMP, and Message Passing on the Hitachi SR8000-F1 Performance in scalar mode
1200
matmul_gemm matmul_opt2 matmul_j5i4kb matmul_j4i5kb matmul_f90matmul best_fortran
1000
Performance in MFlops
1361
800
600
400
200
0
1
10
100 Vector Length
1000
Fig. 9. Non-parallel (single-IP) matrix multiplication. The following variants are illustrated: 1. matmul gemm: Hitachi BLAS, 2. matmul opt2, matmul j5i4kb, matmul j4i5kb: blocking and loop unrolling done by hand, 3. matmul f90matmul: Fortran 90 intrinsic, 4. best fortran: Hitachi’s MATRIX/MPP implementation.
third-lowest curve of Figure 8. The drawback of MATRIX/MPP is that there is a different API and presumably more working space has to be used.
4
Conclusion
Hitachi’s SR8000-F1 installation easily manages to provide the computational power demanded by LRZ’s requirements and Hitachi’s own commitments. Our first tests indicate that generally more effective usage of the machine may be made by using the COMPAS mode as opposed to intra-node MPI (MPP-mode). The automatic parallelization features available with the Fortran and C compilers make it a relatively easy task to optimize high performance computing code.
5
Further Reading and Details
A Superscalar RISC Processor with 160 FPRs for Large Scale Scientific Processing: http://www.lrz-muenchen.de/services/compute/hlrb/system-en/Iccd99.pdf Node Architecture and Performance Evaluation of the Hitachi SR8000: http://www.lrz-muenchen.de/services/compute/hlrb/system-en/NodeArch.pdf An overview of the Hitachi SR8000-F1 by Hitachi may also be found at: http://www.hitachi-eu.com/hel/hpcc Some of the STREAM results were taken from: http://www.cs.virginia.edu/stream/top10/Bandwidth.html
Index of Authors
Aberdeen, Douglas, 980 Acquaviva, Jean-Thomas, 539 Adle, Roxane, 340 Afrati, Foto, 288 Agha, Gul A., 1029 Agrawal, Divyakant, 427 Agrawal, Gagan, 625 Ahmed, Nawaaz, 368 Ahrem, Regine, 1315 Aiguier, Marc, 340 Al-Sadi, Jehad, 935 Alcover, Rosa, 909 Almeida, Francisco, 320 Antoniu, Gabriel, 1039 Arnau, Vicente, 1206 Arnold, Dorian C., 1213 Ast, Markus, 519 Atun, Murat, 234 Avresky, Dimiter, 1148 Babayan, Boris, 18 Bachmann, Dieter, 1213 Baden, Scott, 617 Bader, Michael, 795 Bader, Reinhold, 1351 Baiardi, Fabrizio, 218 Baker, Mark, 1115 Bal, Henri E., 690 BaJla, Piotr, 511 Baldoni, Roberto, 609 Bampis, Evripidis, 288 Bandera, Gerardo, 331 Barrado, Cristina, 519 Barry, Ed, Jr., 739 Barton, John, 1031 Bartzis, Constantinos, 877 Baxter, Jonathan, 980 Beˇcka, Martin, 861 Beckert, Armin, 1315 Becuzzi, Primo, 218 Behrens, J¨ orn, 815 Benkner, Siegfried, 647 Benner, Peter, 824 Bermudo, Nerina, 194 Beyls, Kristof E., 998
Bianco, Mauro, 638 Bischof, Christian H., 86 B¨ ohm, Klemens, 435 Bonhomme, Alice, 1110 Bouabdallah, Abdelmadjid, 600 Boug´e, Luc, 1039 Bourgeois, Julien, 208 Brandes, Thomas, 647 Brehm, Matthias, 1351 Brezany, Peter, 1251 Brodlie, Ken, 1323 Bromling, Steven, 95 B¨ ucker, H. Martin, 86 Budiu, Mihai, 969 Bungartz, Hans-Joachim, 771 Busch, Costas, 575 Butz, Torsten, 829 Buyya, Rajkumar, 1115 Cagnard, Paul-Jean, 767 Cai, Wentong, 189 Cain, Harold W., 108 Calder, Brad, 70 Cameron, Kirk W., 141 Campo, Renato, 849 Cappello, Peter, 1231 Caragiannis, Ioannis, 877 Cela, Jos´e, 519 Chang, Lung-Chung, 994 Chapman, Barbara, 329 Chassin de Kergommeaux, Jacques, 133 Chen, Ying, 1302 Chiao, Hsin-Ta, 1053 Chirivella, Vicente, 909 Chiti, Sarah, 218 Choudhary, Alok, 1263, 1292 Chung, Chung-Ping, 994 Clark, Terry W., 511 Coghlan, Brian A., 1143 Collard, Jean-Fran¸cois, 329 Corporaal, Henk, 349, 965, 1105 Costa, V´ıtor Santos, 744 Cotofana, Sorin, 965 Cunha, Jos´e C., 1313 Cunniffe, Ronan, 1143
1364
Index of Authors
Danelutto, Marco, 1175 Dang Tran, Fr´ed´eric, 1061 Daoudi, El Mostafa, 506 D’Apuzzo, Marco, 839 Darling, Gordon J., 500 Darlington, John, 686 Darte, Alain, 357, 405 David, Pierre, 1201 Davis, M. Kei, 739 Day, Khaled, 935 Decker, Thomas, 277 De Dios Hern´ andez, Agust´ın, 550 Delaplace, Franck, 340 Demetriou, Neophytos, 575 D’Hollander, Erik H., 998 D´ıaz de Cerio, Luis, 591 Diderich, Claude, 405 Dobrev, Stefan, 927 Dongarra, Jack, 1213 Drozdowski, Maciej, 311 Duato, Jose, 875 Ebcioglu, Kemal, 939 Ebner, Ralf, 1351 Efraimidis, Pavlos S., 456 El Abbadi, Amr, 427 Espinosa, Antonio, 173 Faden, Michael, 1315 Fahringer, Thomas, 105 Farrens, Matthew K., 989, 1008 Feautrier, Paul, 1201 Fink, Torsten, 1223 Finta, Lucian, 288 Fischer, Rolf, 519 Fodor, Eugene F., 729 Ford, Rupert W., 395 Fox, Geoffrey, 1211 Franke, Hubertus, 242 Freytag, Johann-Christoph, 445 Gan, Boon-Ping, 189 Gao, Guang R., 625 Gemund, Arjan J.C. van, 272 Gengler, Marc, 405 Gentzsch, Wolfgang, 1313 Gerholt, Thomas, 1315 Gerndt, Michael, 45 Gerner, Philippe, 668 G´erodolle, Anne, 1061
Getov, Vladimir, 617 Gin´e, Francesc, 1165 Godlevsky, Alexander B., 754 Golby, David, 1345 Goldstein, Seth C., 969 Gonz´ alez, Antonio, 194, 591 Gonz´ alez, Daniel, 320 Gonzalez, Jesus A., 682 Gorlatch, Sergei, 617 Gottschling, Peter, 784 Grabs, Torsten, 435 Gray, Paul A., 709 Grelck, Clemens, 620 Grundmann, Tobias, 1081 Gubitoso, Marco Dimas, 160 Guirado, Fernando, 262 Gupta, Sandeep K.S., 600 Gurd, John R., 75 G¨ ursoy, Attila, 234 Hadj, Yahya Ould Mohamed El, 506 Hammond, Kevin, 739 Hanusse, Nicolas, 583 Harper, John S., 149 Hatcher, Philip J., 1039 Haveraaen, Magne, 758 Heber, Gerd, 625 Hedges, Richard, 1253 Heine, Felix, 415 Heinrich, Ralf, 1315 Heller, Helmut, 1351 Hempel, Rolf, 1251 Herlihy, Maurice, 575 Hern´ andez, Porfidio, 1165 Hern´ andez, Vicente, 824 Hirayama, Hiroyuki, 527 Hluch´ y, Ladislav, 754 Holliday, JoAnne, 427 Holmerin, Jonas, 762 Hong, Lee Wei, 700 Hoogerbrugge, Jan, 950 Hovland, Paul D., 86 Hr´ uz, Tom´ aˇs, 861 Hsu, Wen-Jing, 595 Hsu, Windsor W., 1302 Hu, Weiwu, 1132 Huedo, Eduardo, 199 Humes, Carlos, Jr., 160 Hyde, Daniel C., 1115
Index of Authors Igai, Mitsuyoshi, 527 Ijaha, Stephen E., 869 Ioki, Nobuhiro, 527 Ionescu, Felicia, 532 Ionescu, Mihail, 532 Ishihara, Takashi, 729 Ishikawa, Yutaka, 1071 Jalba, Andrei, 532 Jalby, William, 539 Jennings, Dave, 168 Ju, Jialin, 718 Juhasz, Zoltan, 1171 Kaklamanis, Christos, 835, 877 Kalantery, Nasser, 869 Kamachi, Tsunehiko, 1239 Kandemir, Mahmut T., 1263 Karkowski, Ireneusz, 349 Karl, Wolfgang, 851 Karypis, George, 252, 296 Kaur, Samian, 1332 Kelly, Paul H.J., 567, 617 Kerbyson, Darren J., 149 Kersken, Hans-Peter, 1315 Kesmarki, Laszlo, 1171 Kewley, John M., 65 Keyes, David E., 1 Khonsari, Ahmad, 900 Kielmann, Thilo, 690 Kik, Marcin, 471 Kikuchi, Sumio, 1023 Kindermann, Stephan, 1223 Klusik, Ulrike, 739 Knoop, Jens, 329 Koniges, Alice E., 1253, 1273 Konstantopoulos, Charalampos, 835 Kranakis, Evangelos, 583 Krizanc, Danny, 583 K¨ ugeler, Edmund, 1315 Kumar, Rishi, 625 Kumar, Vipin, 252, 296 Kurmann, Christian, 1118 KutyJlowski, MirosJlaw, 455 Kyushima, Ichiro, 1023 Labarta, Jes´ us, 519 ´ Laborda, Oscar, 519 Lafage, Thierry, 178 Laforenza, Domenico, 1211
Lancaster, David, 1340 Larriba-Pey, Josep-L., 940 von Laszewski, Gregor, 22 Leinberger, William, 252 Leon, Coromoto, 682 Li, Maozhen, 1345 Li, Tiejun, 729 Liao, Ching-Jung, 57 Lim, Chu-Cheow, 189 Lin, Hong, 481 Lindenmaier, G¨ otz, 223 Lisper, Bj¨ orn, 762 Liu, Haiming, 1132 Llaber´ıa, Jos´e Mar´ıa, 960 Llanos Ferraris, Diego R., 550 Llorente, Ignacio M., 199 Llosa, Josep, 194 Loidl, Hans-Wolfgang, 739 Low, Yoke-Hean, 189 Luque, Emilio, 173, 262, 1165 Lynch, Robert E., 481 Lysne, Olav, 890 MacBeth, Mark, 1039 MacDonald, Steve, 95 Manneback, Pierre, 506 Manniesing, Rashindra, 349 Manz, Hartmut, 519 Margalef, Tomas, 173 Marinescu, Dan C., 481 Marino, Marina, 839 Martonosi, Margaret, 1018 Massingill, Berna L., 678 Mateev, Nikolay, 379 Mattson, Timothy G., 678 Mavronicolas, Marios, 575 May, David, 545 Mayer auf der Heide, Friedhelm, 455 Mayer, Anthony, 686 Mayo, Rafael, 824 Mayr, Ernst W., 573 McGuigan, Keith, 1039 McKinley, Kathryn S., 223 Mechelli, Marco, 609 Mehra, Pankaj, 1148 Melas, Panagiotis, 183 Melideo, Giovanna, 609 Memik, Gokhan, 1263 Menon, Vijay, 379 Meuer, Hans Werner, 43
1365
1366
Index of Authors
Meyer, Ulrich, 461 Meziane, Abdelouafi, 506 Midkiff, Samuel P., 329 Milenkovic, Aleksandar, 558 Milis, Ioannis, 288 Miller, Barton P., 45, 108 Milutinovic, Veljko, 558 Min, Geyong, 904 Mitschang, Bernhard, 425 Mittermaier, Christian, 1196 Mohr, Bernd, 123 Mohr, Marcus, 806 Monien, Burkhard, 277 Morancho, Enric, 960 More, Sachin, 1292 Moreira, Jose E., 242 Moreno, Luz Marina, 320 Moreno, Salvador, 1206 Mori, Paolo, 218 Motokawa, Keiko, 1023 Mueller, Frank, 1185 Mukherjee, Nandini, 75 Mulholland, Connor, 500 Muller, Henk, 545 M¨ uller, Silvia, 537 Muralidhar, Rajeev, 1332 Muraoka, Yoichi, 22 Nagel, Wolfgang E., 105, 784 Namyst, Raymond, 1039 Naono, Ken, 527 Navarro, Carlos, 940 Neary, Michael O., 1231 Newhouse, Steven, 686 Ngo, Ton, 1031 Nieplocha, Jarek, 718 van Nieuwpoort, Rob V., 690 Nishiyama, Hiroyasu, 1023 Nolte, J¨ org, 1071 Novillo, Diego, 389 Nudd, Graham R., 149 O’Boyle, Michael F.P., 395 ` Oliv´e, Angel, 960 Olsson, Ronald A., 729 Ordu˜ na, Juan M., 1206 Ould-Khaoua, Mohamed, 900, 904, 935 Papaefstathiou, Efstathios, 149 Parashar, Manish, 1332
Pardalos, Panos M., 839 Pasquarelli, Antonello, 861 Pedroso, Hernˆ ani, 1157 Peinl, Peter, 451 Peng, Shietung, 1086 Peyton Jones, Simon L., 739 Phipps, Alan, 1231 Piccoli, Fabiana, 682 Pingali, Keshav, 368, 379 Post, Peter, 1315 Preis, Robert, 277 Prieto, Manuel, 199 Printista, Marcela, 682 Priol, Thierry, 1239, 1313 Procter, Jim, 1323 Prodan, Radu, 65 Prost, Jean-Pierre, 1253 Prylli, Lo¨ıc, 1110 Pucci, Geppino, 638 Qiu, Ling, 595 Quintana-Ort´ı, Enrique S., 824 Rabenseifner, Rolf, 1273 R˘ adulescu, Andrei, 272 Ragde, Prabhakar, 455 Ram´ırez, Alex, 940 Ramirez, Rafael, 700 Rana, Omer F., 168, 1345 Rathmayer, Sabine, 47 Rauch, Felix, 1118 Raynal, Michel, 35, 605 ´ Reb´ on Portillo, Alvaro J., 739 Reeve, Jeff S., 1340 Reinefeld, Alexander, 1211 Renault, Eric, 1201 Ren´e, Christophe, 1239 Resch, Michael, 479 Ricci, Laura, 218 Rich, Kevin D., 989, 1008 Richert, Thomas, 325 Richman, Steven, 1231 Riley, Graham D., 75 Ripoll, Ana, 262 Ritt, Marcus, 1081 Robles, Antonio, 882 Rocha, Ricardo, 744 Roda, Jos´e L., 682 Rodr´ıguez, Casiano, 320 Rodrigues, Lu´ıs, 605
Index of Authors Rodriguez, Casiano, 682 R¨ ohm, Uwe, 435 Roig, Concepci´ o, 262 Romero, Luis F., 491 Romero, Sergio, 491 Rosenstiel, Wolfgang, 1081 R¨ ude, Ulrich, 771 Ruiz, Aurelio, 1206 Sahelices Fern´ andez, Benjam´ın, 550 Sakr, Majd, 969 Salinger, Petr, 931 Sancho, Jos´e Carlos, 882 de Sande, Francisco, 682 Sanders, Beverly A., 678 Sanders, Peter, 461, 918 Santosa, Andrew E., 700 Sarkar, Prasenjit, 1284 Sato, Mitsuhisa, 1071 Schaeffer, Jonathan, 95, 389 Scheffner, Dieter, 445 Schek, Hans-J¨ org, 435 Schimmler, Manfred, 1085 Schloegel, Kirk, 296 Schmidt, Bertil, 1095 Schnor, Bettina, 217 Scholz, Sven-Bodo, 620 Schot, Henjo, 1105 Schreiber, Andreas, 1315 Schreiner, Wolfgang, 1196 Schulz, Martin, 851 Schulz, Uwe, 519 Sedukhin, Stanislav, 1086 Seidel, Edward, 1211 Sen, Shondip, 545 Senar, Miquel A., 262 Seznec, Andr´e, 178 Sherwood, Timothy, 70 Shields, Matthew S., 1345 Shriver, Elizabeth, 1251 Shudo, Kazuyuki, 22 Shurbanov, Vladimir, 1148 Sibeyn, Jop F., 918 Silber, Georges-Andr´e, 357 Silva, Fernando, 744 Silva, Jo˜ ao Gabriel, 1157 Sivasubramaniam, Anand, 242 Sloan, Terence M., 500 Slowik, Adrian, 415 Solsona, Francesc, 1165
Spaccamela, Alberto Marchetti, 609 Spies, Fran¸cois, 208 Spirakis, Paul G., 456 Srimani, Pradip K., 600 Stanca, Marian, 965 Stefanovi´c, Darko, 1018 Stein, Benhur de Oliveira, 133 Stigliani, Massimiliano, 1175 Stillger, Michael, 445 St¨ ohr, Elena A., 395 Straatsma, Tjerk P., 718 Stenstr¨ om, Per, 537 Stricker, Thomas M., 1118 Strietzel, Martin, 1315 von Stryk, Oskar, 829 Sun, Xian-He, 141 Sunderam, Vaidy S., 709 Svolos, Andreas, 835 Szafron, Duane, 95 Talbot, Sarah A.M., 567 Tavangarian, Djamshid, 1115 Temam, Olivier, 223 Thakur, Rajeev, 1251 Theiss, Ingebjørg, 890 Theobald, Kevin B., 625 Thulasiram, Ruppa K., 625 Tirado, Francisco, 199 Ton, Lee-Ren, 994 Toraldo, Gerardo, 839 Treumann, Richard, 1253 Trinder, Philip W., 739 Tvrd´ık, Pavel, 931 Unrau, Ronald C., 389 Valero, Mateo, 537, 940 Valero, Rodrigo, 1206 Valero-Garc´ıa, Miguel, 591 Vassiliadis, Stamatis, 537, 965 Vera, Xavier, 194 Vergados, Ioannis, 877 Violard, Eric, 668 Vivien, Fr´ed´eric, 405 V¨ olk, Martin, 851 Vrt’o, Imrich, 927 Wagner, Claus, 1185 Walker, David W., 1313, 1345
1367
1368
Index of Authors
Walker, Kip, 969 Wang, Hong, 984 Watson, William, 1148 White, Alison, 1253 Wilcox, Daniel V., 149 Winkler, Franz, 1196 Winter, Stephen C., 869 Wise, David S., 774 Wism¨ uller, Roland, 849 Wolf, Felix, 123 Wolf, Klaus, 1315 Wolniewicz, PaweJl, 311 Wolter, Thieß-Magnus, 829 Wood, Jason, 1323 Wright, Helen, 1323
Wu, Chi-Houng, 1053 Wylie, Brian J.N., 108 Yamamoto, Yusaku, 527 Y´eh, Thomas Y., 984 Young, Honesty C., 1302 Yuan, Shyan-Ming, 1053 Zaluska, Ed J., 183 Zapata, Emilio L., 331, 491 Zavanella, Andrea, 658 Zenger, Christoph, 795 Zhang, Fuxin, 1132 Zhang, Yanyong, 242 Ziegler, Sibylle, 851 Zimmermann, Jens, 815
