Proceedings of the International Conference ParCo2001
PARALLEL COMPUTING Advances and Current Issues
*++ Editors
G. R. Joubert A. Murli F. J. Peters M. Vanneschi
Imperial College Press
PARALLEL COMPUTING Advances and Current Issues
This page is intentionally left blank
Proceedings of the International Conference ParCo2001
PARALLEL COMPUTING Advances a n d C u r r e n t Issues Naples, Italy
4-7 September 2001
Editors
G. R. Joubert Clausthal University of Technology, Clausthal, Germany
A. Murli University of Naples "Federico II", Naples, Italy
F. J. Peters Philips Research, Eindhoven, The Netherlands
M. Vanneschi University of Pisa, Italy
Imperial College Press
Published by Imperial College Press 57 Shelton Street Covent Garden London WC2H 9HE Distributed by World Scientific Publishing Co. Pte. Ltd. P O Box 128, Farrer Road, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.
PARALLEL COMPUTING: ADVANCES AND CURRENT ISSUES Proceedings of the International Conference ParCo2001 Copyright © 2002 by Imperial College Press All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.
ISBN
1-86094-315-2
Printed in Singapore by Uto-Print
CONFERENCE COMMITTEE Gerhard R. Joubert (Germany/Netherlands) (Conference Chair) Almerico Murli (Italy) Frans J. Peters (Netherlands) Roberto Vaccaro (Italy) Marco Vanneschi (Italy)
STEERING COMMITTEE Frans Peters (Netherlands) (Chair) Bob Hiromoto (USA) Masaaki Shimasaki (Japan) Dennis Trystram (France) Marian Vajtersic (Slovakia)
ORGANISING COMMITTEE Almerico Murli (Italy) (Chair) Roberto Vaccaro (Italy) (Vice-Chair) Pasqua D'Ambra (Italy) Luisa D'Amore (Italy) Riccardo Simino (Italy) (Exhibition)
FINANCE COMMITTEE Frans J. Peters (Netherlands) (Chair)
SPONSORS Center for Research on Parallel Computing & Supercomputers (CPS) - CNR, Naples and University of Naples "Federico II" v
INTERNATIONAL PROGRAM COMMITTEE Marco Vanneschi (Italy) (Program Committee Chair) Almerico Murli (Italy) (Program Committee Vice-Chair) HojjatAdeli(USA)
Lucio Grandinetti (Italy)
Andreas Reuter (Germany)
Giovanni Aloisio (Italy)
Rolf Hempel (Germany)
Dirk Roose (Belgium)
Hamid Arabnia (USA)
Rudolf Kober (Germany)
Domenico Sacca (Italy)
Farhad Arbab (Netherlands)
Norbert Kroll (Germany)
Giuseppe Serazzi (Italy)
Lidia Arcipiani (Italy)
Giuliano Laccetti (Italy)
Henk Sips (Netherlands)
Fabrizio Baiardi (Italy)
Domenico Laforenza (Italy)
Antonio Sgamellotti (Italy)
Arndt Bode (Germany)
Antonio Lagana (Italy)
Giandomenico Spezzano (Italy)
Alan Chalmers (UK)
Keqin Li (USA)
Vaidy Sunderam (USA)
Jacques Chassin de Kergommeaux (France)
Alberto Machi (Italy)
Domenico Talia (Italy)
Andrea Clematis (Italy)
Enzo Marinari (Italy)
Marco Tomassini (Switzerland)
Pasqua D'Ambra (Italy)
Valero Mateo (Spain)
Theo Ungerer (Germany)
Luisa D'Amore (Italy)
John Mellor-Crummey (USA)
Dennis Trystram (France)
Marco Danelutto (Italy)
Paul Messina (USA)
Marian Vajtersic (Slovakia)
Erik D'Hollander (Belgium)
Hermann Mierendorff (Germany)
Henk van der Vorst (Netherlands)
Koen De Bosschere (Belgium)
Giovanni Milillo (Italy)
Arjan van Gemund (Netherlands)
Giuseppe De Pietro (Italy)
Wolfgang Nagel (Germany)
Nicola Veneziani (Italy)
Jack Dongarra (USA)
Salvatore Orlando (Italy)
Heinrich Voss (Germany)
Ian Duff (UK)
Yoshio Oyanagi (Japan)
Helmut Weberpals (Germany)
Salvatore Gaglio (Italy)
Nikolay Petkov (Netherlands)
David Womble (USA)
Wolfgang Gentzsch (USA)
Wilfried Philips (Belgium)
Andrea Zavanella (Italy)
Giulio Giunta (Italy)
Erhard Rahm (Germany)
Hans Zima (Austria)
VI
VII
ADDITIONAL REFEREES Marco Aldinucci (Italy)
Yves Denneulin (France)
Hans-Peter Kersken (Germany)
Paolo Palmerini (Italy)
Alberto Apostolico (Italy)
Daniela di Serafino (Italy)
Jochen Kreuzinger (Germany)
Enric Pastor (Spain)
Eduard Ayguade (Spain)
J. Diaz
Norbert Kroll (Germany)
Matthias Pfeffer (Germany)
Kristof Beyls (Belgium)
Jose Duato (Spain)
Uwe Lehmann (Germany)
Francesco Scarcello (Italy)
Steeve Champagneux (France)
Bernardo Favini (Italy)
Giuseppe Lo Re (Italy)
R. Sommerhalder (Netherlands)
Guilhem Chevalier (France)
Uwe Fladrich (Germany)
Maria Lucka (Austria)
Gerardo Toraldo (Italy)
Massimo Coppola (Italy)
Ilio Galligani (Italy)
Giuseppe Manco (Italy)
Salvatore Vitabile (Italy)
Benedicte Cuenot (France)
Serge Gratton (France)
Eduard Mehofer (Austria)
Manuela Winkler (Germany)
Marco D'Apuzzo (Italy)
Mario Guarracino (Italy)
Lorenzo Muttoni (Italy)
Yijun Yu (Belgium)
Monica De Martino (Italy)
Juan GuillenDaniel Ortega Scholten (Holland)
Peter Zoeteweij (Netherlands)
This page is intentionally left blank
PREFACE The international conference ParCo2001 was held in September 2001 in Naples, Italy. This biannual conference, which is the longest running and most prestigious international conference on all aspects of parallel computing in Europe, again formed a milestone in assessing the status quo and highlighting future trends. Whereas many aspects of parallel computing have already become part of main stream computing, challenging new application areas are opening up. These aspects were highlighted by the invited speakers and the panel discussion. Together with the contributed papers and the mini-symposia an overall scenario of on the one hand consolidation of parallel computing technologies and on the other emerging new areas of research and development was presented. New areas in which parallel computing is fast becoming a strategic technology are image and video processing, multimedia applications, financial modelling, data warehousing and mining, to name but a few. New definitions of the parallel computing paradigm in the form of cluster and grid computing are gradually reaching the stage where their widespread application to a multitude of problem areas will become a viable option. With the ParCo conferences the emphasis has always been on quality rather than quantity. This approach resulted in the decision to run these conferences on a biannual basis, such that noticeable strides in technology development can be highlighted. Furthermore, all contributions were in the past reviewed during and after the conference. This latter approach had the disadvantage that proceedings were only available many months after the conference. In an attempt to shorten the time to publication in the case of ParCo2001, authors were requested to submit full versions of papers for the first review and selection process. Electronic versions of revised papers were made accessible to all registered delegates before the start of the conference. A final revision during and after the conference resulted in the papers incuded in these proceedings. It should be noted that papers presented as part of the various mini-symposia are not included in the printed proceedings. Their inclusion would have seriously delayed publication. Such papers will be considered for publication in Special Issues of Parallel Computing journal.
IX
X
The editors are indebted to the members of the International Program Committee, the Steering Committee and the Organising Committee for the time they spent in making this conference such a successful event. Special thanks are due to the staff of the Center for Research on Parallel Computing & Supercomputers (CPS) - CNR in Naples for their enthusiastic support. In this regard particular mention should be made of the key roles played by Almerico Murli (Organising Committee Chair), Pasqua D'Ambra and Luisa D'Amore in making this event such a great success. Gerhard Joubert Germany November 2001
Almerico Murli Italy
Frans Peters Netherlands
Marco Vanneschi Italy
CONTENTS
Committees
v
Preface
ix
Invited Papers
1
Deploying Parallel Numerical Library Routines to Cluster Computing in a Self Adapting Fashion J. J. Dongarra andK. J. Roche
3
GRID: Earth and Space Science Applications Perspective L. Fusco
31
e-Science, e-Business and the Grid T.Hey
33
Graph Partitioning for Dynamic, Adaptive and Multi-Phase Scientific Simulations V. Kumar, K. Schloegel and G. Karypis
34
Challenges and Opportunities in Data-Intensive Grid Computing P. Messina
40
Applications
41
Giant Eigenproblems from Lattice Gauge Theory on CRAY T3E Systems N. Attig, Th. Lippert, H. Neff, J. Negele andK. Schilling
43
Parallel Consistency Checking of Automotive Product Data W. Blochinger, C. Sinz and W. Kiichlin
50
Implementation of an Integrated Efficient Parallel Multiblock Flow Solver T. Bonisch and R. Ruhle
58
Distributed Simulation Environment for Coupled Cellular Automata in Java M. Briesen and J. Weimar
66
XI
XII
Human Exposure in the Near-Field of a Radiobase-Station Antenna: A Numerical Solution Using Massively Parallel Systems L. Catarinucci, P. Palazzari and L. Tarricone Tranquillity Mapping Using a Network of Heterogeneous PC A. Clematis, M. De Martino, G. Alessio, S. Bini, and S. Feltri Parallel Skeletons and Computational Grain in Quantum Reactive Scattering Calculations S. Crocchianti, A. Lagana, L. Pacifici and V. Piermarini Parallel Image Reconstruction Using ENO Interpolation J. Czerwinska and W. E. Nagel Training On-Line Radial Basis Function Networks on a MIMD Parallel Computer A. D'Acierno Parallel Simulation of a Cellular Landslide Model Using Camelot G. Dattilo and G. Spezzano Parallel Numerical Simulation of Pyroclastic Flow Dynamics at Vesuvius T. Esposti Ongaro, C. Cavazzoni, G. Erbacci, A. Neri and G. Macedonio
75
83
91
101
109
117
125
A Fast Domain Decomposition Algorithm for the Simulation of Turbomachinery Flows P. Giangiacomo, V. Michelassi and G. Chiatti
13 3
Massively Parallel Image Restoration with Spatially Varying Point-Spread-Functions G. Gorman, A. Shearer, N. Wilson, T. O'Doherty andR. Butler
141
Asynchronous Algorithms for Problem of Reconstruction from Total Image JV. M Gubareni
149
Parallel Flood Modeling L. Hluchy, G. T. Nguyen, L. Halada and V. D. Tran
157
XIII
The Xyce Parallel Electronic Simulator — An Overview S. Hutchinson, E. Keiter, R. Hoekstra, H. Watts, A. Waters, T. Russo, R. Schells, S. Wix and C. Bogdan
165
A Breakthrough in Parallel Solutions of MSC. Software L. Komzsik, S. Mayer, P. Poschmann, P. Vanderwalt, R. Sadeghi, C. Bruzzo and V. Giorgis
173
Implementation of a Parallel Car-Parrinello Code on High Performance Linux-Based Clusters 5. Letardi, M. Celino, F. Cleri, V. Rosato, A. De Vita and M. Stengel
181
PRAN: Special Purpose Parallel Architecture for Protein Analysis A. Marongiu, P. Palazzari and V. Rosato
189
Design of a Parallel and Distributed Web Search Engine S. Orlando, R. Per ego andF. Silvestri
197
Active Contour Based Image Segmentation: A Parallel Computing Approach V. Positano, M. F. Santarelli, A. Benassi, C. Pietra andL. Landini
205
Parallelization of an Unstructured Finite Volume Solver for the Maxwell Equations J. Rantakokko and F. Edelvik
213
An Hybrid OpenMP-MPI Parallelization of the Princeton Ocean Model G. Sannino, V. Artale and P. Lanucara
222
A SIMD Solution to Biosequence Database Scanning B. Schmidt, H. Schroder and T. Srikanthan
230
Parallel Program Package for 3D Unsteady Flows Simulation E. Shilnikov andM. A. Shoomkov
238
Parallel Lossless Compression Algorithm for Medical Images by Using Wavefront and Subframe Approaches A. Wakatani Adaptivity and Parallelism in Semiconductor Device Simulation H. Weberpals and S. Thomas
246
254
XIV
Algorithms
263
Analysis of Communication Overhead in Parallel Clustering of Large Data Sets with P-AutoClass S. Basta and D. Talia
265
A Coarse-Grain Parallel Solver for Periodic Riccati Equations P. Benner, R. Mayo, E. S. Quintana-Orti and V. Hernandez
274
Preconditioning of Sequential and Parallel Jacobi-Davidson Method L. Bergamaschi, G. Pini and F. Sartoretto
282
Parallel Checkpointing Facility on a Metasystem Y. Cardinale andE. Hernandez
290
An Efficient Dynamic Programming Parallel Algorithm for the 0-1 Knapsack Problem M. Elkihel andD. El Baz
298
Parallel Algorithms to Obtain the Slotine-Li Adaptive Control Linear Relationship J. C. Fernandez, L. Pehalver and V. Hernandez
306
A High-Performance Gepp-Based Sparse Solver A. Gupta
314
System Model for Image Retrieval on Symmetric Multiprocessors O.Kao
322
Granularity and Programming Paradigms in Parallel MPP Image Coding R. Norcen and A. Uhl
330
Parallel Quasi-Newton Optimization on Distributed Memory Multiprocessors /. Pardines and F. F. Rivera
338
A Parallel Condensation-Based Method for the Structural Dynamic Reanalysis Problem A. Pelzer andH. Voss
346
Combined MPI/OpenMP Implementations for a Stochastic Programming Solver D. Rotiroti, C. TrikiandL. Grandinetti
354
XV
The Same PDE Code on Many Different Parallel Computers W. Schonauer, T. Adolph and H. Hafner What Do We Gain from Hyper-Systolic Algorithms on Cluster Computers? W. Schroers, Th. Lippert andK. Schilling
362
370
Leader Election in Wrapped Butterfly Networks W. Shi, A. Bouabdallah, D. Talia and P. KSrimani
382
Software Technology and Architectures
391
Assembling Dynamic Components for Metacomputing Using CORBA A. Amar, P. Boulet and J. Dekeyser
393
Simulation-Based Assessment of Parallel Architectures for Image Databases T. Bretschneider, S. Geisler and O. Kao
401
Structured Parallel Programming and Shared Objects: Experiences in Data Mining Classifiers G. Carletti and M. Coppola
409
Access Histories Versus Segment Histories for Datarace Detection M. Christiaens, M. Ronsse andK. De Bosschere
417
On Skeletons & Design Patterns M. Danelutto
425
A Portable Middleware for Building High Performance Metacomputers M. Di Santo, F. Frattolillo, E. Zimeo and W. Russo
433
Using a Parallel Library of Sparse Linear Algebra in a Fluid Dynamics Application Code on Linux Clusters S. Filippone, P. D'Ambra andM. Colajanni Performance Evaluation of a Graphic Accelerators Cluster M. R. Guarracino, G. Laccetti and D. Romano
441
449
xvi Analysis and Improvement of Data Locality for the Transposition of a Sparse Matrix D. B. Heras, J. C. Cabaleiro andF. F. Rivera
457
Automatic Multithreaded Parallel Program Generation for Message Passing Multiprocessors Using Parameterized Task Graphs E. Jeannot
465
Parallel Launcher for Cluster of PC C. Martin and O. Richard
473
Parallel Program Development Using the MetaPL Notation System ./V. Mazzocca, M. Rak and U. Villano
481
PAVIS: A Parallel Virtual Environment for Solving Large Mathematical Problems D. Petcu and D. Gheorchiu
490
Development of Parallel Paradigms Templates for Semi-Automatic Digital Film Restoration Algorithms G. Sardisco and A. Machl
498
Exploiting the Data-Level Parallelism in Modern Microprocessors for Neural Network Simulation A. Strey andM. Bange
510
Extending the Applicability of Software DSM by Adding User Redefinable Memory Semantics B. Vinter, O. J. Anshus, T. Larsen and J. M. Bjorndalen
518
Industrial Perspective
527
Porting and Optimizing Cray T3E Applications to Compaq AlphaServer SC Series J. Pareti
529
Author Index
541
INVITED PAPERS
This page is intentionally left blank
DEPLOYING PARALLEL N U M E R I C A L LIBRARY R O U T I N E S TO CLUSTER C O M P U T I N G LN A SELF A D A P T I N G FASHION K E N N E T H J. R O C H E , J A C K J . D O N G A R R A Department
of Computer Science, The University 203 Claxton Complex, Knoxville, Tennessee 37996-3450 roche,
[email protected]
of
Tennessee,
This paper discusses middleware under development which couples cluster system information with the specifics of a user problem to launch cluster based applications on the best set of resources available. The user is responsible for stating his numerical problem and is assumed to be working in a serial environment. He calls the middleware to execute his application. The middleware assesses the possibility of solving the problem faster on some subset of available resources based on information describing the state of the system. If so, the user's data is redistributed over the subset of processors, the problem is executed in parallel, and the solution is returned to the user. If it is no faster, or slower, the user's problem is solved with the best appropriate serial routine. The feasibility of this approach is empirically investigated on a typical target system and results reported which validate the method.
1
Overview
On multi-user computing systems in which the resources are dedicated but the allocation of resources is controlled by a scheduler, one expects a job, once allocated, to finish execution in a predictable amount of time. In fact, the user on such a system usually submits a job through a batch script which requires an upper bound on the predicted runtime of the task being queued. The user is billed for the entire time requested and thus has a responsibility to himself and other users to understand the behavior of his code. Scheduling schemes on such systems attempt to order the computations in a fair manner so that user jobs show progress towards completion in a timely fashion while the overall system throughput is maximized. One problem with this approach is that such a scheduling scheme is easier to talk about than to implement.(ref. [1]) It is a fact of life that the multi-processor scheduling problem is iVP-complete in the strong sense." Thus, developers must look for algorithms which are "Theoretically, a decision problem II is NP-complete in the strong sense if (II 6 NP)A(3Tlp which is JVP-complete). For decision problem II and polynomial p, defined over the integers, lip is the restriction of II t o instances / 3 Max[i] < p(Length[I]). If II is solvable by a pseudo-polynomial time algorithm, then n p is solvable in polynomial time. Consider the
3
4
efficient. This is a difficult and time consuming task (which can't really be avoided on production scale or commodity clusters such as those at national laboratories and supercomputing centers). The large variance in the average duration and demand of user jobs, for instance, further complicates the task. It's complicated. Even though there's no provably optimal way of addressing this problem, people do it all the time because it has to be done. We don't make further considerations of such systems in this paper. In shared, multi-user computing environments, such as clusters of workstations in a local area network, the notion of determinism in computations can be lost due to resource contention. Thus, successive runs of the same linear algebraic kernel with the same problem parameters, for instance, may result in grossly different wall clock times for completion due to the variability of the work load on the CPUs from competing tasks. If users compute responsibly and in coordination with one another such systems can be and are successful. (Administrators intervene otherwise to mediate serious contention problems.) This approach is often more efficient, for instance, in the developmental stage of parallel application codes due to the constant testing and debugging of software, or in groups where the average user job tends not to saturate the system resources and runs to completion on a relatively short time scale (e.g. minutes or even hours). One way for the user to possibly make better use of the available resources in such an environment is to employ the information describing the state of the computational system at runtime to select an appropriate subset of resources for the specific kernel at hand. It is acknowledged that making low risk predictions in such a system when multiple users are sharing the resources cannot be done with certainty. There is nothing to preclude the event that the demand on the resources may change dramatically in the time which transpires between deciding on a set of resources and getting one's problem set up and ready to go on the resources. Nonetheless, it seems negligent not to try to use available system related data at runtime. In the very least a user can identify saturated resources in the system and avoid allocating them for his/her run. In the event that the overall system behavior fluctuates about some time sensitive normal level of activity, then a statistical analysis of the collected data can be used as the basis for a predictive model of the system's behavior at some future time. Software, such as NWS, the multi-processor scheduling problem TIMS'- given a finite set J of jobs, a length l(j) e Z+Vj e J , a number, m e Z+, of processors, and deadline D e Z+ , is there a partition J = Ji U Ji U • • • U Jm of J into m disjoint sets such that m a i [ J ] . - j . l(j) : 1 < i < m] < Dl This problem is NP-complete in the strong sense and thus cannot be solved by a pseudo-polynomial time algorithm unless P = NP. (For proof see reference [2], for related information see references [3,4,5,6,7,8].)
5
Network Weather Service, operates sensors in a distributed computing environment and periodically (in time) collects measured data from them.(ref. [9]) NWS includes sensors for end-to-end T C P / I P performance (bandwidth and latency), available CPU percentage, and available non-paged memory. The collected data is kept and analyzed as a time series which attempts to forecast the future behavior of the system through low order ARM A, autoregressive moving averages, methods. This paper discusses software being developed which couples system information with information specifically related to the numerical kernel of interest. The model being used is that a user is assumed to contact the middleware through a library function call during a serial run. The middleware assesses the possibility of solving the problem faster on some subset of available resources based on information describing the state of the system. If so, the user's data is redistributed over the subset of processors, the problem is executed in parallel, and the solution is returned to the user. If it is no faster, or slower, the user's problem is solved with the best appropriate serial routine. It is conjectured that if the underlying application software is scalable then there will be a problem size which marks a turning point, Ntp, for which the time saved because of the parallel run (as opposed to the best serial runs for the same problem) will be greater than the time lost moving the user's data around. At this value, such software is deemed useful in the sense that it provides an answer to the user's numerical question faster than had the user done as well as an expert working in the same serial environment. That is, it benefits even the expert user working on a single node of the shared system to use the proposed software for problem sizes in which Nuaer > Ntp. As a case study we consider the problem of solving a system of dense, linear equations on a shared cluster of workstations using the ScaLAPACK software, (ref. [10]) A discussion of some specific implementations tested is made and results of selected experiments are presented. It is observed that even with naive data handling the conjecture is validated in a finite ensemble of test cases. Thus there is motivation for future studies in this area. It is also observed that the expert user in the parallel environment can always complete the dense, algebraic task at hand faster than the proposed software. (There are no clear winners for small problem sizes since each approach solves the problem serially with the best available version of the library routine.) This is no surprise since even in the most ideal cases, the proposed software has to touch the user's data at least enough to impart the relevant data structure expected by the logical process grid. The parallel expert, on the other hand, is assumed to be able to generate relevant data structures in-core, in-parallel at the time of the distributed run. This user also knows how to initialize the
6
numerical library routine, and make compilation time optimizations. He/she is probably not the typical scientist who has likely already labored just to reduce their problem to linear algebra. There are, in fact, many details to account for by any user before the parallel kernel runs correctly. Reporting the results of this expert user provides a basis for comparison to the other projected users and scenarios. 2
2.1
Numerical libraries in shared, homogeneous, distributed environments The computing
environment
In the development of the current investigation heterogeneous grid computing systems have not been the central focus. (See references [11,12,13,14,15,16,17].) However, it is noteworthy that one of the goals in resource selection when considering a pool of heterogeneous (and potentially geographically distributed) candidate resources is to achieve as much homogeneity in the allocated resources possible. Here's at least one complication of scheduling in a shared distributed system which is general to both heterogeneous and homogeneous systems: the scheduler of resources for a task in question has to try and allocate resources which not only look homogeneous at the instant of inquiry, but remain as homogeneous as possible for the duration of time that the task is in (parallel) execution. In short, even if we could solve the general multi-processor scheduling problem at some specific instant in time, we can't count on this partitioning to assist us in forecasting the state of the system resources at some time in the future. This is due to the fluctuating properties of system resources which one can observe in a shared environment. This brief subsection intends to describe the notion of homogeneity in the context of the current study. Some sample results of timing various operations in one of the systems tested demonstrates the notion as it is observed empirically. In complex mechanical systems the notion of homogeneity usually implies that the system behaves in a predictable manner when performing a specific task only in the absence of external influences. If this definition applies to computational systems (see Figure 1 for a sample computing environment), then it cannot be that a shared set of resources alone, such as a cluster of workstations in a local area network, is homogeneous. Usually such a system is only meaningful when responding to a user's requests. Users' requests are developed externally and then serviced by the system at runtime. Since there is no way to know when a user intends to make requests in such an open
7 Users, etc.
100 Mbit
5 Network File System Sun's NFS (RPC/UDP)
100 Gbit Switch, (fully connected)
100 Mbit Switch, (fully connected)
Remote memory server, e.g. IBP(TCP)
QQ Figure 1. The figure is a diagram of part of the local area net work in which many of the current investigations were made. It is noteworthy for the purposes of interpreting some of the results presented in this paper that the memory depot and network file server are separate machines in reality sitting on a shared network. The clusters on which we have developed the current study are removed from the shared network through one of two switches through which the cluster of workstations is said to be fully connected. It is a factor for the types of studies we have made that there is only one, 100Mbit line for all of the network flow to and from the network disk or the memory server.
system, any specific task, such as solving a set of linear equations, is likely to exhibit different total wall times for completion on subsequent runs. So what does one mean by a shared homogeneous, distributed computing environment? Naively, it is assumed that the hardware specifications and available software are duplicated between compute nodes on such a system. This is not enough however (and may not be necessary). The notion of homogeneity has meaning only in terms of some specific system state engaged in some specified activity as observed in an ensemble of test cases. Let us elaborate on this thought a little. Physical system parameters, when observed at equidistant time intervals and kept in collections of ordered data, comprise a time series, (see ref-
8
erences[18,19,20,21]) Because of the inherent fluctuations in the system parameters, CPU loads on a shared cluster for instance, the time series of the state of the shared system is a non-deterministic function. (For instance, the activity level of system resources often reflects the underlying nature of the humans which use the system. At lunch time, dinner time, or bed time one often observes a decrease in overall system activity. In the morning, one often observes some adiabatic growth of activity in the system as users arrive and begin working. This growth continues until some normal level of system activity is achieved. For some duration of time, one may expect the system resource activity levels to fluctuate around this normal. But the point is that the activity norm is time of day dependent more often than not.) Non-deterministic time series can only be described by statistical laws or models. To study such systems formally one assumes that a time series can only be described at a given instant in time, t, by a (discrete) random variable, Xt, and its associated probability distribution, fxtThus, an observed time series of the system parameters can be regarded as one realization of an infinite ensemble of functions which might have been generated by a stochastic process -in this case multi-users sharing a common set of resources as a function of time. Stochastic processes are strictly stationary when the joint probability distribution of any set of observations is invariant in time and are particularly useful for modeling processes whose parameters tend to remain in equilibrium about a stationary mean. Any stationary stochastic process can be described by estimating the statistical mean p (x = n _ 1 £)"=i xt), its variance a2 (s% = n _ 1 J2t=i(x* ~ ^)2)> * n e autocovariance function {cxx(k) = n _ 1 S£Ti (xt — x)ixt-k — x) e.g. the extent to which two random variables are linearly independent), and the sample autocorrelation function which is a kind of correlation coefficient (rXx(k) — Cxa(fc)(cx»(0))-1,fc = 0,...,n - l ) . A discrete random process for which all the observed random variables are independent is the simplest form of a stationary stochastic process. For this process, the autocovariance is zero (for all lags not zero) and thus such a process is referred to as purely random, or white noise. Usually, the observable system parameters such as CPU loads and available memory are not independent. Thus, the notion of homogeneity is manifest only in observing a specific task on the system, such as data I/O or multiplying matrices in-core, repeatedly under normal system activity on each of the computing nodes said to comprise the system. An expectation value (fi) for the specified task will be formed for each unit only in this time tested man-
9 ner. 6 One can subsequently compare the results from each of the units and determine a level of similarity between them. Ideally, the time to complete any serial task would be the same within some standard deviation (approximated by the square root of the variance) regardless of the compute node executing it. Further, the error bars should ideally tend to zero as the number of observations tends to infinity. This is not achievable in practice, clearly. (For non-stationary processes, one filters the available data sets thus transforming the problem into a stationary form.) Figures 2, 3, 4, and 5 illustrate the results of portions of an empirical study on one of the systems used to develop the current investigation. In each of the figures, successive runs of a task are time stamped and recorded to file. The results presented were analyzed statistically and only the mean and root of the variance are reported. The runs were conducted over the course of weeks and, as much as possible, during the same hours of the day (10am until 5pm,EST). No attempt was made to push users off the shared system accept during development of some test cases. In Figure 2, an assessment of the CPU of each node when executing (serial runs) a numerical kernel rich in floating point operations as a function of the numerical problem size is made. In this case the time to solution and performance are reported and there is a clear correlation in the expected behavior of each node. Figure 3 is composed of three plots. The plots look at the read and write I/O times per node as a function of bytes on each node's local disk, on the local network users' disk (in our case operating under Sun's NFS utilizing RPCs, UDP/IP), and on a memory server (running IBPC (ref. [23]), TCP/IP) on the same local network but with a different IP address from any node on the cluster or the NFS server itself. In plot one, the local disk accesses, one can only imagine that sporadic user activity is responsible for the larger variances in some of the reported results. To within the error bars 6
If observational data is to be of use in developing advanced software, a standard metric has to be agreed upon and needs to be reliable across different platforms. PAP I (ref. [22]), which accesses hardware counters as well as the system clock, provides such a tool and has been used throughout this investigation when recording observations. C IBP, the Internet Backplane Protocol, is software for managing and using memory on a distribution of disks across a network. Its design is really intended for large scale, logistical networking. However, because it was designed with a client /server model in mind, is also useful for our purposes (as will be discussed further in this report.) The client accesses remote resources through function calls in C, for instance. The multi-threaded servers have their own IP addresses. The client has to know this address in advance as well as to which port the server listens. Usually one sets up his own IBP depot and can choose the port. The IBP group also manages some public domain (free) depots across the country which one can use.
10 TORC(Homogeneity(CPlJ)) Performance of solve routine _GESV() in Gflops
TORC(Homogeneity(CPU)) Time to Solve Ax=b:: _GESVQ (ATLAS)
50
|
T
i
|
i
|
i
|
i
|
ns*!
0.40
/ CPU homogeneity on TORC::
/
/ 40
Time to solve Ax=b::_GESV()
30
• Morel * — »torc2
/
CPU homogeneity on TORC:: 0.38 -
Performance of _GESV() in GFLOPS
Tim
*
«
?0
<
10
«torc4 torc5 -:~ torc6 torc7
/ /
-•• torc8
~~P~
-h-H"" I
1500
2000
N Figure 2. Performance and time to completion numbers for the serial, dense, linear solve routine gesvQ from ATLAS are reported as an example. CPU homogeneity in the shared cluster is a very important criterion for developing numerical software intended for distributed environments.
the wall times reported are within seconds of one another and thus invoke some sense of homogeneity. In the software designed to date, we don't make explicit use local disk I/O. In plot two, the accesses to the local network disk as controlled by NFS, we again see fluctuations, in particular during the UNIX system reads. This data is of particular interest to us, as will become clear in the sections to follow. It is recalled that to access {NFS controlled) the network disk that data is moved over a single shared local communication line. Further, multiple users tax NFS due to the design of the shared local file system. One expects larger variances here. Nonetheless, the results again instill some confidence within the expected error. Further, we have to deal with reality. The systems being tested ore supposedly identical. However, in open systems we must work with the expectation that this notion is a fallacy -we can only make sensible predictions within some confidence limits which are set by the actual behavior of the system. It is fun, however, to guess at why the writes appear to have much tighter error bars than the reads.
u TORC(Homogeneiry(I/0))
TORC(Hom0geneity(I/O))
Time to WRITE versus number of bytes
Time to READ versus number of bytes
*
•torcl
*
* torc2
*-
* torc3
*
< torc4
i
•
i
i
I" —
_ =,= J - - j f J=
.
—
•^**f— -— '
i l.OH-Og
1
'
1
'
-
•
^t ;
*" torcS
O.OiM-00
1
• • torcl » *torc2 * *torc3 •*—
~
•• torc6 torc7
*>
'
-
torcS -
*
Local disk(s)
i , i 2.0»0g 3.0»08 Bytes
Local disk(s)
=c:
y* T*"™*-"
'_'
'.^£
, 4.0M-08
S.Oc+08
0.0e+00
l.Oe+Og
2.0e+0g
3.0W-08
4.0e+08
TORC(Homogeneirj1I/0))
TORC(Homogeneity(I/0))
Time to WRITE versus number of bytes
Time to R E A D versus n u m b e r of bytes
5 Oo+OS
150
150
• *
• torcl •» torc2
* +—*
s
••>
•" torc6 torc7
"»•
| 100
*
UDP/IP, Network
torc3 torc4 torcS
• * * <
125
Pile System
100
EH
a
75
B
50
• torcl * torc2 * torc3 4torc4 torc5 T; torc6 torc7
UDP/IP Network File System
25 0-
0.0e+00
150
100
1.0w08
2.0M-08
3.0»08 Bytes
5.0H-08
6.0M-08
0.OH-0O
l.Ow-08
2.0e+O8
lCe+08 Bytes
4,0c+08
5.0o+Og
TORC(Homogeneity(I/0))
TORC(Homogeneity(I/0))
Time to W R I T E versus n u m b e r of bytes
Time to R E A D versus n u m b e r of bytes
• *
•torcl < torc2
* •*
* torc3 « torc4 torcS
1
1
50
$jfi~ i ¥ O.Ow-00 1.0e+08
1
I • » *•• *
IBP.Remote Memory Server (TCP/IP)
........ -
, * '
toic7 «-—-i lorc8
J
4.OW-08
~^^r
*££ .
l?8^ i 3.0e+08 Bytes
t
:
-
-
-jJE i 2.0e+Q8
•>
I 75
1 1 I I • torcl IBP .Remote Memory a torc2 Server (TCP/IP) *torc3 «torc4 torc5 -.': torco" torc7 <•• torc8
6.0O+08
i 4.0e+08
i 5.0e+08
l 6.0M-Q8
O.Ow-00
1.0
i Z0e+08
i 3.&H-08
i 4.0e+08
i 5.0e+08
6.0W-08
Figure 3. Empirical study of what is meant by I/O homogeneity on the shared cluster.
12
One can't be certain, of course, but maybe NFS is buffering part of the data to be written. Since NFS utilizes RPCs which use the U D P / I P protocol, the application does not require a response before sending the next message. Thus to the user, it appears as though all is well. This is a reliability issue and not addressed here. For the reads, the user doesn't report a time until he actually has his data. This may add to the fluctuations, but it is more likely due to the fact that writes require that the CPU (or I/O device) send both address and data and require no return of data, whereas usually the CPU must be waiting between sending the address and receiving the data on a read. Often, the CPU will not wait on writes. In plot three, the results are quite remarkable in some sense. For one thing, the accesses to the memory server, IBP on a LINUX workstation in the local area, do have to utilize the shared communication line as with the NFS case. However, this server is on a different machine in the local area from that which serves as the NFS server. Thus, one suspects that there is much less overall activity on this server as opposed to the the local network disk itself. In Figure 3, there are two important points to keep in mind when considering the plots. One, for all three data sets, care must be taken to collect such numbers since buffering by various levels of software skews the numbers if successive runs aren't replicated scenarios. For instance, in the NFS runs if one collects such data with naive nested loop approaches, one can observe (not shown) that on the first run the time to report is always larger than subsequent runs. This is because of the buffering of data by either NFS, or possibly locally. If this is not accounted for, very spurious averages are formed which do not reflect the likely reality of servicing a user's request in numerical libraries such as those we are trying to build. The point is, unless the user makes successive requests with the same data set in mind, moving his/her data will be the unbuffered case -which is considerably larger in time. This is accounted for in the presentation here. Next, the reader is advised not to make serious comparisons of plots two and three. The machines servicing these requests employ different hardware and operating systems as well as network protocols. It is noteworthy that the NFS will likely exhibit larger variances since it is subject to a larger average activity in the local area network. One should keep in mind that the requests are all generated from some arbitrary node in our target computing system -this is why we care about these observations. Figure 4 reports available physical memory in kilobytes per node. The plot is important because it demonstrates the reality that homogeneity can't be taken for granted -despite common resources at the onset. In particular, on the cluster under observation, there are certain nodes which are more
13 TORC(HOMOGENEITY(AvailaWe Memory))
i •
•
I
• available unpaged memory / node
-le+05
i
i
i
4 5 Machine Number
Figure 4. The available unpaged physical memory is reported in KBytes per node. Available memory is a critical criterion in resource selection since a user's job must fit in physical memory lest it suffer serious time penalties due to page swapping during execution. The variances for these observations are large since during heavy activity levels on a node, physical memory tends to be nearly fully in use -at other times, nearly completely idle save the OS.
commonly used than others. Who knows why this is, it just is. Until now, we have ignored discussing some important observables such as time for broadcasts, sends, and receives as a function of bytes when a collection of machines from the available resources is executing in parallel. In some sense, it is up to the library developer to know his target system. In other words, the parallel application may impose the system properties of interest. In our test case, we study the solution of systems of linear equations. We will discuss this more later, but the parallel kernel is known to be rich in matrix multiplies. We've seen CPU homogeneity already per node and so expect the local performance to be a good fit for such a kernel. In addition, during the factorization of A, the kernel is also rich in broadcasts. For this reason, the behavior of broadcasts from a root node (the BLACS (ref. [24]) communication library implementation) as observed in our shared system is
14
O.OetflO
TORC(Homogeneity(Communication))
TORC(Homogeneity(Communication))
Time to broadcast versus number of bytes
Time to send/receive versus number of bytes
5.0e4<>7
L.0e403
t.5e+08
O.Oe+OO
5.0etO7
1.0e408
l.Se+08
Figure 5. The plot on the left reports the time for root to broadcast N bytes of data during a parallel run in which 8 processors are selected from the cluster under investigation. Each computing node acts as root. Although the behavior is clearly homogeneous looking, there is a spurious node in the cluster as regards parallel communication of data. The fact is exposed through an investigation of the send/receive times (point to point communication) as a function of bytes. Plot two reports results from two different nodes acting as the lead node in different runs. The lead node contacts each other node in turn and sends N bytes of data to be received.
also reported. Each node is considered as root and results as a function of bytes broadcast timed. We wait for confirmation that each node has received the data before stopping the clock. The slowest node should always dominate -in this sense the results are meaningless for understanding the behavior of a single node's parallel communication with another particular node. The small errors are expected since the machines are fully connected through a common switch. Just in case there is bad communication between two nodes, e.g. to address an issue which the broadcast cannot, one can investigate the behavior of the point to point communications. Results are presented to this end. Two identical runs are considered (there are simply too many permutations of possible collections of allocated resources to report all such numbers here) in which the lead node is different. In the runs, root conducts BLACS sends
15
and receives as a function of message size in bytes (always double precision data in our sample implementation) individually to each other process in the allocated resources. The results in this subsection are intended help to classify our use of the term homogeneous as it applies to individual compute nodes from a pool of candidate resources. The important point is to understand why this is a desirable property for the development of user friendly numerical software. Consider block factorizations from linear algebra which depend upon lower level block operations that have been tuned to be optimal (ref. [25]) on the system for which they intend to be used. Multiplying matrices is one such operation and it is common knowledge that tuned versions of this kernel exist on most any platform. When factorizations, such as A —• P~XLU, are implemented in parallel it remains important for matrix multiplication to be homogeneous between nodes in the parallel run to retain load balance and performance (and in fact scalability in the parallel case). Otherwise, there may be a single node which imparts large delays on the remaining nodes or sits idle waiting for the remaining nodes during execution of the application. Although the numerical answer should be the same regardless, time is the factor which the library routine attempts to minimize. (Naturally, the same could be said of the time to communicate a set number of bytes between processors during a solve. We hope that each node in a set of allocated resources is capable of sending and receiving data at the same rate. If this is not the case, the resources can be stuck waiting. Etc.) Thus, in some sense, to within the errors of each task sampled, there is some expectation as to how the system resources will behave. There is a theoretical isotropy in the compute nodes when homogeneity can be expected. That is, we expect that any particular computing node may be of equal use to us in the selection process subject to interpreting its current state.
2.2
The user
Our target user has a numerical application to execute and intends to do so on a single node of a shared, distributed computing environment which exhibits a strong degree of homogeneity (as described in the previous section). It is assumed that the user invokes one of our library routines to this end. We appeal to the user who may benefit from some user friendly software which intervenes on his behalf to determine if the problem can be solved more quickly through a redistribution of the user's data onto a team of computing nodes, solved in parallel, and the solution mapped back. A standard structure of our
16
target user's code is: User_Code(){ Define J)ata_H an dle(daia Jiandle(out)); Generate _Data(daia(out),dataJiand/e(in)); Invoke -Numerical -Application Jtoutine(dataJiancMe(in/out), routine -name (in) ,routine -parameter 8 (in)); Use_Solution(); Clean-Up(); Exit(); } The user's data will be double precision matrix and vector elements, for instance, in most linear algebra kernels which concern us now. The storage may assume some sparse or dense (row or column major) data structure on input. At this point, we are inclined to assume that the user has correctly formatted his data for a serial execution of a specific routine from some specific numerical library. He has correctly identified the input routine-parameters for the routine-name. The data-handle is what the middleware uses as a key to handling the user's data. For now, suffice it to note that the user can invoke the application routine with his data in-core, in a file on the local network disk (NFS case), or in a file on a memory depot (IBP). The data-handles are a pointer in physical memory, a file name (e.g. a path to the data in the file system) on the local network disk, or an IBP capability respectively. (A capability is the interface to a specific memory location in an IBP memory depot. It is the key to allocateing, storeing, loading, and manageing a logically contiguous block of bytes on the memory server.) One motivation for adding a remote memory depot is to allow for a user to generate problems of a magnitude which would otherwise, due to system constraints, not be feasible on a single node. This can be accomplished through buffered stores to a depot which has abundant available memory. We provide an interface for the user to achieve this set-up as well. Further, a user can choose to generate a data set in the memory depot and share the capability with others if so desired. This allows multiple users to collaborate on the generation of a data set and to share in the solution to a potentially common problem with no difficulties. 2.3
The middleware model
The middleware provides several services which are discussed in turn briefly. The entry point into the middleware is from the user's serial call Invoke-Numerical-Application-Routine() . The middleware, once invoked,
17
first assembles a collection of system specific parameters about the available resources which reflect the current state of the shared system. As previously mentioned NWS, or the like, can be used to achieve this step. Information about the number of candidate computing nodes, the free available memory per node, the CPU load (a number between 0 and 1 for each processor of each node) per node, and the bandwidth and latency for communication between each node in the cluster is gathered. Let us collect this information into the systemjxirameters. A resource selection process is started which relies upon the evaluation of a time function which depends explicitly upon the computational demands oi routine.name as a function of the routine-parameters, and the system .parameters returned from the previous step. Let us call the team of resources which the middleware intends to allocate the selected-resources. Next, if the dataJiandle specifies that the user's data has been generated in-core, then the middleware writes the user's data to the local network disk. (As will be discussed in the comments on data movement later in this paper, the developer has to decide whether to map the user's data at the time of this write or to simply write the data as is and allow the mapping to be made in the application routine prior to executing the parallel application. Both cases are considered in our experiments and results are presented later.) Otherwise, and after the write for the in-core case, the middleware assembles the command line and the machine file necessary for launching the parallel application. The application routine is forked and waited upon by the middleware. On return, if the user expects an answer in-core, the middleware assembles the solution (from disk) and passes it to the user. Otherwise, the middleware returns the modified dataJiandle which now contains updated information regarding where to find the expected solution. The process can be described as pseudo-code as follows:
18
Invoke-Numerical _Application_Routine(dataJiand/e(in/out), routine -name(in),rout*ne-parametera(in)){ Get_State_of_System(«2/a£em-parametera(out)); Do_Resource_Selection(ae/ected sesources(ovX) ,systemj)arameter8(m), routine jname(in),routine4)arameters(m)); If(proceaa==l){ Special-Case -Serial _Run(da£aJMind/e(in/out), selectedjresource8(in),routineJWTne(m),routine-paranieter8(in)y, Return_ControLtO-User(); } Create-Machinefile-and_Dress_CommandJLine(comroandJine(out), selected jre8ource8(io.) ,routinejname(m) ,routine.parameters(m)); If(data Jiandle==" incore") Write_Data-tO-Disk(daiaJiand/e(in)); Fork_Application_Routine_and-Wait(cornmondJine(in)); If( data Jiandle==" incore") Read-Data_from_Disk(data-hond/e(in/out)); Return-Control-to_User(); } It is useful to look at DoJtesourceSelectioni). this stage of the middleware reads:
A general structure for
DoJR^ource_Selection(ae/erfed_re«
19
imparts its own congestion control (e.g. sliding windows) when the network becomes congested complicates matters.(See references [26,27].) Further, we don't know whether the system handling the disk accesses employs DMA or, if not, how many accesses to disk may be necessary to service a request, the time per disk access, etc. If we did know this, we could write a function to approximate the procedure. For instance, suppose we wanted to load N bytes from a network disk into local physical memory. Suppose we know the bandwidth and latency between the source and destination machines, the number of bytes loadable per disk access as well as the time overhead per disk access. In this case, the total number of disk accesses is (N bytes) / (X bytes loaded per disk access), or N/X disk accesses. The time just to get the data to the network will then be {N/X disk accesses) x (Y seconds per disk access), or NY/X seconds, for instance. Next, to move the data over the network, we could assume the ideal and thus, the time to move the data over the network would be simply the latency (converted to seconds) plus N bytes divided by the network bandwidth (converted into bytes per second). Unfortunately, these simplifications ignore the true complexities of the underlying resources and protocols which are used in practice. Such models fail to yield realistic predictions in a shared system. Again, it's complicated and a work in progress. Naively, and for the sake of commentary, one assumes that the typical form for a time function will be of t h e SOrt, 1 solve-user' s-problem — 1 handle -user'8-data.
+Tex ecute -parallel
-application •
Here, Texecute-paraiiei-appiicati<m and Thandie_Uaer's-data are functions of the routine-parameters and system-parameters. In these terms, our original conjecture for a study may be restated as: if Tseriai_expert — Texecute-parallel-application
>
Thandle-user1
s-data,
then
the USer benefits
fwm
invoking the middleware. Otherwise, the user's problem is simply solved serially without any benefits from having invoked the middleware. Again, if the parallel application routine is scalable, we expect to find the problem size which marks the turning point which validates the method. We return to the issue when considering the test case. 2-4
The application layer
The application level of our current effort is built upon pre-existing numerical packages such as PETSc (ref. [28]) or ScaLAPACK. We have relied upon the scalability of such libraries to more than account for the time required for the middleware to handle the user's data. It is the developer's burden to understand the application routines from such libraries at an intricate level so that accurate time functions can be written to be
20
used by the middleware during the resource selection process. At this point, the typical time function for Texecute_paraiiei-appiication has the form -* execute-parallel-application
— •*• communicate-data
"r -* floating -point-operations
• *"
the example discussed in the next section, details of such a function are placed in context of an actual parallel kernel. The application routine has to get the user's data in-core before the parallel execution. There have been several approaches tested to this end. Basically, the data may be pre-mapped by the middleware in which case the parallel application starts with a parallel read of the data from the local network filesystem. The user's data may reside on the local network disk or a memory server in an unmapped format. In this case, either each node assembles its own data through a series of random accesses to the file housing the data, or a single, lead, node is responsible for bringing the data incore and distributing it in a mapped manner -or in bulk letting each node claim its own data- through a series of send/receives or broadcasts respectively. Let us agree for simplicity to lump all these scenarios into the routine GetJJser's-DataJownJncoreQ. Each processor in the allocated team of resources will use this. Then, Parallel_Application_Routine(command Jine(in)) { Parse_Command_Line(env*ronment Jnformation(out) ,dataJiandle(out), routine -parameters(o\ii),com,mandJine(m)); Initialize_ParalleI_Environment(ent;iroT»men<Jn/ormotion(in)); Get_User's_DataJownJncore(do^a(out),dotoJiondle(in)); Execute-Parallel .Application Jloutine((2ata(in/out),r0irf»ne .parameters(in)); Collect _Answer_to _Root(da
3 3.1
Sample software implementation and results Some comments on the kernel, pdgesvi)
In this section, results from a study of the kernel pdgesvi) (from the numerical library ScaLAPACK) are presented and briefly discussed. In each case, it is assumed that the matrix elements form a well determined system, (ref. [29]) Routine pdgesvi) computes solutions to the system of equations Ax = b where A € ftm'n, x € SR", and b e SRm. In particular, a dense, block based factorization of the matrix A is made reducing it to the form P~lLU where P - 1 is a pivot array, L is a lower triangular matrix (unit diagonal), and U is upper triangular. (Thus we expect the blocks -Lii2, and E/2,i to be zero.)
21
After said reduction, two relatively trivial systems of equations are left to solve instead of the original set. Thus, one has PA = LU or A = P_1LU, solves Ly = 6 for y, and finally solves Ux = y for a;. There are a couple of important points to make here. The algorithm for the factorization is applied recursively. Thus, the routine factors (Gaussian elimination) the Ait\, ^2,1 blocks of A recursively employing partial pivoting over rows in a single column of the process grid (the other nodes remain idle). At the stop case, each process in the current, active, process column broadcasts the pivot information to all the remaining columns of processes. Each process can then apply the row interchanges (to all the columns that were not involved in the iteration) to reflect the changes from the previous step. After each such factorization, the Liti, £2,15 and f/i,i blocks are known. The evaluation of each block row of the matrix U requires the solution of a lower triangular system of equations over the elements in a single row of the process grid. Thus, Liti is broadcast along the current row of processes converting panel A\i2 to f/li2 -e.g. J7ii2 *- L^\Ait2- The last step in any iteration of the factorization is the Schur update of the trailing sub-matrix. The column block Z/2,1 is broadcast over rows across all columns of the process grid. £A,2 is broadcast over columns along all the rows of the process grid. Then the Schur update modifies each process' local portion of the block ^2,2- (^2,2 4— ^2,2 — L2,1^1,2) This operation is a parallel matrix multiplication, pdgemmQ (ref. [30]), and it dominates the execution time. After the update, the process is begun again recursively on block A? p. It is important to note that the factorization proceeds from left-to-right, top-to-bottom. Thus, the amount of work per node becomes uneven as the factorization progresses. It is noteworthy that the expert user recognizes this point as a need for optimizing both the block size and the grid aspect ratio (the logical dimensions of the rectangular process grid). The block size is generally chosen to coincide with the problem size which achieves 90% of the performance expected by dgemmQ (ATLAS) on each node. Again, we have a need for homogeneity between nodes. The grid aspect ratio is no big deal really but it does change the time of execution for solving the problem. An example is provided in Figure 6. It is observed quite generally that for a p x q logical rectangular process grid (p,q denotes the number of logical process rows,columns respectively) p/q < 1 yields better performance for the specific kernel at hand than the cases where p/q > 1. When allocating resources from a pool of computing resources for the kernel pdgesvQ, the times of particular interest are tfactorization, hroadcasu and tupdate- On a user's request to solve a system of linear equations, the middleware ascertains the state of the shared system and invokes the selection process which attempts to minimize a time function. This function purports
22 TORC:: Grid Aspect Ratio, pdgemfl
2x4 Proces Grid: Parallel AXEB ScaLAPACK vasus KPL
ThMmftradkacf grid ihape and proManriK
•—•(Pfl)=(S.U *-*(Pfl)=W) lp^=[W) *—»(PA)=(W) •*• --:(pfl) = (W) v - :--GMj) = (W) • >(pfl) = (U) — - (Pfl) = (6.L)
j
t
//SM -
•
S00O
TOCO
N(pnUantlM)
Figure 6. The developer is responsible for understanding how to get the most out of the kernel being implemented in the proposed library. The plot on the left demonstrates a time range of over lOQseconds for the 8 processor run due to either a very poor choice of logical grid layouts, or the best possible. The plot on the right compares (a ratio of execution times is formed) the best run of the ScaLAPACK routine pdgesvQ versus the High Performance Linpack benchmark over the same number of nodes on the system. HPL is highly tuned and is really hard to beat in practice, even for experts. The results demonstrate the reliability of ScaLAPACK to perform well provided the correct system parameters are identified. Tuning the major kernels in ScaLAPACK has been a major concern on the side of the application routine developers.
to model the kernel as a function of the problem parameters as well as the state of the system. The basic communication model assumes a linear relation between any two nodes i, j such that the time to send a message of size X bytes is tcommii^ibandwidthij,latencyij) ~ latencyij + bandwidthj~j • X. Fora,pxq process grid, the factorization of a m x n panel, again, (see the documentation regarding HPL (ref. [31])) occurs within a single process column (p processors). Since the algorithm recurses within a panel, the assumption is made that any of the p processors within the panel perform at level 3 BLAS (ref. [32]) rates. Here is where we employ the notion of homogeneity in our target system as regards both communication and computational expectations. In the kernel of interest we are always bound by the slowest machine in the broadcast and update phases. Thus we select p processors which reflect homogeneity in the sense defined in this paper. If one only considers the time the nearest neighboring process columns will spend in the broadcast phase (after the fac-
23
torization), then the time to factor and broadcast a block panel is estimated for a collection of homogeneous resources by tfactor(m, n) + hroadcaat(m, n) ~ fdgemmi— - *jf) + latency{l + nlgip)) + (bandwidth)'1 (2n21 g(p) + nst). The Schur update of the remaining n x n matrix is approximated for homogeneous systems by tup(iate — Sq'1 (bandwidth)'1 n • nb + latency(p + hip) — 1) + fdgemm • n • nb(*f + 2-£). ^ere "^ *s *ke block size, m,n are the panel dimensions (not global) for the factorization and fdgemm approximates the performance of matrix multiply on an arbitrary node of our homogeneous system. Naturally, the system constraints have to be considered despite the assumption of homogeneity. The total time to perform this factorization in a homogeneous system may be approximated as ~ (start,i = 0) £"+=n6(*/actor(H ~ *>«&) + hTOadcast(n ~ *,nb) + tupdate(n
- t -
nb,nb)).
Quite frankly, one really requires a separate paper simply to work through the details of how we arrive at this time function and how it is used in the selection process. Clearly, we wish to minimize this function. Scheduling schemes are truly difficult however. People try simulated annealing, genetic algorithms, apply low order time series analyses, etc. The problem is of interest to the community in general. We will address the issue at this time. Future papers will describe our efforts in this area in detail. For now, suffice it to say, we crudley minimize this function based on the available system resources through an adhoc means and observe how well the allocated resources complete the task relative to the time we predict it should take based on analyzing the time function. The plots in Figure 7 reveal the faithfulness of our model in its current form. 3.2
Data movement scenarios
Moving and mapping m • n double precision matrix elements is informally discussed. The major consideration is on getting the data, matrices A e 3J m,n t-> (m*n*sizeof(double))bytes and vectors b € 5ft"1 •-> (m*sizeof(double))bytes , from the user and in place for the parallel solve routine. d It is assumed that the data is generated in a natural ordering, which is to say, in the language C for instance, that *(A + i + j*n) references the value of the matrix element in row i and column j of matrix A when 0 < i < m and 0 < j < n. This mapping is of general importance because the scalability of numerical calculations in a distributed computing environment is dependent upon spending more time computing than moving a user's data around. In particular, if numerical libraries such as that being investigated here are to be successful, it is necessary d
In the test cases m = n a s w e consider square matrices in the parallel solve routine.
24 Busy TORC Rons •: Total Wall Clock Time :PDGESVKandOnly
Busy TORC Runs:: PDGESV Kernel Only
Perfonnance Model:: Predicted vs. Measured Results
Ratio of Measured Tune vs. Tune Predicted by the Performance Model
; _s*> i™a jrt*^
; A—i. Ideal
1 '•* P
1.2
||.IS
^ /
f>
i ; - ^
(
/
/
State
• K
-
"
•—•
:
*
*
f--
P a9 |
;
X
0.«5
g oi |
0.75 0.7
Znofa
•
•
1000
2000
3000
5000
6000
s(DD
9O0O
10000
Figure 7. In plot one, the predicted wall times for execution given several different instances of the state of the system is plotted against the resulting execution times actually measured. The second plot is a dimensionless ratio of the two data sets of plot one. The ideal case is the value 1.
that the user's data can be forward mapped into a form expected at the application level (the parallel routines) and the solution backward mapped into natural form for the user in a timely fashion. For dense linear algebra kernels being studied in parallel computing environments it is known that a 2d blockcyclic mapping of the naturally structured data (e.g. matrix A and vector b in Ax = b) provides excellent load balance during parallel runs. (See references [33,34,35] for instance.) The mapping is a function of the problem size (mn, or n for square matrices where [A e 5Rn'n >->• (n 2 * sizeof(dauble))bytes]-^ problem size = n), the block sizes (nbjrow, nb-column, or nb), the number of process columns in the logical rectangular process grid (npcols), the number of process rows in the logical rectangular process grid (nprows), and the matrix elements, A. In fact, the mapping is quite easy to implement naively but it is found that real improvements on such a mapping are more difficult to achieve than expected. In the remainder of this section we outline the general scenarios that we have considered. In each instance, the corresponding read of the data, from the perspective of the parallel application routine, is also discussed. In scenario one, the user first generates the data in-core. There are n2 +n
IIOOO
25
elements and each requires sizeof (double) bytes of memory. This user then passes a pointer to the middleware which in turn (after deciding on the process grid) writes (nprows x npcols) 2d block cyclically pre-mapped work files to the network disk (NFS) as well as nprows work files for the vector elements. In total, of course, only (n 2 + n) * sizeof (double) bytes are written to disk. However, it is useful to compare the time it takes to simply write natural A to a single file on the network disk versus the time to write the elements of A to multiple files in an order imposed by the 2d block cyclic mapping, (see Figure ) In this scenario, the processors involved in the parallel application routine have only to read their matrix elements from disk before executing the parallel application. It is noted that no extra elements are read per processor as the mapping has already taken place in the middleware and is exact. The second general scenario begins with the user's data having been written to a file on either the local network disk or on the memory depot. After the middleware determines a set of processes for the application routine, the dataJiandle is passed to the application routine from the middleware. The natural data is brought in-core by a chosen processor (root) from the logical process grid. The data may be distributed by root in a manner which imposes the mapping during point to point communication with the other nodes(2a). Alternatively, root may proceed to broadcast the data as is and let the mapping be done by each process locally(2b). (We label these scenarios here so the reader can make sense of the plots.) In the final scenario we consider the case where the natural data is brought in-core in parallel by each processor in the logical process grid. The mapping may be imposed exactly during the load phase through random access to the file (3a). Alternatively, bulk data can be brought over the network by each process and the mapping carried out in-core (3b). It is noted that one may contend with network congestion in this scenario. In reality, the developer has to experiment with these approaches before truly understanding the best approach for a given system. 4
Results and conclusions
Figure 8 is interesting to think about. For one thing, we clearly see Ntp exists for each of the scenarios reported on the graph. Notice that the scale of the graph is linear-log. Thus, small separations on the graph suggest large separations in time. Not all the scenarios are reported but the point is clearly made -the user will benefit from interfacing with the proposed software. Plot two of Figure 8 demonstrates at once the strength ScaLAPACK for the dense algebraic kernel, and the reason we pursued this idea in the first place
26 Total Wall Times Solving Ax=b:: TORC (data_generation (and movement)) + (solve)
Total Wall Time for kernel .pdgesvQ:: TORC
10000
10000
1000
1000
NPROOCS-.
^
100
Tolal_wail_time[expen/distr] Tolal_walljime[NFS_scenario(l)] » — « Toul_wall_timetIBP_scenario(2a)] «! ToIal_Wall_Time[IBP_scenalio(3b)|l' ^ Tota]_wall_time(expen/seria!j
4000 6000 N (problem_size)
.pgesv[expert/dislr] * _pgesv[NFS_scenario(l)] » * •s
• _pgesv[lBP_scenario(2a)] ^ _gesv[expert/serial] -s _pgesv[IBP_scenario(3b)]
I i I 4000 6000 N (problem_size)
I 8000
Figure 8. The total time is reported for multiple approaches to solving the linear problem Ax — b. The serial and parallel expert cases proceed without the intervention of additional software. All other runs reported invoke the middleware just as desribed in the current work.
-scalability. For each run on the graph the number of processors allocated for the parallel runs is provided. The expert user in the serial environment can no longer conduct his normal activities in a timely manner. His task is over lOOOseconds behind the proposed software when we stop counting. Figure 9 is also interesting because it displays the overheads involved in simply porting the user's data into a form amenable to the parallel application routine. The times reported for the top two plots are simply the time it takes for the expert user (serial, or parallel) to generate his data before calling the library routine. Next, if we look at the local network disk data (NFS), handling there are multiple plots. The original time it took for the middleware to generate the block cyclic data files for each process in the logical process grid was due to a careless mapping. This is an easy mistake for user's -even developers- to make. Simply compare the time it takes to write some set number of bytes to the network disk with these maps. The only difference in the poorly implemented case (which also affects the numbers in Figure 9) and the faster mappings is in buffering. Similarly, in the load of the workfiles in scenario (2a) attempts to first map the user's natural data and then communicate it. It is simply faster to let multiple requests engage the memory server. Of course, this is only true until such traffic generates
27
2000
T"
4000
6000
8000
10000
~T
T~
0
2000
4000
6000
8000
10000
1000
IBP.DATA
NFS DATA
100
geiKraie_A,bJiicore[nst/seiial] « wntc_wo4_Files[mw/serial] •* load_A',b'_wrile_x[appl/disci] (1) - seiial_wriie_no_map D writtj»oikJilesj3i]f[n]w/serial]
I generate, A,b_remotely[usr+mw/serial] I load_A,b_write_x[appl/distr] (2a) s load_A,b_writc_x[appl/distr] (3b)
_l_ 2000
4000
6000
8000
10000
N
0
2000
4000
6000
8000
10000
N
Figure 9.
congestion on the network. We don't see that here but easily could as larger problems and methods are investigated. There is much left that could be discussed about such next generation software which we have not had time to even mention. For instance, the issue of how to build fault tolerance into such a set of library routines, or assembling contracts which attempt to monitor the progress of a user's task and act if necessary. The general problem of redistributing the user's data is also important.
28
In closing, we have attempted to define the target system through identifying a set of criterion relevant to the numerical library routines we are investigating. Homogeneity is an important criterion and as has been discussed, cannot be taken for granted. We have denned our target user and provided an interface for this user into the middleware which has been designed to utilize existing, scalable library routines. We have reserved a detailed conversation for scheduling in shared, homogeneous system for another time. This subject is of great importance however and cannot be avoided in practice. It is possible, if not likely, that techniques from time sensitive statistical investigations will play a major role in coming to terms with large scale systems -especially as they only become larger. Motivation to further pursue such approaches to numerical libraries on shared, homogeneous systems was provided. We are proposing a more thorough study at this point which attempts to ferret out many of the outstanding mysteries. References 1. See documentation on IBM's LoadLeveler batch system. For instance http :: //uagibm.nerscgov/docs/LoadL/index.html provides documentation of a specific implementation on NERSC's IBM SP/RS6000, currently known as seaborg.nersc.gov. 2. Garey, M. and Johnson, D.,Computers and Intractability, a Guide to the Theory of NP-Completeness,Bel\ Telephone Laboratories, 1979 3. Hopcroft, J. E. and Ullman, J. D.,Introduction to Automata Theory, Languages, and Computation, Addison-Wesley, 1979 4. Ullman, J. T).,"NP-complete scheduling problems,",Journal of Computer and Systems Sciences 10:3, pp. 384-393 5. Knuth, D. E.,Fundamental Algorithms, volume 1 of The Art of Computer Programming, Addison-Wesley, 1968 6. Knuth, D. E.,Seminumerical Algorithms, volume 2 of The Art of Computer Programming, Addison-Wesley, 1969 7. Knuth, D. E.,Sorting and Searching, volume 3 of The Art of Computer Programming, Addison-Wesley, 1973 8. Cormen, T. H., Leiserson, C. E, Rivest, R. L., Stein, C,Introduction to Algorithms, 2nd edition, MIT Press, 2001 9. NWS, The Network Weather Service, http://nws.cs.ucsb.edu/ 10. Blackford, L. S., Choi, J., Cleary, A., D'Azevedo, E., Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R. C.,ScaLAPACK Users' Guide, SIAM, 1997 11. Berman, F., Chien, A., Cooper, K., Dongarra, J., Foster, I., Gannon, D.,
29
12.
13.
14. 15.
16.
17.
18. 19. 20. 21. 22. 23. 24.
25.
Johnsson, L., Kennedy, K., Kesselman, C , Mellor-Crummey, J., Reed, D., Torczon, L., Wolski, R.,"The GrADS Project: Software Support for High-Level Grid Application Development," 2001, Rice University, Houston, Texas; see also http://www.hipersoft.rice.edu/grads Foster, I. and Kesselman, C,"GLOBUS: A metacomputing infrastructure toolkit ^International Journal of High Performance Computing Applications, vol. 1 1 , pp. 115-128 Petitet, A., Blackford, S., Dongarra, J., Ellis, B., Fagg, G., Roche, K., Vadhiyar, S.,"Numerical Libraries and the Grid, "International Journal of High Performance Computing Applications, vol. 15, pp. 359-374 Foster, I. and Kesselman, C,The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, San Francisco, 1999 Petitet, A., Blackford, S., Dongarra, J., Ellis, B., Fagg, G., Roche, K., Vadhiyar, S.,Numerical Libraries and the Grid: The GrADS Experiments with ScaLAPACK, Computer Science Department Technical Report UTCS-01-460, University of Tennessee, Knoxville, Tennessee 37996-3450 Boulet, P., Dongarra, J., Rastello, F., Robert, Y., Vivien, F.,"Algorithmic issues on heterogenous computing platforms," Parallel Processing Letters, 9:2, pp. 197-213, 1999 Kalinov, A. and Lastovetsky, A.,"Heterogeneous Distribution of Computations While Solving Linear Algebra Problems on Networks of Heterogenous Computers," Journal of Parallel and Distributed Computing, 6 1 , 4, pp. 520-535, 2001 Oksendal, B.,Stochastic Differential Equations, 3rd edition, SpringerVerlag (Berlin), 1992 Bremaud, P.,Point Processes and Queues, Martingale Dynamics, Springer-Verlag (New York), p.181 , 1981 Lipster, R. and Shiryayev, A.,Statistics of Random Processes: General Theory, Springer-Verlag (New York), 1977 Lipster, R. and Shiryayev, A..Statistics of Random Processes: Applications, Springer-Verlag (New York), 1978 PAP I, Performance API, http://icl.cs.utk.edu/projects/papi IBP, Internet Backplane Protocol, http://loci.cs.utk.edu/ibp Whaley, R. C,Basic linear algebra communication subprograms: Analysis and implementation across multiple parallel architectures, Computer Science Department Technical Report UT-CS-94-234, University of Tennessee, Knoxville, Tennessee 37996-3450 Whaley, R. C , Petitet, A., Dongarra, J. J., "Automated empirical optimizations of software and the ATLAS project," Parallel Computing, 27, 1,2, pp.3-35, 2001
30
26. Peterson, L. L. and Davie, B. S., Computer Networks: A Systems Approach, Morgan Kaufmann (San Francisco), 1996 27. Stevens, W. R., Unix Network Programming, Prentice-Hall, 1990 28. Balay, S., Gropp, W., Curfman Mclnnes, L., Smith, B.,PETSc Users Manual, rev. 2.1.0, Argonne National Laboratory, 9700 South Cass Avenue, Argonne, Illinois 60439 29. Golub, G. H. and Van Loan, C. F.,Afatrix Computations, 3rd edition, John's Hopkins University Press, 1996 30. Choi, J., Dongarra, J., Ostrouchov, S., Petitet, A., Walker, D., Whaley, R. C.,A Proposal for a Set of Parallel Basic Linear Algebra Subprograms, Computer Science Department Technical Report UT-CS-95-292, University of Tennessee, Knoxville, Tennessee 37996-3450 31. HPL, High Performance Linpack benchmaxk,http://www. netlib. org/benchmark/hpl 32. Dongarra, J., Du Croz, J., Duff, I. S., Hammarling, S.,"A Set of Level 3 Basic Linear Algebra Subprograms," ACM Trans. Math. Soft., 14, pp. 1-17, 1988 33. Lichtenstein and Johnson, "Block-cyclic dense linear algebra," SI AM J. Sci. Stat. Compt., 14, pp.1259-1288,1993 34. Petitet, A. Algorithmic Redistribution Methods for Block Cyclic Decompositions, PhD thesis, Department of Computer Science, University of Tennessee, Knoxville, Tennessee 37996-3450 35. Beaumont, O., Legrand, A., Rastello, F., Robert, Y., "Dense linear algebra kernels on heterogeneous platforms: Redistribution issues,", Parallel Computing, 28, (issue 2), pp. 155-185, 2002 36. Kernighan, B. W. and Ritchie, D. M.,The C Programming Language, 2nd edition,Bei\ Telephone Laboratories, Inc., 1988 37. Snir, M. , Otto, S. W., Huss-Lederman S., Walker, D. W., and Dongarra, J. 3.,MPI: The Complete #e/erence,MIT Press,1996
GRID: Earth and Space Science Applications Perspective Luigi Fusco Senior Advisor for EO Applications ESA - ESRIN, 00044 Frascati, Italy [email protected]
The integration of space and information technologies is an asset for many science and operational applications in various disciplines. For example, limiting to the case of Earth Observation, many environmental satellites have been and are operated to provide large and repetitive volumes of geophysical measurements for natural resource management and environmental monitoring of our planet, which require sophisticated computing resources. The worldwide scientific user community is relatively large, in the order of many thousands, and the International programmes (e.g. IGBP, ICSU, WCRP, IOC) are the key consumers of EO data and its derived information. At the same time operational institutional and commercial users have real time and off-line access to data and services. At present, the provision of user information services is far from being optimal, due to the complexity of product format, algorithms and processing required to meet the specific user needs (e.g. immediate access to integrated EO and other information, for the specific application, in the end user system environment). To meet the user communities' aims it is required the effective co-operation of all involved actors and the sharing of experience, methods, resources, etc. Many obstacles have to be removed, for example: • access to large data archives (which today are mainly in close operational environment); • common definition of metadata and data formats (interoperability across various data providers); • access to relevant ancillary information; • adequate network capacity across space data handling dedicated facilities, value adding, science and service industries; • access to adequate computing resources; • development of real time services; • development of effective user tools for integrating different data sets; • agreements on common or compatible data policies for data access. The GRID technology (in terms of network, distributed and high performing computing power, collaborative environment and large data archive resources) can and should help
31
32 the process of supporting the user community in their scientific and operational applications. The European Space Agency' has plans for handling the dedicated archives, operational and future missions, and to support their scientific, operational and commercial exploitation. Efforts in demonstrating the potential of the GRID technology elements and services (e.g. middleware for distributed processing and access to distributed archives) are on going. The plans to encourage a European strong presence in this domain are essential actions to provide the necessary support and infrastructure as required by complex environmental applications.
e-Science, e-Business and the Grid
Tony Hey University of Southampton, UK
The talk begins with a rapid survey of the technological drivers behind the IT revolution. Increasingly, many areas of science involve access to distributed computing and data resources, remote access to specialized and expensive facilities and world-wide collaborations of scientists. There are many examples of such 'e-Science' endeavours ranging from bioinformatics and proteomics to collaborative engineering and earth observation. To enable scientists to maximize the science derived in such a fashion we will see the emergence of a new IT infrastructure called the Grid. The Web gave us easy access to html pages and information: the Grid will give us seamless access to a much wider range of distributed resources. It will enable us to form transient 'Virtual Organisations' without compromising security or privacy. The Grid infrastructure developed to allow interoperability and scalability of such heterogeneous and dynamic resources has obvious interest for industry. The talk concludes with some examples of Grid technology in an industrial context. The UK e-Science Programme is also briefly described.
33
G R A P H PARTITIONING FOR D Y N A M I C , A D A P T I V E A N D MULTI-PHASE SCIENTIFIC SIMULATIONS KIRK SCHLOEGEL, GEORGE ~KARYPIS, AND VIPIN KUMAR Dept. of Computer Science and Engineering University of Minnesota E-mail: kirk, karypis, kumarQcs.umn.edu The efficient execution of scientific simulations on HPC systems requires a partitioning of the underlying mesh among the processors such that the load is balanced and the inter-processor communication is minimized. Graph partitioning algorithms have been applied with much success for this purpose. However, the parallelization of multi-phase and multi-physics computations poses new challenges that require fundamental advances in graph partitioning technology. In addition, most existing graph partitioning algorithms are not suited for the newer heterogeneous high-performance computing platforms. This talk will describe research efforts in our group that are focused on developing novel multi-constraint and multi-objective graph partitioning algorithms that can support the advancing state-of-the-art in numerical simulation technologies. In addition, we will present our preliminary work on new partitioning algorithms that are well suited for heterogeneous architectures.
1
Introduction
Algorithms that find good partitionings of unstructured and irregular graphs are critical for the efficient execution of scientific simulations on highperformance parallel computers. In these simulations, computation is performed iteratively on each element (and/or node) of a physical two- or threedimensional mesh. Information is then exchanged between adjacent mesh elements. Efficient execution of such simulations on distributed-memory machines requires a mapping of the computational mesh onto the processors that equalizes the number of mesh elements assigned to each processor and minimizes the interprocessor communication required to perform the information exchange between adjacent elements 1 0 . Such a mapping is commonly found by solving a graph partitioning problem 3 ' 4 . Simulations performed on shared-memory multiprocessors also benefit from partitioning, as this increases data locality, and so, leads to better cache performance. Although the graph partitioning problem is NP-complete, good heuristic solutions for instances arising in scientific simulation can be found using multilevel algorithms. Many of these algorithms are incorporated in software packages such as Chaco 2 , METIS 6 , and SCOTCH 7 . However, the parallelization of multi-phase and multi-physics computa-
34
35
tions poses new challenges that require fundamental advances in graph partitioning technology. In addition, most existing graph partitioning algorithms are not suited for the newer heterogeneous high-performance computing platforms. This talk will describe research efforts in our group that are focused on developing novel multi-constraint and multi-objective graph partitioning algorithms that can support the advancing state-of-the-art in numerical simulation technologies. In addition, we will present our preliminary work on new partitioning algorithms that are well suited for heterogeneous architectures. 2
Multi-Constraint and Multi-Objective Partitioning Algorithms
The traditional graph-partitioning problem is a single objective optimization problem subject to a single balance constraint. Our research has focused on generalizing this problem to allow for multiple optimization objectives as well as multiple balance constraints by assigning weight vectors to the vertices and edges of the graph 5 ' 8 . The resulting generalized algorithms have enabled effective partitioning for a variety of applications such as weapon-target interaction simulations involving multiple computational meshes and particlein-cells computations that underlie diesel combustion engine simulations. As an example, consider a multi-physics simulation in which a variety of materials and/or processes are simulated together. The result is a class of problems in which the computation as well as the memory requirements are not uniform across the mesh. Existing partitioning schemes can be used to divide the mesh among the processors such that either the amount of computation or the amount of memory required is balanced across the processors. However, they cannot be used to compute a partitioning that simultaneously balances both of these quantities. Our inability to do so can either lead to significant computational imbalances, limiting efficiency, or significant memory imbalances, limiting the size of problems that can be solved using parallel computers. Figure 1 illustrates this problem. It shows three possible partitionings of a graph in which the amount of computation and memory associated with a vertex can be different throughout the graph. The partitioning in Figure 1(b) balances the computation among the subdomains, but creates a serious imbalance for memory requirements. The partitioning in Figure 1(c) balances the memory requirement, while leaving the computation unbalanced. The partitioning in Figure 1(d), that balances both of these, is the desired solution. In general, multi-physics simulations require the partitioning to satisfy not just one, but a multiple number of balance constraints. (In this case, the partitioning must balance two constraints, computation and memory).
36
Figure 1. An example of a computation with nonuniform memory requirements. Each vertex in the graph is split into two amounts. The size of the lightly-shaded portion represents the amount of computation associated with the vertex, while the size of the dark portion represents the amount of memory associated with the vertex. The bisection in (b) balances the computation. The bisection in (c) balances the memory, but only the bisection in (d) balances both of these.
Other examples of scientific simulations that require partitionings with multiple constraints and multiple objectives are presented in s , 1 ° . Serial and parallel algorithm for solving such multi-objective, multi-constraint graph partitioning problems are described in 5 ' 8 . 9 > n . New Challenges As numerical simulation technologies continue to become more sophisticated and as the number of processors routinely used increases into the thousands and tens of thousands, partitionings are required to satisfy more and more generalized constraints and optimize many different types of objectives in order to ensure good parallel efficiencies. Many of these objectives cannot be defined on the vertices and the edges of the graph, but are instead defined in terms of the overall structure of the partitioning. As an example, many types of the parallel indirect solvers require partitionings that, in addition to minimizing the inter-processor communication, are also composed of well-shaped subdomains (i.e., the subdomains have good aspect ratios) x. As another example, domain-decomposition-based numerical simulations, such as those proposed for computational structural mechanics, require that the resulting partition-
37
ing simultaneously balances (i) the amount of time required to factorize the local subproblem using direct factorization, (ii) the size of the interface problem assigned to each processor, and (iii) the number of subdomains to which each subdomain is adjacent. In both of these examples, the various objectives and constraints cannot be modeled by assigning appropriate weights to the vertices and/or edges of the graph, as they depend on the structure of the partitioning. Developing algorithms for solving such problems is particularly challenging, as it requires that the partitioning algorithm balance quantities that can be measured only after a partitioning has been computed. Also, in many time dependent computations the physics or subdomain meshes change as a function of time. For such computations to remain load balanced, the mesh must be redistributed periodically. This requires an adaptive repartitioning algorithm that has yet an additional optimization objective on top of any others specified by the user (i.e., the minimization of the amount of data that needs to be redistributed during load balancing). In our research, we are continuing to develop a general partitioning framework that allows constraints and objectives to be specified in terms of the structure of the desired partitioning as well as the development of algorithms suitable for this framework. 3
Partitioning for Heterogeneous Computing Platforms
Most existing scalable parallel multi-disciplinary simulation codes can be easily ported to a wide range of parallel architectures as they employ a standard messaging layer such as MPI. However, their performance on these depends to a large degree on the architecture of the parallel platform. In particular, many core parallel algorithms were designed under the assumption that the target architecture is flat and homogeneous. However, the emergence of parallel computing platforms built using commercial-off-the-shelf components has resulted in high-performance machines becoming more and more heterogeneous. This trend is also influenced by the geographically distributed nature of computing grids as well as the effects of increasingly complex memory hierarchies. This heterogeneity presents real challenges to the scalable execution of scientific and engineering simulation codes. A promising approach for addressing this problem is to develop a new class of architecture-aware graph partitioning algorithms that optimally decomposes computations given the architecture of the parallel platform. Ideally, such as intelligent partitioning capability could alleviate the need for major restructuring of scientific codes. We are in the process of developing graph-partitioning algorithms that take into account the heterogeneity of the underlying parallel computing ar-
38
chitecture, and hence, compute partitionings that will allow existing scientific codes to achieve the highest levels of performance on a wide range of platforms. The intent is to develop an extensible hierarchical framework for describing the various aspects of the target platform that captures the underlying network topologies, inter-connection network bandwidths and latencies, processor speeds, memory capacities, and the various levels of the memory hierarchy. New graph-partitioning algorithms will then be designed that can use information from this framework to optimize partitionings with respect to the specified architecture. Acknowledgments This work was supported by DOE contract number LLNL B347881, by NSF grants CCR-9972519, EIA-9986042, and ACI-9982274, by Army Research Office contracts DA/DAAG55-98-1-0014, by Army High Performance Computing Research Center cooperative agreement number DAAD19-01-2-0014, the content of which does not necessarily reflect the position or the policy of the government, and no official endorsement should be inferred. Additional support was provided by the IBM Partnership Award, and by the IBM SUR equipment grant. Access to computing facilities was provided by AHPCRC and the Minnesota Supercomputer Institute. References [1] R. Diekmann, R. Preis, F. Schlimbach and C. Walshaw. Shape-Optimized Mesh Partitioning and Load Balancing for Parallel Adaptive FEM. Parallel Computing 26, 12 (2000). [2] B. Hendrickson and R. Leland. The Chaco User's Guide, Version 2.0. Technical Report Sandia National Laboratories SAND94-2692, 1994. [3] B. Hendrickson and R. Leland. A Multilevel Algorithm for Partitioning Graphs. Proceedings Supercomputing '95, 1995. [4] G. Karypis and V. Kumar. A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs. SIAM Journal on Scientific Computing 20, 1 (1998). [5] G. Karypis and V. Kumar. Multilevel Algorithms for Multi-constraint Graph Partitioning. Proceedings Supercomputing '98, 1998. [6] G. Karypis, K. Schloegel and V. Kumar. ParMETIS: Parallel Graph Partitioning and Sparse Matrix Ordering Library, Version 3.0, 2002. [7] F. Pellegrini and J. Roman. Scotch: A Software Package for Static Mapping by Dual Recursive Bipartitioning of Process and Architecture
39
Graphs. HPCN-Europe, Springer LNCS 1067, 1996. [8] K. Schloegel, G. Karypis and V. Kumar. A New Algorithm for Multiobjective Graph Partitioning. Proceedings EuroPar '99, pages 322-331, 1999. [9] K. Schloegel, G. Karypis and V. Kumar. Parallel Multilevel Algorithms for Multi-constraint Graph Partitioning. Proceedings EuroPar2000, pages 296-310, 2000. [10] K. Schloegel, G. Karypis and V. Kumar. Graph Partitioning for High Performance Scientific Simulations. In CRPC Parallel Computing Handbook, Morgan Kaufmann. To appear. [11] K. Schloegel, G. Karypis and V. Kumar. Parallel Static and Dynamic Multi-constraint Graph Partitioning. Concurrency: Practice and Experience, To appear.
Panel Session on Challenges and Opportunities in Data -Intensive Grid Computing Paul Messina Center for Advanced Computing Research, CalTech [email protected] The panel will discuss software and hardware technology issues related to very large, geographically distributed scientific data sets: archiving, accessing, and analyzing them. Discussion topics would include distributed, on-demand computing for analysis, federation of databases for simultaneous access to various databases, data mining, network middleware, scalability, and distributed visualization, among others.
40
APPLICATIONS
This page is intentionally left blank
G I A N T EIGENPROBLEMS F R O M LATTICE G A U G E THEORY ON CRAY T3E SYSTEMS N. ATTIG 1 , TH. LIPPERT 2 , H. NEFF 1 , J. NEGELE 3 , AND K. SCHILLING 1 ' 2 John von Neumann Institute for Computing Research Center Mlich, 52425 Jiilich, Germany 2 Dept. of Physics, University of Wuppertal, 42097 Wuppertal, Germany 3 Center for Theoretical Physics, MIT, Cambridge, USA The determination of physical properties of flavor singlet objects like the if meson by computer simulation requires the computation of functionals of the inverse fermionic matrix A/ - 1 . So far, only stochastic methods could cope with the enormous size of M. In this paper, we introduce an alternative approach which is based on the computation of a subset of low-lying eigenmodes of the fermionic matrix. The high quality of this 'truncated eigenmode approximation' (TEA) is demonstrated by comparison with the pion correlator, a flavor octet quantity, which is readily computable through a linear system of equations. We show that TEA can sue- cessfully approximate the flavor singlet rf correlator. We find that the systematic error of the method is tolerable. As the determination of the chosen subset of 300 eigenmodes requires about 3.5 Tflops-hours CPU-time per canonical ensemble and at least 15 GBytes of memory, the power of high-end supercomputers like the CRAY T3E is indispensable.
1
Introduction
A major goal of non-perturbative lattice quantum chromodynamics (LQCD) is the determination of hadronic mass states which are characterized by nonvalence contributions, such as flavor singlet mesons. Their correlation functions, CV'(ti - t 2 ) , contain so-called 'disconnected diagrams', i.e. correlators between closed virtual fermion loops. The reliable determination of these disconnected diagrams has been a long- standing issue ever since the early days of lattice gauge theory. It can be reduced to the numerical problem of how to achieve information about functionals of the inverse fermionic matrix M _ 1 . The first attempts in this direction have been started only a few years ago, using the so-called stochastic estimator method (SE) * to compute the trace of M _ 1 . This approach requires to solve the linear system Mx = £ on some hundred source vectors £, with £ being Z2 or Gaussian noise vectors. Meanwhile, substantial progress could be achieved for the determination of the T]1 by application of refined smearing methods 2 , where for the first time a proper signal-to-noise-ratio could be established. However, SE introduces stochastic noise, in addition to the stochastics already inherited from the
43
44
Monte Carlo process. In the following, we describe the determination of the 77' mass based on the computation of a set of low-lying eigenmodes of Q = 75 M, the hermitian form of M. We use the implicitly restarted Arnoldi method, a generalization of the standard Lanczos procedure. A crucial ingredient is the Chebyshev acceleration technique to achieve a transformation of the spectrum to a form suitable for the Arnoldi eigenvalue determination. The low-lying modes given, it is possible to estimate the entire matrix Q~l and those matrix functionals or functions of Q and M which are sensitive to long-range physics. In section 1, we introduce the meson correlators and in section 2, we shortly review their computation by conventional means. Section 3 is devoted to TEA and the organization of the computation on the CRAY T3E by use of the parallel Arnoldi package (PARPACK). In section 4, we assess the viability of TEA by comparing the correlator of the TT meson as computed from TEA with the result from the conventional approach. As the IT is a flavor octet quantity it can easily be computed through the solution of a linear system of equations by iterative Krylov subspace algorithms 3 . Finally, we apply TEA to the computation of the 77' meson correlator and compare with results from SE computations. 2
Meson Correlators
In LQCD, hadronic masses are extracted from the large-time behavior of correlation functions. The correlator of the flavor octet n meson is defined as C»(t = t 1 - t 2 ) = / ^ T r [ g - 1 ( n 1 t 1 ; T n , * a ) g - 1 ( m l t 2 ; n > t i ) ] \ \n,m
,
(1)
I y
while the flavor singlet 77' meson correlator is composed of two terms, one being connected and equivalent to the pion correlator, the second being the disconnected contribution from the correlation of virtual quark loops: C„-(ti-<2) = a ( i 1 - t 2 ) - 2 / ^ T r [ Q - 1 ( n , t 1 ; n , t 1 ) ] T r [ Q - 1 ( m , t 2 ; m , t 2 ) \n,m
1 u
(2) (.. .)v indicates the average over a canonical ensemble of gauge field configurations. Q is the hermitian Wilson-Dirac matrix 3 , i.e. Q = 75M; n and m denote spatial lattice sites, ti and ti determine the time separation t. The color and Dirac indices are suppressed. For large times t, the respective correlation functions become proportional to exp(-m 0 £), where mo is the mass
45
of the particle described by the correlation function. Since our lattice has anti-periodic boundary conditions in time direction, the correlation functions actually consist of a sum over two contributions, exp(-m 0 i) and exp(+m 0 £), i.e. they will exhibit a cos/i-like behaviour (C ~ cosh(m0t)). 3
Conventional Computation of Correlators
In the conventional computation of the ^-correlator (1) the source point is held fixed (e.g. at index tupel (1,1) = (1,1)), where the first index symbolizes the spatial vector (1,1,1) and the second one denotes the time component: CW(*I) = / £ T T [ Q - H M I ; 1 , I ) Q - H I , 1 ; M I ) ] )
•
(3)
IV
\ n t
Using 75M75 = M , we obtain:
C,(ti) = (52Tr[M- 1 (n J t 1 ;l,l)Af- lt (l ) l;n,ti)]\ .
(4)
IV
\ n
Thus it suffices to determine 12 columns ( 3 x 4 for the color and Dirac indices) of M_1 in order to compute (4). The columns c(n, t) of M~x are obtained by solving the linear system M(n,ti;m,t2)c(m,t2)
= 6(1, l;n, t{);
(5)
where 6 is the Kronecker delta function. Of course, the statistics could be improved by averaging over many or even all source points. However this would be prohibitively expensive as the effort increases with the number of sources. For Cy, however, the second term, J2rft[Q~1(n,ti;n,t1)}Tr[Q-1(m,t2;m,t2)],
(6)
n,m
depends on the diagonal elements of Q~x which cannot be determined from one source point alone. Instead of going through all sites the method of choice is the stochastic estimator technique (SE). One creates series of complex numbers (a)j such that they converge to the diagonal elements Q~l(i, i). The series are constructed through noise vectors r)k, 3
(ci)i = Tir,k(i)Q-1rik(i)/j *=i
(7)
46
Q~lT)k is determined by solving the corresponding linear system Qx = 77. In order to achieve a satisfying approximation, 400 noise vectors r]k are required 2 . Therefore the flavor singlet calculations are about 30 times more expensive than the octet ones. 4
TEA
We start from the following equation:
Q-Hn,t1]m,t,) = £ ^ \ ^ m ^ i
At
t
(8)
(ipi\ipi)
where A* and tpi are the eigenvalues and the eigenvectors of Q respectively. Note that Q is hermitian indefinite0. We approximate the sum on the right hand side by restriction to the 300 lowest-lying eigenvalues and their corresponding eigenvectors. Due to the factor 1/A* one can hope that the low-lying eigenmodes will dominate the sum 6 . We emphasize that we have an approximation for the entire matrix Q _ 1 (i, j). Therefore, we can retrieve the diagonal elements of the disconnected diagrams as well as the n correlator on all source points. To compute the eigenvalues and their corresponding eigenvectors we employ the Implicitly Restarted Arnoldi Method (IRAM). The huge size of Q requires parallel supercomputers. We work on two CRAY T3E systems with 512 nodes each, located at the Research Center Jiilich in Germany and NERSC, Berkeley, USA. A comfortable parallel implementation of IRAM is provided by the PARPACK package 5 . In order to overcome the problem of slow convergence for the low-lying eigenvalues we apply the Chebyshev polynomial acceleration technique, where the eigenvalue spectrum is transformed such that the 300 smallest eigenvalues become much larger than the rest of the spectrum, a situation favorable for eigenvalue calculations by IRAM. We work on a 163 x 32 lattice i.e. 16 lattice sites in space and 32 lattice sites in time direction. Taking\into account the Dirac and color indices, we see that the Dirac matrix acts on a 12 x 163 x 32 = 1.572.864 dimensional vector space. This explains why we cannot invert the entire Dirac matrix, since this would need about 40 TByte memory space, whereas the determination of " T h e hermitian matrix Q leads to an orthogonal eigenbase. If we compute the eigenmodes from the non-normal matrix M instead, the resulting eigenbase is non-orthogonal. In our investigations, we found no convergence of the low modes from the non-orthogonal eigenbase. 6 In the result section, we will come back to the question of cancellation effects due to positive and negative eigenvalues.
47
300 low-lying eigenvectors leads to about 15 GByte memory space only. Our computations are based on canonical ensembles of 200 field configurations with n/ = 2 flavors of dynamical sea quarks, generated at 4 different quark masses, in the framework of the SESAM project 2 . Fast and automated access to an archive space of approximately 6 TBytes is required to store all eigenvectors from the SESAM ensembles. Thus, computations of this kind are not feasible without the facilities available at supercomputer centers. It takes about 3.5 Tflops-hours to solve for 300 low-lying modes on each ensemble. In the eigenmode approach the CPU-time is decreasing with lighter quark masses, as in that case the eigenvalues become smaller and one can expect that they will dominate the sum earlier. This is a substantial advantage of TEA compared to the SE approach, where smaller eigenvalues lead to a slower convergence of the linear system solvers. With 400 stochastic estimates it takes about 1.5 Tflops-hours to treat an ensemble of 200 gauge configurations. However, in future simulations (i.e. for lighter quark masses) TEA will soon become superior to SE. 5
Results
The correlator C-* can serve as simple test for the quality of TEA. Fig. 1 compares CV as determined by eq. (5) with the result from TEA. The low modes are expected to describe long range physics. -Accordingly, TEA underestimates the true Cff for small time separations (t < 8). OnHhe other hand, the large time behavior is represented quite well. Since the IT correlator can be determined extremely accurately by the conventional method, a tiny truncation effect is still visible in the large time range of the propagator. This deviation decreases for smaller quark masses, as well as for a larger number of eigenmodes. Nevertheless, the deviation for large t still are surprisingly small: Let us sum (1) over ti and t2,
U,t2
i
l
Here all contributions are positive and no cancellation occurs as in the case
for Cfo.e situation is even more favorable for the disconnected part of rj'. Let us again consider the sum over t\ and t
T±[Q-1(n,t1]n,t1)\Tr[Q-1(m,t3-,rn,t2J\
= ( E ^ )
•
<10>
48 Pion Correlation Function Conventional Method >—t-— Alternative Method >•••*•••
0.1 •
***
***,
0.001
0
5
10
15
20
25
30
time Figure 1. Comparison of CV from TEA and from the conventional computation.
Obviously, the positive and negative eigenvalues can cancel each other here. We find that convergence is achieved from about 150 eigenvalues on! TEA and SE agree well within the error bars that are due to the gauge field fluctuations. Twoloop Correlation Function
m*F 0.001
0.0001
Conventional Melhod • — i — Alternative Method ••••*••
¥*.
"?*,
%i
10
**
mm^m 15 time
20
if"
25
30
Figure 2. Comparison of the disconnected correlator from SE and TEA. The large errors in both cases are solely due to gauge field noise.
In conclusion our results show that TEA is comparable to SE for the computation disconnected diagrams 4 . The costs are similar to SE computations at present. Going to realistic light masses, TEA will become superior to SE
49
since low modes tend to dominate more and more. Acknowledgments We thank the staff of the computer centers at Research Center Julich and NERSC at Berkeley for their friendly support. References 1. 2. 3. 4. 5.
S. Giisken et. al. Phys. Rev., D59:114502, 1999. T. Struckmann et. al. hep-lat/0010005. A. Frommer et al. Int. J. Mod. Phys., C5:1073, 1994. see also L. Venkataraman and G. Kilcup, hep-lat/9711006. http://www.caam.rice.edu/~kristyn/parpack_nome.html
PARALLEL CONSISTENCY CHECKING OF AUTOMOTIVE PRODUCT DATA WOLFGANG BLOCHINGER, CARSTEN SINZ AND WOLFGANG KUCHLIN Symbolic Computation Group, WSlfor Computer Science, Universitdt Tubingen, 72076 Tubingen, Germany http://www-sr.informatik.uni-tuebingen.de This paper deals with a parallel approach to the verification of consistency aspects of an industrial product configuration data base. The data base we analyze is used by DaimlerChrysler to check the orders for cars and commercial vehicles of their Mercedes lines. By formalizing the ordering process and employing techniques from symbolic computation we could establish a set of tools that allow the automatic execution of huge series of consistency checks, thereby ultimately enhancing the quality of the product data. However, occasional occurrences of computation intensive checks are a limiting factor for the usability of the tools. Therefore, a prototypical parallel re-implementation using our Distributed Object-Oriented Threads System (DOTS) was carried out. Performance measurements on a heterogeneous cluster of sharedmemory multiprocessor Unix workstations and standard Windows PCs revealed considerable speed-ups and substantially reduced the average waiting time for individual checks. We thus arrive at a noticeable improvement in usability of the consistency checking tools.
1
Introduction
Today's automotive industry manages to supply customers with highly individualized products by configuring each vehicle individually from a very large set of possible options. E.g., the Mercedes C-class of passenger cars allows far more than a thousand options, and on the average more than 30,000 cars will be manufactured before an order is repeated identically. Heavy commercial trucks are even more individualized, and every truck configuration is built only very few times on average. The space of possible variations is so great that the validity of each order needs to be checked electronically against a product data base which encodes the constraints governing legal combinations of options1. But the maintenance of a data base with thousands of logical rules is error-prone, especially since it is under constant change due to the phasing in and out of models. Every fault in the data base may lead to a valid order rejected, or an invalid (non constructible) order accepted which may ultimately result in the assembly line to be stopped. Therefore, reaching correctness of the product data base is a high priority goal. DaimlerChrysler employs the electronic product data management (EPDM) system DIALOG for the configuration of their Mercedes lines. Within this system, a customer's order consists of a basic model class selection together with a set of further equipment codes describing additional features. Each equipment code is represented by a Boolean variable, and choosing some piece of equipment is reflected by setting
50
51
the corresponding variable to true. An order is processed in three major steps, as depicted in Figure 1. All of these steps are controlled by logical rules of the EPDM system: 1. Order completion: Supplement the customer's order by additional (implied) codes. 2. Constructibility check: Are all constraints on constructive models fulfilled by this order? 3. Parts list generation: Transform the (possibly supplemented) order into a list of parts (a bill of materials).
Customer's Order
Supplemented Customer's Order
Checked and Supplemented Customer's Order
Order's Parts List
Figure 1. Processing a customer's order.
In order to systematically detect defects in the rule system, we developed a formal model of the ordering process. Thus, we are able to apply an automatic theorem prover to check certain consistency criteria of the rule base as a whole: Necessary and inadmissible codes: Are there codes which must invariably appear in each constructible order? Are there codes which cannot possibly appear in any constructible order? Superfluous parts: Are there parts which cannot occur in any constructible order? Both criteria can be formulated as propositional logic satisfiability (SAT) problems2, and our interactive consistency support tool BIS 3 contains an implementation of a Davis-Putnam-style4 propositional prover to verify them. To completely check the above mentioned criteria for only one model class, up to 10,000 prover runs have to be performed. Most of these automatic proofs are completed in a few seconds, but there remains a small fraction that requires comparatively high run-times of up to several hours. Unfortunately, there is no known method to estimate the run-times in advance, and in an interactive system like BIS, users hardly accept long and unpredictable waiting times. It may happen that one of the first in a sequence of proofs requires a very long run-time, which causes a delay
52
in the presentation of the results of all the other proofs: The user does not get any result until the first proof is completed. Our parallelization approach therefore is two-fold: we execute a set of proofs in parallel, and if we hit a long-running proof we additionally process this individual proof in parallel. Thus, the average waiting time for the result of an individual proof can be reduced considerably. Before we describe our parallelization in more detail, we will give an overview of our software infrastructure DOTS used in this approach. We will then explain our method in terms of the DOTS system, before presenting experimental results, related work, and our conclusions. 2
Parallelization Infrastructure
As parallelization infrastructure, our Distributed Object-Oriented Threads System DOTS5 is used. DOTS is a C++ parallel programming toolkit that integrates a wide range of different computing platforms into a single system environment for high performance computing. 2.1
The programming paradigm of DOTS
The main idea of DOTS is to make the threads programming paradigm used on shared-memory machines available in a distributed memory environment. With DOTS, a hierarchical multiprocessor, consisting of a (heterogeneous) cluster of shared-memory multiprocessor systems, can be efficiently programmed using a single paradigm. The DOTS API provides primitives for DOTS thread creation (dots.fork), synchronization with the results computed by other DOTS threads (dots-join), and DOTS thread cancellation (dots.cancel). All primitives can also be used in conjunction with so called thread groups. Thread groups are a means of representing related DOTS threads. When applied with thread groups, the semantics of each primitive is automatically changed to the appropriate group semantics. E.g. when using the join primitive with a thread group, join-any semantics will be applied. 2.2
The Architecture of DOTS
Figure 2 gives an overview of the basic components of the architecture of DOTS. When a DOTS thread is created with the dotsJork primitive, a so called thread object is instantiated that represents the DOTS thread within the system during the complete execution process. It stores all information that is necessary to execute the DOTS thread.
53 Execution Unit i;-lrM'-l.|.-hl4.k-in-l-l--n thread queue
_
Figure 2. The DOTS Architecture
ready queues
Figure 3. The Execution Unit
DOTS threads are executed within the Execution Unit (see Figure 3). It contains a thread queue in which newly created thread objects are enqueued. A pool of (OS native) worker threads dequeue thread objects from the queue and execute the corresponding DOTS threads. The number of worker threads can be determined by the programmer. Normally, for each node the number of available processors is chosen. After the execution of a DOTS thread is completed, its thread object is placed into a ready queue. To support the execution of DOTS threads in a distributed environment, the DOTS architecture includes additional components. The Thread Transfer Unit transfers (serialized) thread objects between queues of execution units residing on different nodes. The Load Monitoring Framework traces all events concerning the execution of DOTS threads and provides status information like the current load or the current length of the thread queue. Based on the Load Monitoring Framework, different load distribution strategies can be implemented. A load distribution strategy is responsible for triggering the transfer of thread objects and selecting destination nodes according to a particular strategy. 3
Parallelization
The presented parallelization approach pursues two major goals. The first goal is to achieve a total speedup of the computation. A second important goal is to reduce the average waiting time for the result of a proof in order to improve the usability of the application. In the subsequently described procedure, hard proofs are determined by setting a limit for the computation time for a proof. If the time for a proof has expired, it is considered as a hard proof and treated separately by executing it in parallel. Consequently, the parallel execution is organized in two phases: • Phase 1: Concurrent execution of proofs.
54
A timeout is set to suspend (long-running) hard proofs. Hard proofs are queued along with their current execution state. • Phase 2: Parallel execution of the queued hard proofs. Each queued proof is treated individually in parallel, starting from its previously saved state computed in phase 1. The execution of phase 1 is organized in a master-slave approach. For all proofs, the root thread creates corresponding DOTS threads that are executed concurrently. The computed results are joined by the root thread and displayed. In the case of a timed out hard proof, its current execution state is joined and queued for later execution in phase 2. The realization of phase 2 requires more sophisticated techniques. We adopted the parallelization scheme for the Davis-Putnam algorithm presented by Zhang et a/.6. Basically, we are dealing with the parallelization of a combinatorial search problem. This implies that the search space has to be divided into mutually disjoint portions to be treated in parallel. However, a (static) generation of balanced subproblems is not feasible, since it is impossible to predict in advance the extent of the problem reduction delivered by the Davis-Putnam procedure. Instead, a dynamic search space splitting approach is carried out. To start the execution of a queued hard proof the root thread forks one DOTS thread that has the entire search space assigned. During the whole computation of phase 2, all DOTS threads periodically monitor the length of the local thread queue (see Section 2.2). If the thread queue is empty, a new DOTS thread is forked. The parent thread splits off a region of its search space and assigns it to the new DOTS thread. Details of the applied search space splitting heuristics can be found in Zhang et a/.6. To prevent uncontrolled splitting actions, a predefined time interval has to be waited before the next split can be carried out by the DOTS thread. The newly created DOTS thread is queued and can be executed by another local worker thread or can be transferred to other nodes (see Section 3.1). The described splitting procedure generates subproblems on demand. This ensures that new subproblems are generated during the initialization phase of the computation to exploit the available processing capacity and every time a subproblem has been completely processed without finding a solution. After forking the initial DOTS thread, the root thread calls dotsjoin to wait for the created DOTS threads. All DOTS threads (except the initial one) are created with the dots-subfork primitive. This means that they can be joined by the root thread (and are not joined by their actual parent threads). The result of a DOTS thread indicates whether a problem solution was found within its assigned search region or not. The processing of a hard proof is completed either if all created DOTS threads have been joined without returning a solution, or when the first DOTS thread that has found a
55
solution is joined. In the latter case, all remaining DOTS threads are immediately canceled to make all processing capacities available for the next queued hard proof. 3.1
Load Distribution
As described in Section 2.2, new load distribution strategies can easily be integrated into DOTS using the Load Monitoring Framework. For the presented application, a customized load distribution strategy was realized that reflects the division of the computation in two phases. In both phases, a work-stealing strategy is applied, i.e. when all worker threads on a node are idle, the distribution strategy tries to transfer thread objects from the thread queues of other nodes. However, the phases are treated differently in the way how the victim node is chosen. Whereas in phase 1 always the master node is selected as victim (all DOTS threads are created by the master in phase 1), victims are chosen randomly in phase 2. It has been shown that applying a randomized work-stealing strategy to distribute the load in backtrack search algorithms is likely to yield a speedup within a constant factor from optimal (when all solution are required)7. Since this approach does not involve central components for load distribution, the scalability of the parallel application is improved. 4
Experimental Results
The parallel environment used for the presented performance measurements consisted of a cluster made up of the following components (all nodes were connected with 100 Mbps switched Fast-Ethernet). • 2 Sun Ultra E450, each with 4 UltraSparcII processors (@400 MHz) and 1 GB of main memory, running under Solaris 7. • 4 PCs, each with 1 Pentiumll processor (@400MHz) and 128 MB of main memory, running under Windows NT 4.0. We have applied the necessary and inadmissible codes checks as well as the superfluous parts checks on configuration data for the C-class and E-class limousines. Tables 1 and 2 show the measured run-times (wall-clock times) in seconds. A timeout of 20 seconds for detecting hard proof was chosen. The tables give sequential runtimes of the prover executed on an E450, on a PC, and the corresponding weighted mean of the sequential run-times. Additionally, run-times of three parallel program runs on all processors are shown. The parallel execution of the prover can exhibit a
56 Table 1. Results for the Mercedes C-class. C-class limousines (code checks) proofs: 520 hard proofs: 4 waiting time run-time 1,417.0 654.6 seq (E450) 1,738.0 800.3 seq (PC) seq (mean) 1,524.0 703.2 parallel 158.0 66.8 (3 runs) 162.0 66.6 164.0 67.1
C-class limousines (part checks) proofs: 512 hard proofs: 6 run-time waiting time seq (E450) 10,447.0 5,459.5 seq (PC) 13,667.0 7,167.1 seq (mean) 11,520.3 6,028.7 parallel 1,026.0 100.6 (3 runs) 1,043.0 100.8 1,039.0 100.8
Table 2. Results for the Mercedes E-class. E-class limousines (code checks) proofs: 525 hard proofs: 6 waiting time run-time seq (E450) 1,935.0 956.1 seq (PC) 2,281.0 1,123.9 seq (mean) 2,050.3 1,012.0 parallel 253.0 111.4 (3 runs) 245.0 110.6 252.0 111.5
E-class limousines (part checks) proofs: 500 hard proofs: 6 run-time waiting time seq (E450) 40,186.0 16,370.5 seq (PC) 51,769.0 21,500.4 seq (mean) 44,047.0 18,080.5 parallel 2,198.0 110.4 (3 runs) 2,174.0 109.3 2,192.0 108.8
non-deterministic behavior, for this reason we don't give averaged times or speedupvalues. The average waiting times were calculated by Y^ waiting Jbirne{i) £—1 Sproofs where waiting Jtime(i) is defined as the time from the start of the whole set of tests until the result of proof i is reported. Since some of the detected hard problems in the considered examples turned out to be satisfiable, the parallel execution of these proofs can lead to super-linear speedups and may consequently greatly reduce the total run-time as well as the mean waiting time for a proof. 5
Related Work and Conclusion
In the realm of industrial verification, Spreeuwenberg et al.s present a tool to verify knowledge bases built with Computer Associate's Aion9. Concerning parallel satisfiability checking, PSATO of Zhang et a/.6 is a distributed prover for propositional
57
logic on a network of workstations. In contrast to our work, a master-slave model is applied, where a central master is responsible for the division of the search space and for assigning the subtasks to the slaves. We see the main contribution of our paper in presenting an industrial realworld application of symbolic computation where employment of parallelization techniques greatly enhances usability by reducing the user's waiting time to an acceptable amount. Sophisticated parallel execution and scheduling methods almost manage to overcome the unpredictability of proof times. References 1. E. Freuder. The role of configuration knowledge in the business process. IEEE Intelligent Systems, 13(4):29-31, July/August 1998. 2. W. Kuchlin and C. Sinz. Proving consistency assertions for automotive product data management. J. Automated Reasoning, 24(1-2): 145-163, February 2000. 3. C. Sinz, A. Kaiser, and W. Kuchlin. Detection of inconsistencies in complex product model data using extended propositional SAT-checking. In FLAIRS'01, 2001. To appear. 4. M. Davis and H. Putnam. A computing procedure for quantification theory. In Journal of the ACM, volume 7, pages 201-215, 1960. 5. Wolfgang Blochinger, Wolfgang Kuchlin, Christoph Ludwig, and Andreas Weber. An object-oriented platform for distributed high-performance symbolic computation. Mathematics and Computers in Simulation, 49:161-178, 1999. 6. H. Zhang, M. P. Bonacina, and J. Hsiang. PSATO: A distributed propositional prover and its application to quasigroup problems. Journal of Symbolic Computation, 21:543-560, 1996. 7. R. M. Karp and Y. Zhang. Randomized parallel algorithms for backtrack search and branch-and-bound computation. Journal of the ACM, 40(3):765-789, July 1993. 8. S. Spreeuwenberg, R. Gerrits, and M. Boekenoogen. VALENS: A Knowledge Based Tool to Validate and Verify an Aion Knowledge Base. In ECAI 2000, 14th European Conference on Artificial Intelligence, pages 731-735. IOS Press, 2000. 9. S. Garone and N. Buck. Capturing, Reusing, and Applying Knowledge for Competitive Advantage: Computer Associate's Aion. International Data Corporation, 2000. IDC White Paper.
IMPLEMENTATION OF AN INTEGRATED EFFICIENT PARALLEL MULTIBLOCK FLOW SOLVER THOMAS BONISCH, ROLAND RUHLE High Performance Computing Center Stuttgart (HLRS), Allmandring 30, 70550 Stuttgart, Germany Email: boenisch(a).hlrs. de This paper describes the effort taken to introduce a parallel multiblock structure into the URANUS simulation code for calculating reentry flows on structured c-meshes. The new data structure, the handling of block sides at physical and inner boundaries and the load balancing approach are presented. Target is the efficient calculation of reentry flows using multiblock meshes on current and future supercomputer platforms.
1
Introduction
Calculating flows around space vehicles during the reentry phase of their mission is a challenging task. Besides the normal flow one has to consider the occurring chemical reactions as they have a significant influence on the flow. The reason for these chemical reactions is the high temperature of the gas flow during the reentry, while the space vehicle is slowed down by the friction of the air. At these temperatures the air's components mainly nitrogen and oxygen react with each other. Modern space vehicles have complex geometries. To calculate flows around such a geometry, there exist several approaches: unstructured meshes, block structured meshes and other technologies like hybrid or overset meshes. Unstructured meshes can be generated automatically, mesh generation is much easier than generating a multiblock mesh. But the calculation of a flow using them costs a lot of computational power as indirect memory access methods have to be used. Therefore, efficient cache usage and vectorization is limited. As the development of the memory speed cannot keep track with the increase in processor performance this will be worse in the future. The second method is using structured meshes, but allowing several blocks of them to be assembled in an unstructured way. So, it is possible to mesh complex geometries. The calculation of such structured topologies profits from cache and vector technology. A good performance is much easier to obtain and it is normally higher than with unstructured meshes. But the mesh generation is a costly task. Sometimes it costs several weeks of an expert to generate a good so called multiblock mesh. The existing URANUS flow solver uses structured 3D C-meshes for its calculation. Considering this and the presented arguments we decided for the more natural step and using multiblock meshes for our calculations instead of rewriting the code completely to handle unstructured meshes.
58
59 2
The flow solver URANUS
In the URANUS (Upwind Relaxation Algorithm for Nonequilibrium Flows of the University Stuttgart) [1.2] flow simulation program the unsteady, compressible Navier-Stokes equations in the integral form are discretized in space using the cellcentred finite volume approach. The inviscid fluxes are formulated in the physical coordinate system and are calculated with Roe/Abgrall's approximate Riemann solver. Second order accuracy is achieved by a linear extrapolation of the characteristic variables from the cell-centres to the cell faces. TVD limiter functions applied on forward, backward and central differences for non-equidistant meshes are used to determine the corresponding slopes inside the cells, thus characterizing the direction of information propagation and preventing oscillation at discontinuities. The viscous fluxes are discretized in the computational domain using classical central and one-sided difference formulas of second order accuracy. Time integration is accomplished by the Euler backward scheme. The resulting implicit system of equations is solved iteratively by Newton's Method, which theoretically provides the possibility of quadratic convergence for initial guesses close to the final solution. The time step is computed locally in each cell from a given CFL number. To gain full advantage of the behaviour of Newton's method, the exact Jacobians of the flux terms and the source term have to be determined. The resulting linear system of equations is iteratively solved by the Jacobi line relaxation method with subiterations to minimize the inversion error. A simple preconditioning technique is used to improve the condition of the linear system and to simplify the LU-decomposition of the block-tridiagonal matrices to be solved in every line relaxation step. The boundary conditions are formulated in a fully implicit manner to preserve the convergence behaviour of Newton's method [3]. The usage of the sequential URANUS program on high-end workstations and on vector computers shows that the compute time and the memory requirements especially when changing to real gas models or using fine meshes for real world problems are too high to use the program on these platforms without the ability to use processors in parallel. Furthermore, with the so far used c-meshes many topologies cannot be meshed. The geometric singularity of such 3D c-meshes is complicated to handle and it often limits the convergence speed. The solution for these problems is to introduce a multiblock structure into this code which is also parallelized because only massively parallel platforms and modern hybrid parallel computers can fulfil the program's needs in memory size and computing power.
3
Requirements to an Efficient Parallel Multiblock Solver
To be able to calculate correct solutions on a multiblock mesh in parallel, a flow solver has to have several additional features. In the serial program using c-meshes
60
each of the different physical boundaries is fixed to a dedicated plane of the mesh. This means, e.g. that the outflow boundaries are always fixed to the mesh plane, where the first index reaches its maximum. In the multiblock case this is not longer true. Theoretically, each boundary type can appear at each of the six plsnes cf the 3D mesh. However, usually the mesh is generated in a way where the most complicated boundary if fixed to one specific plane of a block. 3o ; the boundary with the condition of the body wall has only to be implemented for one index direction. Nevertheless, this limits the usability of the code, because this does not work for some special topologies where at least one of the blocks has to calculate wall boundary conditions at least at two of its boundaries. The ability of turning the mesh blocks such mat always the same block surface fits to the body wall requires that each of the blocks can have its cv/n local coordinate system. This is necessary anyway as the bloc!cs itself are disposed unstructured around the geometry. At particular points where more ere less than four blocks in 2D or eight blocks in 3D respectively are connected together,
Figure 1. The different local coordinate systems and how they occure
the local coordinate system of at least one of the blocks neighbours has to be different (Figure 1). Furthermore, in a multiblock mesh each block can have multiple neighbours on each of its block sides, not only one as it happens, when you are cutting a C-mesh into parts, or when using special mesh generators. As we are not using our own meshes within this project, we are depending on meshes of our partners. Additionally, we wanted to be able to calculate results for as many cases and with as much project partners as possible. So we were not interested in limiting ourselves too much in choosing one special multiblock mesh type. We wanted to keep a high flexibility of our code. So, using meshes from project partners should be as easy as adapting and extending the L/5ut routines. The program parts for the parallelization the multiblock structure and the flow simulation core should be separated as far as possible. This increases maintainability as the modeling people still can recognize and update thsir flow patterns as the parallelization people can enhance the parallel performance.
61 As the flow simulator mentioned is memory and calculation intensive we aim at using the newest available hardware platforms. This requires a portable code to be able to move easily from one platform to another. In the parallel case this requires the usage of a standard parallel programming model. In order to be able to run on all platform types, like MPP's, SMP's and the new hybrid architectures consisting of distributed SMP nodes, we decided to use MPI as communication library. Currently, we use a pure distributed memory approach as this performs also well on today's SMP machines and on hybrid architectures. In a future version it would also be possible to add OpenMP directives to profit in addition from SMP architectures. A pure SMP approach was not considered, because this would limit the platforms to be used. Additionally, the experience of NASA shows that such an approach on today's ccNUMA architectures requires an effort comparable to an implementation with MPI to achieve a good performance[4]. 4
The parallel multiblock approach
The new parallel program needs a newly designed data structure to meet the requirements of the multiblock structure. Using Fortran90 we have the whole ability of dynamic and structured data types. Therefore, it is not longer necessary to implement a data management which stores the 3D arrays of all blocks contiguously into a one dimensional array or to waste memory by allocating the same amount of memory for each block calculated on one processor, like it was usual when using FORTRAN77 [5]. 4.1
The new data structure
In the new multiblock flow simulation program all information regarding one block is combined in a new data type. Each block is a instance of this data type having exactly the size the block needs. The blocks residing onto one process are organized in a pointered list of these instances. So, there is no waste of memory as only the really required memory is allocated. There is no additional programmer effort in manually handling the memory layout as the run time system cares for this. The information is easily accessible and the data structure itself is easy to extend to meet future requirements. The data structure for a block includes the ability to use several meshes with different resolutions for one block. So the data structure is already prepared for additional features like local refinement and multilevel or multigrid technologies. 4.2
Domain Decomposition and data exchange between the blocks
For the parallelization we used the domain decomposition approach. Each block has a two cell overlap region at the inner boundaries to maintain the second order scheme without additional communication. The values of the overlap region
62
are stored according to the local mesh values. This means, that they are stored converted to the local coordinate system of the block, even if the coordinate system of the block where they are originally located is different. Consequendy, the conversion of the halo cell data to the particular local coordinate system is done during the data transfer between the blocks. This communication between neighbours is necessary to update the results of the cells in the overlap region. During the solving step additional communication between the neighbours is performed to exchange intermediate data. This ensures a more accurate solution and a better convergence. Due to the irregular structure of the mesh blocks within the topology, the identification number of a neighbour block cannot be calculated out of the blocks own number and the block side. Additionally, a block can have several neighbours at each of its sides as mentioned above. Accordingly, there is a data structure implemented to store all the information about the relationship of the block and its particular neighbours. This information enfold the block number and the block side the local block is connected to, the neighbour blocks processor as far as the orientation of the neighbour block and the part of the local blocks side which is adjoined to the neighbour. Each block is treated separately on its processor. There is no difference in the handling of its neighbour blocks independent where they are located. This means, the communication between two blocks residing on the same processor is also done using the regular communication routines of the program. In this case from the processor point of view the processor is sending a messages to itself. A specific data structure also exists for each physical boundary type of a block, where all the data of one physical boundary type is specified. It contains the subtype and the exact position of this physical boundary on the block. Using this data structure, there is no branching necessary for each of the six block sides where a physical boundary can reside. Additionally, all physical boundaries of one type at a block can be handled efficiently in a loop. 4.3
Load Balancing
To obtain good performance on any kind of parallel computers it is essential to have a good load balance between the processors allocated to the parallel job. Thus, we should allocate the same portion of load to each processor. In our case this means that every processor should make calculations for more or less the same number of mesh cells. Compared to an approach where the blocks which are too large are cut in the middle as long as there are empty processors[6], we used a different technique: we also have to cut all blocks which are too large to fit onto one processor. But our cutting algorithm first calculates the number of necessary pieces for each block to be cut into. This number must only contain powers of two and powers of three as divider. If the calculated number does not accomplish this requirement the next appropriate larger number of pieces doing so is calculated. As the resulting parts may become to small we allow a slight overload of the processors
63
Figure 2. Different neighbour configurations
to avoid cutting into too much pieces. Then, the number of cuts necessary in each of the three dimensions is calculated in a way that the resulting blocks are not misshapen. This is the reason for allowing only part numbers containing powers of two and three. Consider the case if we have to cut a block into fourteen pieces. Then, we have to cut one time in one dimension, six times in a second and we cannot divide in the third dimension. If we are dividing the block into 16 parts instead, we can freely choose how to place the cuts within the three dimensions. This strategy has an additional advantage as we are cutting all the blocks in that way: assume a block being divided two times in one dimension and its neighbour being divided four times in the same dimension. Assuming the same local coordinate system, two of the three block parts have to communicate with two neighbour blocks each and one has to talk to three neighbours (figure 2). Having six
Figure 3. Solution of an Nonequilibrium Euler flow around X-38, angle of attack 40 degrees, mach number 19.8, printed is the mach number
64
parts within this dimension on the neighbouring block, each of the three blocks has two neighbours and the dependencies are simpler. It can happen that we have to adjust the cutting line by one cell to ensure this (but this is not yet implemented). Now, we have to rearrange the resulting blocks in a way that no processor is overloaded. For this step we use the load balancing tool jostle[7]. Jostle was primarily designed to be used for unstructured meshes, but it does not know anything about meshes itself. As any other load balancing tool jostle works on graphs. So, we only have to translate our new block structure, obtained after the cutting step, with its connections to the neighbours into a weighted graph where the block size is the weight of the node in the graph which corresponds to the block. If possible we get a load balanced block distribution and have to redistribute the blocks in the way proposed by jostle. As each block has its own data structure and is handled separately, it is easy to redistribute the blocks. Nevertheless, the time needed for this data transfer may be significant. 500 Cra --T3E
450
*
400 350
9-
300
t
290
a.
200
^
/
<£
190 100 90 0 0
50
100
190
200 250 300 Prazsssorzahl
350
400
490
500
Figure 4. Speedup for 20 Iterations on a 192 000 cell colibri mesh on the Cray T3E
5
Results
The obtained parallel multiblock flow solver has been used to perform reentry calculations for the colibri capsule and the X-38 (Figure 3), the prototype of the crew rescue vehicle for the international space station (ISS). Due to the usage of Fortran90 and MPI the code is portable and was tested on NEC SX-5, Hitachi SR8000, Cray T3E, IBM SP3 and a DEC alpha cluster. The convergence for the tested cases is good. The performance on the NEC-SX5e with a well sized problem
65
is nearly 1.5 Gflops which is 37% of the peak performance. The scaled speedup on the Cray T3E is given in table 1. Table 1.: Scaled speedup on the Cray T3E Mesh size 24 000 192 000
Mesh blocks 5 5
Blocksno. for calculation i.e. Proc. count 18 144
Simulation time 255.3 285.7
Efficency 1.0 0.893
A speedup measurement for a case with 192 000 mesh cells (figure 4) was also done on the Cray T3E. To compare the times 20 iterations were calculated. The smallest processor number possible with this case is 72 because of memory limitations. The given speedup values are compared to the case with 72 processors. The reason for the shown superlinear speedup for 108 and 216 processors is a better block shape in this cases compared to that running on 72 processors. 6
References 1. Scholl, E., Fruhauf, H.-H.: An Accurate and Efficient Implicit Upwind Solver for the Navier-Stokes Equations in Notes on Numerical Fluid Mechanics, Numerical Methods for the Navier-Stokes Equations, Hebecker F.-K., Ranacher R., Wittum G. (Eds.), Proceedings of the International Workshop on Numerical Methods for the Navier-Stokes Equations, Heidelberg, Germany, October 1993, Vieweg, 1994. 2. Friihauf, H.-H., DaiB, A., Gerlinger, U., Knab, O., Scholl, E.: Computation of Reentry Nonequilibrium Flows in a Wide Altitude and Velocity Regime, AIAA Paper 94-1961, June 1994. 3. Gerlinger, U., Fruhauf, H.-H., Bonisch, T. : Implicit Upwind Navier-Stokes Solver for Reentry Nonequilibrium Flows, 32nd Thermophysics Conference, AIAA 97-2547, 1997. 4. R.B. Ciotti, J.R. Taft, XPetersohn: Early Experiences with the 512 Processor Single System Image Origin2000, Proceedings of the 42nd CUG Conference (CUG Summit 2000), 2000 (CDROM) 5. F.S. Lien, W.L. Chen and M.A. Leschziner: A Multiblock Implementation of a Non-Orthogonal, Collocated Finite Volume Algorithm for Complex Turbulent Flows, International Journal for Numerical Methods in Fluids, Vol. 23, pp. 567-588, 1996 6. H.M. Bleecke et al: 'Benchmarks and Large Scale Examples' in A. Schiiller 'Portable Parallelization of Industrial Aerodynamic Applications (POPINDA)', Notes on Numerical Fluid Mechanics, Volume 71, Vieweg, 1999 7. C. Walshaw, M. Cross and M. Everett, 'Parallel Dynamic Graph Partitioning for Adaptive Unstructured Meshes', Journal of Parallel and Distributed Computing, 47(2), pp. 102-108, 1997
D I S T R I B U T E D SIMULATION E N V I R O N M E N T FOR COUPLED CELLULAR AUTOMATA IN JAVA M A R C U S B R I E S E N ' A N D J O R G R. W E I M A R * Institute of Scientific Computing, Technical University Braunschweig, D-38092 Braunschweig, Germany ; E-mail: J.Weimarffltu-bs.de
We describe a software environment for the coupling of different cellular automata. The system can couple CA for parallelization, can couple CA with different lattice sizes, different spatial or temporal resolutions, and even different state sets. It can also couple CA to other (possibly non-CA) simulations. The complete system, which is integrated with the CA simulation system JCASim, is written in Java and and uses remote method invocation as the communication method. The parallel efficiency is found to be satisfactory for large enough simulations.
1
Introduction
The concept of cellular automata is a framework for simulating highly parallel processes. They are very easy to parallelize and are often called "embarrassingly parallel". Usually, a cellular automaton consists of a regular lattice of cells, each of which contains a state from a finite set of states. From one time step to the next, all cells change their state in parallel depending on their current state and the state of a finite set of neighbors. Since the lattice is uniform, cellular automata are well suited only for the simulation of spatialtemporal phenomena in fairly uniform spatial domains. Some non-uniformity can be introduced by the initial conditions, but in other cases it would be better to combine different cellular automata for different regions in space. In this paper we demonstrate several types of combinations of cellular automata in the context of a cellular automata simulation system written in Java, JCASim 1 ' 2 . In this system, cellular automata can be described in a compact way using Java, or in a specialized CA language, CDL (which is then translated into Java). The system allows the simulation of cellular automata in one, two, and three dimensions, has several built-in boundary conditions, and allows the display of CA using text, colors, or icons.
'Present address: disy Informationssysteme GmbH, Stephanienstr. 30, D-76133 Karlsruhe, Germany. tTo whom correspondence should be addressed.
66
67
We consider five different types of coupling between cellular automata (CA), which can also be combined:
1.1
Coupling identical CA
If one or more identical CA are coupled through interactions at their borders, this can be used as a parallelization strategy if the simulation of the CA runs on different processors or computers. This option can also be used to model simulation domains which can not efficiently be covered by one rectangle, e.g., an L-shaped domain. Of course, the coupling interface must be able to connect the corresponding cells on the borders of the different sub-regions.
1.2
Coupling CA with different state sets
In order to couple two CA with different state sets, the programmer of the CA must specify how the state of a cell in one CA can be converted into the state of the other C A. In some cases this can lead to a loss of information, but this should be under the control of the designer of the CA. All the facilities required for case 1 are also required here.
1.3
Coupling CA with different spatial or temporal resolution
In some simulations, certain regions need to be resolved much better than others. This can be achieved by using a fine grid in some regions and a coarse grid elsewhere. In order to combine CA with different resolutions, one must provide interpolation routines and ensure the correct synchronization of the different CA. The interpolation is a more difficult problem for cellular automata than for finite difference or finite element methods, since CA are discrete in space and state, therefore averaging different cells is not directly possible.
1.4
Composing CA
The preceding cases all describe the coupling of different CA at a common border. A different case is the coupling over the whole space, which can also be viewed as the composition of different CA. In order for this kind of composition to work, the programmer must again provide methods for the translation of one state into the other.
68
1.5
Coupling CA with different (possibly non - CA) simulations
CA can also be coupled to other simulations that need not be cellular automata. An example could be a solver for an ordinary differential equation, which provides the boundary conditions for a CA. 2
Implementation
The JCASim system is completely implemented in Java. Therefore the coupling extension3 is also implemented in Java. In order to enable the parallelization in a distributed environment, each CA can run on a separate virtual machine (VM), and communication between them is handled via remote method invocation (RMI). An alternative would have been to use CORBA. Here we don't need the language independence of CORBA and prefer the RMI approach, as it is in integrated into the language standard. Furthermore, later migration is expected to be possible with automated tools. Another possibility would be to restrict parallelism to shared memory machines, and simply use multithreading. This approach will be added later for improved performance. 2.1
Structure of JCASim
Figure 1 shows the central classes of the JCASim system. A CellularAutomaton has a L a t t i c e of a certain dimension and type. The L a t t i c e contains Cells, and each Cell contains the S t a t e . This S t a t e is the class written by the user and contains all the information specific to a special CA, apart from configuration options specified in the simulation system. The user writes a subclass of S t a t e which contains variables to be held in each cell and contains the method for changing the state in one time step. Access to the neighbors is through a method of the Cell, getNeighbors(). If a cell accesses a neighbor outside the simulation region, a special object of type BoundaryHandler is called to handle this request. There are three basic boundary handlers in JCASim: PeriodicBoundaryHandler, Ref lectiveBoundaryHandler, and ConstantBoundaryHandler. They return the appropriate state of the lattice at the periodic or reflected position or a special constant state. For coupled CA, a different BoundaryHandler is introduced, which is able to get a cell from the remote CA. This RemoteBoundaryHandler cooperates with a BoundaryServer on the remote site. The BoundaryHandler and the BoundaryServer also need to ensure the proper synchronization of the CA. Each CA can only start the next iteration if the cell data are not required by
69 Interface
emstm.L m ttfcm
4-HEXAGOWAlTnt
-lx:lnt -ry:tnt -l*:int #boundaryHandJer:Bour>daryHarii
+setStMt*CI*si(it*tttCI*SM;Cl*SMtvot4 +setLMttfcet>*fimtto*0ihlatttc*DeffaHlmnl +9ttStafCfass0:ClMSS +^aUattheDtfMtH>aO.-lMtt/<:eOmffmftfoa +teUMttkt0±attke +r***tO,voM
fraiatftvold *b*ckup0:vi>id * t n r t a ittonttv oid «oSt«pO:voW fdoNftapsOwofd *9ttSt*te6t*rt£St*te *rttSt*re(k:/mry.*rtj:Sr*te *»*tSt*t*(xfat,r:tirt^JM).-$tmte
• T K IANGLE AHCU:lnt-l
+d65tmp1htoM +b*ckapO&otd +tr*asltkmQ.-voM +fttG*nerMtten0.iitt +fetStMttnit;fittly.1itttz.-int&tattt
ca,a1m.L«nlc«3D
+9etSute(x':intly :'lBt ,z :lnt):Sttt« +gatOMStata(x:lnt,r:tRt^:int>:St
g>aima.attlc«20
+gat$tataO(:lat,y:lnt}:Siata +««tOWStatatx:lm,Y:lat)3iat(
QcaalmJ-attlcalD x:kni V:bit z:bit
+ c.H.:C.tlfl
caslm.CALocal -rcadyForTrarti It Ion iboolcatt
+CALo»IO +CALocaf
i a t t k c D t f imt ion&attfce Definition iUteCliis:Clait lattice l a t t i c e flaneratlon:lnt
caalmXall «backupS t i t c S t a t c -cur rent Stat e:State #f*ttlce:Uttrca
#C«IIOiLattlca,st«ta:Clasi) #backup0:vold ftrarulttonfrvold gatOUStata&State djetStataftStat* getNctgMwriDrStareD oetNaighborR«latlvcO(:lnt,y:lnt l z:lnt):St
copyfr$tifeJR/*i mutsttlaitOarekl aatColarOiColor +toStrlng05trlng
merits Ck
caalm.Lamc«D«nnltlBf
•E
cmmtimJm**mttdmryHaiHUmr 0 _ #tattlca:L*ttlca
ca.almXawatatKKoundaryHaiid.il
+BoundaiYHandUr(my DKn:lnt,po» ltrtfe:baolei iXTat +0iitOuuMeStMte0c:/ittlyJatfx:fnt&tat*
+ConatKKlloundafvH*ndlartrnyOini:lnt +aatOuttldaStata(x:Mtt,y:int f z:lnt^tat
dtmantlon:l(tt a»liw.Mflactlv«»PundaryH*ndl vcoupliiaa |«ckaga cacomb nb I1 '
IV |_ ** *•-• - ^ .
RamonSoundairyHancih ramoteSafva l o u r d a r y i a r v t r
TJ
+ M f la«tya»o»ndary Handla r(my Dlm:l 4-gctOutildeStatc(x:lnt,y:lnt,z:lnt):Sta
• alaw.Par'kofllcioutidaryHand'li
\
\
+r>arlodlcloundaryHandlar(myD!m:lnt
\ tpllfoimdaryHandUr
bavndaryHandlaM Vector ipflrpoat at
getloundarySrataahlerth hit) Lattice raadyFortackiipttltta dMble)-va|d readyFoiTranflc ondliiM daub a h r a M
tSpittCauifdaryffandterOnyD m-tnt&mttht oaohtafi Lattice] +connactUp1rtfolnt kitJtl-taundaryHaadlar ti2 lojuidaryHandler) , vo raaayForlaeKupRlnia dounfla>-vold ready F a r T n n i tkwftlme double) void +aatOutf IdaStatiGr. kit y lnt,z I n t l d t a t e
Figure 1. Basic class structure of the JCASim package.
70 CilutorAutomaton
-1 I
I ^orUII slates on boundary)
^ I
I
I
•
I I I I
Figure 2. Sequence diagram for the transfer of cells from one CA to another.
another coupled CA. The synchronicity requirement is somewhat relaxed by the fact that each CA keeps a copy of the old state for access to the neighbors during the state transition. 2.2
Remote Data Transfer
The transfer of data from one CA to the other goes through a number of stagesas shown in Figure 2. Within one time step, an access to an outside cell of CA 1 leads to a call to g e t O u t s i d e S t a t e of the appropriate RemoteBoundaryHandler. The RemoteBoundaryHandler (RBH) checks whether the requested cell state is available in the cache. If so, it is returned from the cache (and no remote communication is necessary). If not, a remote method invocation to the corresponding BoundaryServer on the remote machine is initiated. The RBH requests not just one cell, but a whole slice in the assumption that most other boundary cells will also be requested within the same time step. The request is serialized (converted into a byte stream) and sent over the network to the remote BoundaryServer, where the request is unpacked. The BoundaryServer then creates a temporary object of class SimpleLattice which contains just those cells of the lattice requested by the RBH. This object is again serialized and sent back to the requesting RBH, which then unpacks the results and stores it in its cache and can finally deliver the requested cell state. Most of the time required for the data exchange is spent in the serialization and deserialization of the SimpleLattice transferring cell states. This could be improved by using the E x t e r n a l i z a b l e interface instead of the built-in S e r i a l i z a b l e mechanism. The E x t e r n a l i z a b l e interface simply replaced the automatic conversion of objects to byte-streams by a programmercontrolled conversion. The disadvantage is that the user must provide code
7?
for this operation, while serialization is automated using the Java reflection mechanism. Even more efficient is the case where all processes run on the same multiprocesor. In this case, a simple shared-memory access is sufficient. The current implementation does not yet detect this situation. 2.3
Split boundaries, interpolation, and state set conversion
The case where two CA meet, but do not share a complete border (see Figure 3, left) is handled by another specialized BoundaryHandler, a SplitBoundaryHandler, which in turn uses RemoteBoundaryHandlers to communicate with the remote CA. The case of different spatial resolutions is also handled by a specialized InterpolatingBoundaryHandler. In order to implement coupling case 2 (different state sets), the RemoteBoundaryHandler compares the classes of the local and the remote cells states. If they are not equal, and the local state implements the StateConversion interface, then the remote boundary handler creates new states of the local class and asks them to import the data from the remote cells. 2.4
Synchronization
Two coupled CA must be synchronized so that each request for states from the other CA can return states at the correct time step. If one CA advances too fast, the states requested by the other CA are not available any more (states from later times are stored in their place). On the other hand, the number of requests is not predetermined, since a CA might during some time steps not request any outside neighboring cells at all. Therefore separate synchronization calls are inserted, where each CA announces to its RemoteBoundaryHandlers the intention to proceed to the next time step. The RBH can delay this progress until the corresponding RemoteServer agrees that the next time step can be executed. 3
Initialization
The different CA, boundary handlers, and boundary servers are initially set up by a central configurator. This configurator has a graphical user interface for setting up the simulation In the configurator it is possible to add a new cellular automaton to the simulation. The boundaries can be connected automatically
72
Figure 3. Left: Greenberg-Hastings CA testing general communication and boundary splitting. Right: Diffusion with two different approaches
or explicitly. The configuration of a simulation can be saved in XML-format, which can then also be edited with a text-editor or with a automated tools to create larger simulations. 4
Simulation
The simulation of coupled cellular automata is controlled by a user interface which can display all of the CA together. The user specifies a time until which the simulation should be run. All the coupled CA receive this information and execute the necessary transition steps to reach this given time. The synchronization is strictly local, i.e., neighboring regions control each other. When the predetermined target time is reached, the controller interface is notified. The GUI can then copy the cell states for display to the user. 4-1
Test-cases
As a first test (Figure 3, left), we coupled three CA of the same type to verify the correct exchange at the (split) border. We selected a Greenberg-Hastings CA in which any error in the coupling can be easily spotted. As an example for coupling different types of CA, we show two differ-
73 t t ••]
so;
(a). y=200/
y=100/*
600 400
y=10 -32.0323
20
50
(c)
It"
CA 2 CA 3 CA 1
v
Figure 4. Measurement results: (a) Time in milliseconds per time step to simulate coupled CA with different sizes x and y (with 4 processors), (b) Speedup for different sizes ( where the problem size scales tith the number of processors p). (c) Configuration used.
e n t c e l l u l a r a u t o m a t a s i m u l a t i n g diffusion.
One a u t o m a t o n uses the lattice
gas approach, the other uses a finite difference averaging with probabilistic rounding 4 . Figure 3(right) shows the simulation after 300 time steps. 4-2 Speed The speed of the simulation is influenced by a number of factors. To obtain a reasonable simulation speed in this Java based system, a number of recommendations should be followed2. Most importantly, no object should be created in the inner simulation loop. For CA requiring random numbers, an efficient random number generator should be used. Besides these recommendations, recent Java virtual machine implementations come with just-in-time compilers which are quite efficient. The coupling interface introduces an additional overhead. This overhead comes from the serialization, which is used to package the border cells, and the remote method invocation, which includes the network communication latency. The current implementation also does not heed the above recommendation to avoid all object creations (Java serialization creates new objects when de-serializing). To measure the overhead, we performed a number of simulations on a Linux-cluster (Pentium 266). The simulations were performed using a typical CA (with diffusion and using random numbers extensively) and different sizes of the CA. We report the simulation times for four processes and indicate the size of the data chunk for each process. In this simulation the
74
processes are arranged in a linear array and the results are shown in Figure 4. The internal calculation of the transition function takes 0.032ms per cell. The border exchange introduces a delay of 85ms plus 1.02ms per border cell. The constant delay is approximately equal to the variable delay for 82 cells, and the total delay for exchanging 82 cells is approximately equal to the time required to update 82 x 64 cells. Therefore this approach is only useful for parallelization of large grids. The speedup (with growing problem size) is nearly linear for two up to 10 processors, since there is no sequential overhead in the synchronization. The border exchange can be improved using the E x t e r n a l i z a b l e interface of Java, which reduces the communication time per cell by 25%. The constant overhead could probably be improved by careful tuning of the synchronization messages. 5
Conclusion
We have demonstrated an extension to the cellular automata simulation system JCASim, which allows the coupling of different CA. The CA can have different state sets and different spatial or temporal resolutions. The coupling uses the Java RMI mechanism and is mainly geared towards flexibility as opposed to speed. Parallelization can be performed using this mechanism, but is efficient only for large enough simulations. The JCASim sytem is available at h t t p : / / w w w . j c a s i m . d e / . References 1. Uwe Freiwald. Eine Java - Simulationsumgebung fur Zellularautomaten. Diplomarbeit, Inst, of Scientific Computing, Techn. University Braunschweig, 1999. 2. Jorg R. Weimar and Uwe Freiwald. JCASim - a Java system for simulating cellular automata. In S. Bandini and T. Worsch, editors, Theoretical and Practical Issues on Cellular Automata (ACRI 2000), pages 47-54, London, 2000. Springer-Verlag. 3. Marcus Briesen. Eine verteilte Simulationsumgebung fur gekoppelte Zellularautomaten in Java. Diplomarbeit, Inst, of Scientific Computing, Techn. University Braunschweig, 1999. 4. Jorg R. Weimar. Simulating reaction-diffusion cellular automata with JCASim. In T. Sonar, editor, Discrete Modelling and Discrete Algorithms in Continuum Mechanics, page (to be published). Logos-Verlag, Berlin, 2001.
HUMAN EXPOSURE IN THE NEAR-FIELD OF A RADIOBASE-STATION ANTENNA: A NUMERICAL SOLUTION USING MASSIVELY PARALLEL SYSTEMS LUCA CATARINUCCI AND PAOLO PALAZZARI E.N.E.A. - HPCN Group, Casaccia, Roma, Italy E-mail: palazzari@casaccia. enea. it LUCIANO TARRICONE D.I.E.I., Via G. Duranti 93, 06125 Perugia Italy E-mail: tarricone@diei. unipg. it Abstract The rigorous characterization of the behaviour of a radiobase antenna for wireless communication systems is a hot topic both for antenna or communication system design and for radioprotection-hazard reasons. Such a characterization deserves a numerical solution, and the use of a Finite-Difference Time-Domain (FD-TD) approach is one of the most attractive candidates. It has strong memory and CPU-time requirements, and parallel computing is the suitable way to tackle this problem. In this work we discuss a parallel implementation of the FD-TD code, present the findings achieved on the APE/Quadrics SIMD massively parallel systems, and discuss results related to the human exposure to the near-field of real radiobase antennas. Results clearly demonstrate that massively parallel processing is a viable approach to solve electromagnetic problems, allowing the simulation of radiating devices which could not be modeled through conventional computing systems: a detailed numerical human phantom is used to solve the human-antenna interaction problem, and the absorbed fields inside it are estimated. The achieved solution accuracy paves the way to a rigorous evaluation of the radiofrequency safety standards in a relevant class of occupational cases.
1
Introduction
The astonishing boom of wireless communication services has focused the attention of researchers on several different aspects: theoretical radiowave propagation problems, both in open and in indoor environments [1-3], experimental set-ups for electromagnetic (EM) dosimetry [4], and source optimum dislocation [5]. Another high-impact item has been represented by the assessment of possible hazards caused by the exposure to EM fields. In this widely-spread class of research, an interesting paper in the literature is focused on an occupational problem, i.e. the near-field exposure of workers in the nearby of radiobase antennas (RBA) [6]. This latter paper is a challenging attempt to experimentally quantify the amount of EM power induced inside a homogeneous human phantom. The approach therein proposed is experimental, and no rigorous theoretical or numerical solutions are
75
76
proposed in the literature for the same problem. This is basically due to the huge numerical effort required to rigorously characterize the near-field behavior of standard RBA. In fact, in the near-field case, the plane-wave approximation does not hold, and the electric and magnetic fields must both be evaluated. Moreover, the rigorous definition of the threshold distance for the far-field zone, is itself a discussed issue. Taking into account the classical distance giving the lower limit for the far field zone [7] 2D 2 O,. — (!) where D is the antenna's maximum geometrical dimension and X is the signal wavelength, we often have to deal with near-field distances in the order of 10-20 meters (we consider a typical case for RBA, such as the array of electric dipoles). The huge dimension of the near-field domain, and the need of accurate full-wave solvers for a rigorous EM characterization, turn the problem into a strong numerical effort. Therefore parallel computing, and in particular Massively Parallel Processing, seems to be the best way to solve the challenging problem of characterizing the near field behavior of RBA. Moreover, as immediately apparent when looking at the wide class of applications reported in [1-6], the development of such a parallel solution can be of great usefulness in a large number of different problems. In this work we recall in Section 2 the Finite Difference in the Time Domain (FD-TD) integration scheme used to solve Maxwell's equations [8-10], and elected as the reference numerical method because of its versatility. In Section 3 a rationale for the choice of the parallel architecture is given. In Section 4 we describe the FDTD implementation on APE/Quadrics platform. A quick discussion of a numerical phantom used to solve the human-antenna interaction problem is then proposed in Section 5, and results are given in Section 6, describing the human-antenna interaction in real cases. Finally some conclusions are drawn. 2
The Finite Difference in the Time Domain (FD-TD) integration scheme
The Finite-Difference Time-Domain (FD-TD) method [8] is one of the most used approaches to solve Maxwell's partial differential equations (PDEs). The Yee's FDTD algorithm transforms the time-dependent Maxwell's curl equations into a set of finite-difference relations which have the following properties [9,10]: a) The space and time derivatives are approximated by using the central discretization, therefore resulting in second-order accurate expressions. b) The location of the electrical (E) and magnetic (H) fields in the Yee grid implicitly enforce the Maxwell's divergence equations.
77
c)
The time-stepping process is folly explicit and the algorithm is nondissipative (numerical wave modes propagating in the mesh do not decay because of nonphysical effects due to the time-stepping algorithm). The FD-TD algorithm is based on temporal and 3D spatial discretizations. It uses a leap frog integration scheme to solve Maxwell's equations: at time step t=n+l/2 it computes, at each grid point (ij,k), the value of the H1^"2 components as a function of the previous value H1*"1 in the same point plus a function of the values of the E components at time t=n in the grid points belonging to the neighborhood of (ij,k); in a similar way the value of the E components at time step t=n+l is computed, at each grid point (ij,k), as a function of the same component at previous time step (En) plus a function of the H components at time r=n+l/2 in the grid points belonging to die neighborhood of (ijjc). The exact expressions for the computation are similar to the following, which is used to compute the Hx component:
H
*G=D.\uyH.Zj+D>\uS
( 17 I" ^y Uj,k+l I 2
r i» y \i,j,k-\ I 2
a
£
CI" V I" "\ ^ ' z \j,j+\ I 2,k ~-°z \i,j-\ I 2,k
Az
Ay
being Da\. . , and Df\. . , some constants which take into account the structure of the material in each grid point [10]. In order to deal with the finite dimensions of the domain on which the integration is performed, we used the 2 nd order Mur absorbing boundary conditions (ABC) [11]. ABC simulate the propagation of a wave incident on the boundaries of the integration domain and they try to generate no reflected waves (i.e. they simulate an infinite domain). We preferred to use the Mur instead of the PML ABC [12,13] because the computational burden is substantially smaller and they guarantee an accuracy appropriate for the addressed problem. In order to ensure numerical stability of FD-TD method, the temporal increment step At must satisfy the relation [10]: Ar<—,
I 1
sec (3) t
1
|
1
being M =
A
the spatial discretization step (k=x,y,z, Nt is the number of samples for
N
k
wavelength in the k direction). In order to avoid modal dispersion, the spatial discretization step must be chosen smaller than X/10, i.e. A^>10 must result ([9,10]). FD-TD simulation of a domain with size \LX xLyXLz\m3 requires a number of floating point operations
78
NFlop = 36xLx xLy xLz x being N the number of samples considered for each wavelength and X the wavelength of the electromagnetic component with highest frequency. Such a large number of floating point operations clearly demonstrate that the simulation of EM fields at high frequencies can obtain a significant improvement when implemented on parallel systems. 3
Choice of the Parallel Architecture
From the considerations presented at the end of previous section, we decided to use parallel systems to simulate, by means of the FD-TD technique, Radio Base antennas and their interaction with human phantoms. Once decided to adopt parallel processing, the target parallel architecture must be selected among the many available on the market. A complete analysis of such item is quite time-taking, and we report here just some hints. If we take into account electrical power consumption and H W volume of the systems, it is apparent that special purpose systems, like the ones of the APE/Quadrics series, offer better performance (for the same HW volume) than general purpose systems: in fact near all the silicon of the system is used to implement operations which are almost always active, thus allowing the achievement of high efficiency. Moreover, the FD-TD integration scheme belongs to a typical class of problems well supported by the APE/Quadrics massively parallel systems, and it can be efficiently programmed on them, with a consequently high performance-to-price ratio. In fact, the FD-TD algorithm has the following characteristics: 1. it is synchronous; 2. all the processors execute the same instructions on different domains; 3. it needs interprocessor communications executed in a synchronous way; 4. it has regular patterns for memory accesses, so deep memory hierarchies are not required. Points 1 and 3 show that the time spent in synchronization phases, required by MIMD systems, is an overhead not required by SIMD machines with synchronous communications. Point 2 shows that all the HW dedicated to manage different program flows in the processors is unnecessary, being sufficient, as in the SIMD systems, one centralized controller of program flow. Point 4 means that cache memory and the related policies are not needed for the efficient implementation of the FD-TD algorithm. Previous reasoning constitutes the rationale for using the APE/Quadrics SIMD systems to efficiently implement the FD-TD integration scheme.
79 4
FD-TD Parallel Implementation on the APE/Quadrics
In this section the sketch of the parallel implementation of the FD-TD algorithm on the APE/Quadrics machine is given. On a machine with n processors, the whole computation domain is divided into n sub-domains (with equal volume and shape); each sub-domain is assigned to a processor and adjacent sub-domains are assigned to adjacent processors (both the algorithm and the machine implementing it have 3D topology). The EM field components are contemporaneously updated in each processor through equation (2) and its companions (for the remaining scalar Ex,yz and Hj,,z components). When the computation updates a field component on the border of me domain, some values belonging to the border of the adjacent domain are required: in order to avoid communications during the computations, each sub-domain is surrounded by the border cells of the other domain. These border values are communicated after the updating phase. The scheme of the parallel algorithm is given in the following: FD-TD parallel algorithm begin FD-TD algorithm Choose a spatial discretization of the domain (Ax, Ay, Az); if the domain has dimensions (L* x Ly x Lz), the grid has Nj x Nj x N k points, being 1,J,K
^ x
.y> z
Determine the time step At (eq. 3); Partition the whole rectangular domain D=[Ni x Nj x N J into P= P; x Pj x Pk rectangular subdomains D'= [N'i x N'j x N ' J , being Pi( Pj, and P k the number of processors along dimension ij and k, and N. N; Nk N; = — - , N : = and N ^ = the dimension (expressed as number of "i
"j
"k
grid points) of the generic subdomain D'. for^OjKT^t^+At) in all the processors do compute the new values of H communicate the H values on the boundary of each subdomain to its neighbor compute the new values of E enddo in all put the correct value in the feed point in the processors containing the source;
80
in the boundary processors do compute the absorbing boundary conditions; in all the processors communicate die E values on the boundary of each subdomain to its neighbor; endfor end FD-TD algorithm. When implementing me algorithm, particular care has been put in the vectorization of memory accesses (to reduce the severe memory start-up penalty), the unrolling of critical loops (to minimize pipeline stalls and die waiting for the consequent initialization latency), the using of the fast internal multiport register file.
5
The Numerical Phantom
A relevant role in the accurate solution of the addressed problem is played by die numerical technique utilized to represent die electromagnetic properties of die exposed human subject. The use of an appropriate numerical phantom, in feet, is the attractive issue to overcome die real problem of several proposed experimental techniques, using homogenous representations of die human body, with a consequent approximation error not adequate to die goals of a radio-protection analysis. The story of die development of accurate numerical phantoms is rich and long, and we address to die specialized literature me interested reader [14 and references tiierein]. Here we recall diat a generical numerical phantom is an organized archive to build up a detailed millimeter-resolution human model so diat die dielectric and conductive properties are fixed for each working frequency in each part of the sample body. In mis work we refer to one of die most appreciated phantoms, die one proposed by die Visible Human Project at Yale University, whose development is supported by [15]. It is also important referring to [16] for a sensitivity analysis of die predicted values witii respect to frequency and voxel size. 6
Results
Fig. 1 report results for a human phantom exposed to a real radiobase antenna (Katiirein 730678). The distance between die operator and die antenna is around 60 cm. The domain is partitioned into 320x320x256 cubic cells, witii a 4 mm edge. The operating frequency is 902 MHz, for die well known GSM cellular system. At this frequency, die wavelength in vacuo is 33.2 cm. As die maximum relative dielectric constant in the phantom is around 60, die use of a 4mm cubic edge is in accordance witii a standard convergence criterion to be used witii FDTD codes, requiring a
81
minimum ratio of 10 between the wavelength and the cell's edge. Nonetheless, it is quite obvious that an adaptive meshing is nearly mandatory, and is currently under development
50
100
150
200
250
50
100
150
200
250
Fig la,b: Human phantom exposure to a real radiobase antenna in two different configurations. Red colour is for high-intensity E fields, blue is for low amplitudes.
As for computing times and performance, for the sake of brevity, we omit here details: an efficiency TJ=—2-- — = 6.68E-01 is achieved for a 26214400 cell tpar
P
domain, thus proving that the FD-TD integration scheme is very well suited to be implemented on massively parallel systems. The accuracy in the solution of the problem allows the correct evaluation of E and H fields wherevere inside the human phantom, as depicted in Fig. la,b. It is just worth noting here that the maximum observed percentage of absorbed E field inside the phantom is around 40 % in the nearby of the chin. 7
Conclusions
In this paper we have proposed an FDTD implementation on massively parallel systems for the evaluation of radiofrequency hazards in humans exposed to the nearfield of radiobase station antennas. The approach is proved to be appropriate, and the FDTD strategy extremely amenable to such a parallel implementation. E and H fields are accurately evaluated inside a numerical phantom, thus solving in a
82 rigorous and efficient fashion a relevant problem in a wide class of real industrial applications.
References 1 2 3
4
5
6 7 8
9
10 11
12 13 14 15 16
G. Wolfle, R. Hoppe, F. M. Landstorfer: 'Radio network planning with ray-optical propagation models for urban, indoor and hybrid scenarios', Proc. of Wireless 99, 1999. G. Wolfle, F. M. Landstorfer: 'Prediction of the field strength inside buildings with empirical, nerual and ray-optical prediction models', COST 259 TD 98-008. R. Hoppe, G. Wolfle, F. M. Landstorfer: 'Fast 3D Ray-tracing for the planning of microcells by intelligent preprocessing of the database', Proc. of COST 259 Workshop, 1998. G. Wolfle, A. J. Rohatscheck, H. Korner, F. M. Landstorfer: 'High resolution measurement equipment for the determination of channel impulse responses for indoor mobile communications', Proc. of PIERS 98,1998. J. Zimmermann, R. Hons, H. Mulhenbein: "The antenna placement problem for mobile radionetworks: an evolutionary approach', Proc. 8th Conf. Tel. Systems, pp. 358-364, 2000. A. Bahr, D. Manteuffel, D. Heberling: 'Occupational safety in the near field of a GSM base station', Proc. of AP2000, Davos, April 2000, 3A6.7. J.D. Kraus: 'Antennas'. Mc Graw Hill - 1988. K. S. Yee: 'Numerical solution of initial boundary value problems involving Maxwell's equations in isotropic media', IEEE Transactions on Antennas and Propagation, AP-14, 4, pp. 302-307, 1996. A. Taflove and M.E. Brodwin: 'Numerical solution of steady-state electromagnetic scattering problems using the time-dependent Maxwell's equations', IEEE Transactions on Microwave Theory and Techniques, MTT-23, 8, pp. 623-630,1975. A. Taflove: 'Computational Electrodynamics: The Finite-Difference Time-Domain Method', Norwood, MA, Artech House, 1995. G.Mur: 'Absorbing boundary conditions for the Finite-Difference approximation of the Time-Domain Electromagnetic-Field equations', IEEE Transactions on Electromagnetic Compability, vol. EMC-23,4, pp. 377-382, 1981. J.-P. Berenger: 'A perfectly matched layer for the absorption of electromagnetic waves', J. Computational Physics, vol. 114, 1994. J.-P. Berenger: 'A perfectly matched layer for the FD-TD solution of wave-structure interaction problems', IEEE Antennas Propag. Symp., vol. 44, n.l, 1996. C. Gabriel, S. Gabriel, E. Corthout, "The dielectric properties of biological tissues: I Literature Survey', Phys. Med. Biol., vol. 41, pp.2231-2249, 1996. C. Gabriel, 'Compilation of the Dielectric Properties of Body Tissues at RF and Microwave Frequancies', Brooks Air Force Technical Report AL/OE-TR-1996-0037. P. A. Mason et al., 'Effects of Frequency, Permittivity and Voxel Size on Predicted SAR values in biological tissues during EMF Exposure', IEEE Transactions on Microwave Theory and Techniques, MTT-48, 11, pp. 2050-2061, 2000.
Tranquillity Mapping Using a Network of Heterogeneous PC Andrea Clematis, Monica De Martino, Giulia Alessio Istituto per la Matematica Applicata - Consiglio Nazionale delle Ricerche Via De Marini 6, 16149 Genova, Italy Sara Bird, Susanna Feltri Amministrazione Provinciate di Savona ViaSormano 6, 17100 Savona, Italy
The use of parallel processing to speed-up Geographic Information System (GIS) algorithms and applications has been widely considered and documented. In this paper we present our experience and early results about the use of parallel computing to speed-up tranquillity mapping, a methodology which is finalised to provide support to the landscape assessment process, and which is of interest for public administrations. The parallelisation of tranquillity mapping algorithm is described. Since the final goal is to make it possible to use the parallel algorithm in a public administration, aspects related with the use of an heterogeneous network of Personal Computers are addressed.
1 Introduction The usefulness of parallel processing for Geographic Information System (GIS) applications is widely documented in several reports and research papers (Healey98). Experiences are available about the development of parallel algorithms which improve the performances of different costly GIS functions like raster to vector data conversion, Digital Terrain Model (DTM) and Triangulated Irregular Network (TIN) construction, Drainage Basins identification, morphological analysis and other. In this paper we are interested in the use of GIS for tranquillity mapping (Bell99), a process aimed to produce a thematic map, which shows the levels of disturbance or tranquillity of a region. This process may be considered as a part of landscape assessment (Cudlip99), and it is becoming of interest for public administration as well as for other subjects like tourist operators or estate agents. The reported experience is part of the PLAINS project (Prototype Landscape Assessment Information System), a research project sponsored by European Union as a Centre for Earth Observation (CEO) activity. The parallelisation of the tranquillity mapping process for an heterogeneous network of PC is described. 2 Tranquillity mapping Tranquillity mapping concerns the idea of mapping areas of landscape that could be regarded as tranquil: away from noise and visual intrusion. This concept was introduced and developed by the ASH Consulting Group in England in 1991, first
83
84
for a study for the Department of Transport, and then for the Countryside Commission and Council for the Protection of Rural England (Bell 99). If a scale or spectrum of tranquillity is drawn up, a "complete" tranquillity would correspond to the absence of unnatural noise and visual intrusion, while at the other end an absolute lack of tranquillity would exist where noise is greatest and/or visual intrusion is significant. A degree of tranquillity depends on the proportion and combination of effects between these two extremes. Moreover it is necessary to consider diat die terrain morphology may affect or improve tranquillity. Here a tranquillity mapping methodology is proposed which takes into account the effect of the presence of disturbance sources. The methodology is characterised by two main activities: 1. 2.
Input data identification and acquisition; Spatial and attribute data processing for tranquillity map generation.
Parallel computing will be used to speed-up the tranquillity map generation step. 2.1 Input data identification and acquisition In our approach the basic input data is the vector cartography of the investigated region. The region we have considered is part of Liguria in North- West of Italy, and it is characterised by a complex terrain morphology. For this reason we adopted a 1:5000 scale for the digital cartography. For other region with a more regular terrain morphology a smaller scale could be adopted. The other input data are disturbance values, represented by attributes attached to each geo-referenced object, which is a source of disturbance. In the present version of our Tranquillity mapping process we have considered factors concerning noise and visual intrusion generated by three categories of unnatural sources, which are road and railroad networks, industrial sites, and urban centres. Each object belonging to one of these categories must be contained in the spatial database as a single feature or as a set of features according wim the appropriate geometry: linear features for road and railroad networks and area features for industrial and urban centres. A disturbance level is attached to each single feature. The whole set of features of a category is identified as a theme. We have defined 8 disturbance levels: 1 is for highly negative influence and 8 is low influence. The definition of a tranquillity spectrum extends the effect of the disturbance level to the influenced area using an appropriate set of distance intervals to identify different areas ranging from very disturbed to very tranquil or remote. See (AlessioOl) for a table, which defines a tranquillity spectrum schema.
85
2.2 Spatial and attribute data processing for tranquillity map generation This activity may be split in two phases: • Calculus of tranquillity map for each theme or category of features • Generation of a final map as a synthesis of all the generated tranquillity maps. For each theme, tranquillity mapping process is characterised by the following steps: • Group features in distinct classes in accordance with their disturbance level; • For each class generate a buffer layer using the table of distance range; • Combine buffer layers generated by different classes of disturbance; • Map optimisation. Considering that we group features according with the disturbance levels, which are eighth, we generate a map with eight buffer layers. The buffer layers are then analysed and synthesised: buffers are combined to generate a unique map by using overlay analysis operation. The resulting map is characterised by a lot of polygons obtained by combining all buffers in such a way that the minimum tranquillity scale (corresponding to a maximum disturbance) of the combined buffers is associated to polygons. The tranquillity map is optimised, by merging adjacent areas with the same tranquillity scale value, in order to reduce the number of distinct polygons. The tranquillity maps derived from the different themes are finally merged together by overlay and a final optimisation is done. xsaaEssa^H^mmmmmmm^m^^^mmmmmmmmmmi^^^^m^m
Figure 1: A tranquillity map
86
Figure 1 shows the tranquillity map for the Letimbro drainage basin in Savona District ( Liguria Region, North West of Italy). 3 Parallelising TM: outline of a strategy for an heterogeneous environment The tranquillity mapping algorithm exhibits an high computational cost and its execution time on an unloaded workstation equipped with a professional GIS environment is of about 12 hours for an area of 400 Km2 characterised by a complex morphology and by a medium level density of disturbance features. Thus to speed-up the mapping algorithm will lead to important improvements especially for simulation purposes. 3.1 The Parallel Program Outline The Tranquillity Mapping algorithm permits to process in an independent way tasks, which may be obtained by partitioning the data domain using a logical approach (different themes) or a spatial approach (different sub-regions) or a mix of the two. To put together the results of the processing of two or more independent tasks requires to properly overlay maps, if the tasks correspond to different themes of the same sub-region, or join maps if tasks correspond to neighbour sub-regions and finally to execute a map optimisation. Note that the overlay operation is the same used in the sequential algorithm, while join does not present particular problems. In the present implementation, if a feature lies over a boundary between two sub-regions it is assigned to one of the two regions. This is sufficient to ensure the correctness of the parallel algorithm. Figure 2 provides an outline of the parallel program for tranquillity mapping. The parallel algorithm is based on master worker paradigm and it is divided in three distinct phases. In the first phase the master provides to data distribution and schedules a task for each worker. In the second phase each worker processes tasks and it notifies the master after each task completion. Results of computation are kept in local database of each worker during this phase. When all tasks have been completed the third phase, dedicated to overlay and join operation may start. These operations are executed as much as possible in parallel, trying also to minimise data transfer among worker. A tree like approach is used to overlay themes of each subregion, and to join maps of different sub-regions. The number of independent tasks progressively decreases during this phase of computation, and workers will leave computation following master instructions. 3.2 Load Balancing and data partitioning Load balancing is the key point to make the parallel algorithm efficient, and to obtain load balancing we have to consider the computational cost of each task,
87
Phase 1: data distribution and start up Master: { copy dataset to each worker local disk; schedule the first set of tasks; send TaskActivation message to each worker; }
Phase 2: Single task processing Master: while WorkToDo { receive TaskDone message; schedule NextTask; send TaskActivation message; } Worker: while WorkToDo { receive TaskActivation message; compute task; send TaskDone message; }
Phase 3: Maps overlay and join Master: { co-ordinate workers to properly overlay thematic maps for each drainage basin; co-ordinate workers to properly join drainage basin maps; accept and merge final dataset } Worker: { receive information from master about WhatToDo; select WhatToDo: Overlay: overlay themes and optimise map following master directives; Join: join and optimise regions following master directives; AcceptData: merge in your local database data provided by another worker; ProvideData: copy your local database to indicated worker location; GiveUp: exit computation }
Figure 2: Parallel tranquillity mapping which depends on the number of features contained in the task, and for tasks with a similar number of features on their density. In fact a higher density of features will lead to an higher fragmentation for the intermediate maps, thus making the optimisation phase more costly. Domain partitioning is primary obtained using the different themes. Since the different themes contain a different number of features we have to find a way to further subdivide more populated themes in appropriate
88
sub-themes. In the considered case this has been done by splitting the transportation infrastructure theme into four sub-themes, namely rail-roads, highways, primary roads and secondary roads. In mis way we obtained a limited number (six in the considered case) of independent tasks with a similar computational cost. To have a small number of tasks with the same computational cost ensures load balancing using few homogeneous resources. Actually we are interested in the use of heterogeneous resources. In diis case we may further divide sub-domain so that to generate a sufficient number of independent tasks with a finer granularity, which permits to exploit the self balancing properties of master - worker programs (Bacci°9, Schmidt94). The thematic sub-domains have been further divided using a spatial approach and individuating independent sub-regions. The sub-regions have been derived using the natural subdivision represented by drainage basins. In the studied region we have eight main drainage basins, which combined witii the six themes lead to a set of 48 tasks of sufficiently fine granularity to ensure an acceptable load balancing for the available set of heterogeneous nodes we have considered (see Section 4). Using a mix of thematic and spatial subdivision we obtain tasks with different computational cost. It is simple to predict the approximate relative computational cost of each task by considering the number of features and their density, this latter obtained using the area of the basin. In the present implementation we based scheduling mainly on a locality principle, and the master tries to assign the set of tasks corresponding to a sub-region to a single worker as much as possible, but still keeping load balancing. The master assigns tasks to processors considering the relative cost of sub-regions, the relative speed of processors, and a simple round robin strategy. This scheduling policy permits to reduce communication of large data set during the third phase of the algorithm, since the overlay operations may be executed using data available on each computing node. 4 Experimental results and conclusions We experienced the parallel tranquillity mapping algorithm on six personal computers running Windows NT and connected on a local area network, based on switched and mixed Ethernet / Fast-Ethernet. In this implementation the data warehouse is hold by an office like database, and the software is implemented using an object based proprietary library for GIS functions and PVM for communications. The PC are equipped with a Pentium III processor, with different clock frequency (from 350 to 700 Mhz), and 128 Mbyte of RAM. The measured network bandwith is around 1.7Mbyte/sec. The relative speeds of die computing nodes are 1, 1.3, 1.5, 2.1, 2.3, 3.0, and are calculated assuming the GIS workstation, used to run the sequential version of the algorithm, as the reference node with speed equal to 1. The relative speed has been measured executing the same task on the unloaded nodes.
89
The total available relative computing power is 11.2. The program has been tested by running it on the dataset of the region described in Section 2. We recall that to run the sequential code for this region on the GIS workstation takes about 12 hours. To discuss performance of the parallel program it is useful to consider its structure as depicted in Figure 2. Phase 1 represents an overhead with respect to the sequential algorithm. The main cost of this phase is due to dataset transfer to the local disk of each node. The input dataset for the considered region is around 80 Mbytes. The measured time for this phase is around 5 minutes. Phase 2 is dedicated to tasks processing, and its execution corresponds with a good approximation to themes processing for the sequential algorithm. This part of the sequential algorithm represents about the 70% of the total execution time, which is around 8 hours and 30 minutes. The measured time for this phase of the parallel program is about 61 minutes, which leads to a speed-up of 8.4, and a weighted efficiency around 75%, considering the available computational power. During this phase speed-up is mainly limited by not balanced computation, since the communication is limited to the exchange of few short messages between master and workers. Monitoring the load of each worker we note an over-scheduling of the node with relative speed equal to 2.3. Indeed this node computes two complete drainage basins (sub-regions) with a quite large number of features, and its relative load overwhelms its relative speed. Phase 3 is dedicated to map overlay and join. The overlay operation corresponds at a large extent to the map optimisation process in the sequential algorithm. The join operation, the dataset transfer and merging represent overheads of the parallel algorithm. The measured execution time for this phase is around 42 minutes. The whole parallel algorithm takes about 110 minutes, thus leading to a speed up around 6.5 and a weighted efficiency around 54%, again considering the available computational power, which is quite satisfactory as a first result. Up to now we do not have the opportunity of experimenting the algorithm on different data sets, because of lack of availability of suitable input data collection. However we made some simulation to better assess the performances of the algorithm. The simulation considers as input parameters the number of processing nodes, their relative speeds, the number of tasks and their costs, the communication cost, and the tree structure to build up the complete solution by properly combining single tasks results by overlay and join operations. The simulation is based on a simple analytical model of the computation derived from (Pasetto97, Clematis98). The simulation results show that the algorithm should maintain the observed performances in most cases, provided that data partitioning is properly supervised.
90 In future works we plan to modify the simple round robin strategy for sub-region assignment to processors with more sophisticated list scheduling heuristics. The reported experience shows that it is possible to use parallel computing for GIS applications developed using proprietary GIS, and to make it available in the public administration domain. The use object based technology seems to provide an effective possibility, while waiting for more advanced solutions which will be able to exploit the computing power provide by large, distributed, and truly heterogeneous GIS systems (Dowers 00 ). Acknowledgements Part of this work has been supported by the PLAINS project founded by European Union (Project ENV-4CT98-0753). The cartographic data used to produce Tranquillity Mapping are properly of Regione Liguria. The traffic data and other data used to derive disturbance levels have been provided by Savona Province. References (AlessioOl) G. Alessio, S. Bini, A. Clematis, M. De Martino, "Landscape characterisation", Tech. Rep. PLAINS-IMA-DE2, Marzo 2001 (Bacci99) Bacci B., Danelutto M., Pelagatti S., and Vanneschi M. "SklE: a heterogeneous environment for HPC applications", Parallel Computing., 25 (1999) 1827-1852. (Bell99) Bell S., "Tranquillity Mapping as an Aid to Forest Planning", Information Notes, March 1999, Forestry Commission, ( www.forestrv.gov.uk/publications/index.html) (Clematis98) A. Clematis, A. Corana, "Performance Analysis of SPMD algorithms on a network of workstation with virtual shared memory", in "Parallel Computing Fundamentals, Applications and New Directions", E.H. D'Hollander et al. Eds., 1998, pp.65 7-664 , North Holland (Cudlip99) Cudlip W. et al, "A new information system in support of landscape assessment: PLAINS", Computers, Environment and Urban Systems, 23 (1999) 459-467 (Dowers99) Dowers S., "Towards a framework for high-performance geocomputation: handling vector-topology within a distributed service environment", GeoComputation 99 (www.geovista.psu.edu/geocomD/geocomp99 ) (Healey98) Healey R. et al, "Parallel processing algorithms for GIS", Taylor and Francis, 1998 (Pasetto97) Pasetto D., Vanneschi M., "Machine independent analytical models for cost evaluation of template-based programs", Fifth Euromicro Workshop on Parallel and Distributed Processing, IEEE Computer Society, Jan. 1997, pp. 485-492 (Schmidt94) Schmidt B.K., Sunderam V.S., "Empirical Analysis of Overheads in cluster Environments", Concurrency Practice & Experience, 6(1994) 1-32
91 Parallel skeletons and computational grain in quantum reactive scattering calculations Stefano Crocchianti, Antonio Lagana, Leonardo Pacifici, Valentina Piermarini Department of Chemistry, University of Perugia, Via Elce di Sotto, 8, 06123 Perugia, Italy Abstract Two quantum reactive scattering computational approaches have been analyzed for parallelization. The parallel structuring of the two codes has been carried out using both the constructs based on directives of the MPI library and the skeletons defined by the SklE coordination language. 1
Introduction
In order to design efficient parallel computational procedures for dealing with complex chemical applications one has first to map the physical problem into a suitable mathematical model and then to transform the mathematical problem into an appropriate algorithmic structure. This process is not simple since there is not a unique correspondence relating physical, mathematical, and algorithmic parameters. A way of learning how these parameters are linked together is to single out of the existing codes the structures suitable for parallelization : . In this paper we examine two computational procedures taken from the problem solving environment SIMBEX 2 (an a priori simulation of molecular beam experiments). Typically, these procedures consist of a central computationally high demanding section. This central section can be schematized as one (or more than one) block(s) 3 having the structure given in Scheme 1: LOOP on a LOOP on b LOOP on z G = g(a,b, END the z loop
,z;a,/3,
w)
END the b loop END the a loop Scheme 1: The scheme of the typical central section of a reactive scattering code. where latin letters are used to indicate parameters for which the calculation of G is separated into uncoupled (albeit, eventually, nested) loops while greek
92
letters are used to indicate parameters for which the calculation of G is not decoupled into independent computations. 2
The computational procedures
The G block of the two quantum reactive scattering programs considered here is devoted to the integration of the Schrodinger equation for reactive processes 4 . The first of these programs is ABM. This program constructs the coupling matrix of the reactive scattering differential equations 5 obtained in quantum time independent approaches. The second program is TIDEP. This program integrates in time the quantum time dependent Schrodinger equation after collocating the system wavepacket on a proper multidimensional grid 6 . Both programs store calculated quantities that, owing to their large size, cannot be kept in memory. This allows the use of these data by the other programs in which the complex computational procedure is articulated. 2.1
ABM
At a given value of the total angular momentum quantum number J (in some approaches this is performed only at a given reference value of J ) , the G block of the ABM program 7 calculates for every point of a grid (the grid along the reaction coordinate that is divided into sectors) the eigenfunctions (given as a combination of some primitive functions) of a proper portion of the Hamiltonian. The program calculates also related eigenvalues and the overlap integrals between the eigenfunctions of adjacent sectors. The structure of ABM is given in Scheme 2: LOOP on the reaction coordinate grid-points LOOP on A Build the local primitive functions Evaluate local eigenfunctions and eigenvalues Store on disk the eigenvalues IF(not first grid-point) THEN Calculate overlaps with previous grid-point eigenfunctions Store on disk the overlap matrix END IF Calculate the coupling matrix END the A loop Store on disk the coupling matrix END the reaction coordinate grid-point loop Scheme 2: Scheme of the G block of the ABM program.
93 where the two nested loops run over the grid-points of the reaction coordinate and the 2 J + 1 projections A of the total angular momentum J on the z axis of a body fixed frame. The central section of the G block calculates the local eigenvalues and surface functions at each allowed value of the projection A. In the same section, overlaps between surface functions calculated at neighboor grid-points and the various contributions to the coupling matrix are calculated. This tightly couples each cycle of the loop on the grid-points to the previous one. Overlaps and eigenvalues are stored on disk for use by subsequent programs. 2.2
TIDEP
The G block of the time-dependent program TIDEP propagates in time t the real part of the wavepacket describing the system 6 . At the beginning (t = 0), for a given pair of values of J and A, the system wavefunction, expressed in a proper functional form using a suitable set of coordinates, and provided with a given amount of energy 6 , is collocated on a sufficiently fine grid of the spatial coordinates. Then the integration on time is performed by repeatedly applying, for about 104 -j- 105 times, the propagator which involves some time consuming matrix operations. After each time step propagation, the wavepacket is expanded, in terms of product states, at an analysis line placed in the product region. The coefficients of the expansion are stored on disk for use by subsequent programs. The structure of TIDEP is given in Scheme 3: LOOP on t LOOP on A Perform time step integration Perform the asymptotic analysis Store C(t) coefficients END loop on A END loop on t Scheme 3: Scheme of the G block of the TIDEP program.
where the two nested loops run over time and A. There are, obviously, some outer loops running on J and other initial conditions, like the vibrational (v) and the rotational (j) quantum numbers, which are fully decoupled. 3
Suitable parallelization schemes
The two programs allow the exploitation of the parallelism at levels external to the G block in an SPMD fashion. In the case of TIDEP this can be applied,
94
for example, to v and j quantum numbers. For both TIDEP and ABM it can also be applied at a lower level to J. This choice is, however, not exempt of problems since an increase of J makes the associated matrices very large and the computing time very long, leading to memory and load imbalance problems. This is the main reason for pushing the decomposition to a finer level inside the G block where a natural parallelization is not applicable. To this end different actions can be undertaken depending on the program considered and, sometimes, on the physics of the problem. 3.1
ABM
In ABM, the coarsest grain of G for which parallelization can be enforced is the loop over the reaction coordinate grid-points. Single grid-point eigenfunction calculations and overlap integral (with eigenfunctions of the previous grid-point) evaluations are, in fact, computational tasks small enough to be easily left with a single processor. At the same time, related memory requirements are limited enough to be handled by the node local memory. This makes it convenient, in certain cases, to group the calculations related to several grid-points and assign them to a single processor in a static way using a data parallel approach (for example, this is what has been found convenient when dealing with reduced dimensionality techniques 8 ). For procedures based on full dimensional approaches cpu and memory requirements are so large that each grid-point calculation has to be assigned to an individual processor and the distribution has to be performed by adopting a dynamic task farm model. However, to make this parallelization scheme viable, the evaluation of the overlap integrals considered has to be decoupled by repeating the calculation of the surface functions performed on the preceeding node 7 . In the actual implementation of ABM, the master process sends to each worker a grid-point for which the eigenfunctions are to be calculated. Once the calculation is performed, the eigenvectors are stored on disk for use by the processor dealing with the subsequent reaction coordinate grid-point. The next processor retrieves from disk the preceeding grid-point eigenvectors and evaluates the eigenfunctions at the quadrature points of the current sector. To prevent attempts to read not yet stored information, nodes are synchronized before reading from disk. This parallelization scheme has been implemented using MPI 9 and the reactive probabilities of the Li + HF reaction 10 have been calculated. Typical processor time measurements for runs performed on the Cray T3E of EPCC (Edinburgh, UK) using 32 (solid line) and 64 (dashed line) processors, are shown in Figure 1. The excellent performance of the model is confirmed by the good balance of the load assigned to the various processors whose values deviate only by a few
95 50 n 32 processors ••• 64 processors 40
30
20
10-
-5
0
_L 5
_EL 10
15
Percentua] deviation from average processor time
Figure 1: Frequency of percentual deviation from the average processor time.
percents from the average time. Calculated speedups S, shown in Figure 2, also indicate that the program scales quite well, since for up to 128 processors the speedup is never smaller than 70 % the ideal value. 3.2
TIDEP
Direct calculations cannot be used to decouple the t loop of TIDEP inside the G block. The solution is, in fact, recursively accumulated at each iteration. Nonetheless, a partial decoupling can be introduced by assuming that A is conserved within each sector during the one propagation. This confines the coupling between different A blocks of the solution matrix to that exerted by the Coriolis term of the Hamiltonian on adjacent blocks. Test calculations were performed on the ORIGIN 3800 at CINECA (Bologna, I) u using a task farm model for the 0( 1 D)+HC1 atom diatom reaction 12 on a gridsize of dimension 127 x 119 x 8. In this model node zero acts as a master. It performs preliminary calculations and distributes fixed J and A propagations to the workers. Average processor computing time for J and A pair calculations are given in Figure 3 as a function of J (each J calculation includes all the fixed A components of one parity). The excellent performance of the model is confirmed by the fact that the average processor computing time increases
96
50
0
50 Number of processors
100
Figure 2: Limit (dashed line) and measured speedup (solid line) for the ABM program
only of about 20 % in going from J = 0 to J = 1 3 . Speedups calculated on the ORIGIN 3800 are shown in Figure 4. They clearly indicate that the scalability of this program is very satisfactory for platforms with up to 16 processors. 4
Towards a SklE based approach
The detailed analysis of a computational application (as the one outlined above for ABM and TIDEP) requires a great deal of know-how not only about the physics background of the code itself but also about the computing platform adopted. An increasingly popular approach is to make use of parallelization environments like SklE (Skeleton-based Integrated Environment) 13 and to exploit the features of the related coordination language (SklE-CL). The use of SklE-CL makes the simplification of the parallel restructuring of the considered codes and the exploitation of alternative parallelization strategies extremely simple as shown below. 4.1
ABM
The parallelism on the initial conditions of ABM (outer loops) can be dealt by SklE as a data parallelism using a map structure (in which the same com-
97
5400
o
5200
o o
/ /
-S5000
p 4800
o
/ Jt
O
4600 s o 4400i 0
^
° '
' 5
'
1 10
' 15
J
Figure 3: Average processor computing time.
putation is applied to all the elements of a data array). In this case, each computation is totally independent of the others, and the main body of the program preserves its sequential structure. On the other hand, when the parallelization is pushed to the finer grain of the reaction coordinate grid-points, inside G, a farm construct needed to be considered in order to assign a virtual processor to each grid-point calculation in a dynamic way. Accordingly, a certain number of grid-point calculations are sent in a data stream and then each of them is assigned to a worker of the farm. As soon as the worker completes its task, it receives the next grid-point calculation as channeled by the stream (here, it would not be needed to recalculate the surface functions of the preceeding grid-point if a shared memory facility could be used). 12
TIDEP
Also TIDEP can use a map construct to parallelize the outer loops of v and j (at fixed value of J ) . A certain amount of work-load imbalance can arise when j and v become large and require a denser grid (and, therefore, a larger dimension of the matrices). When J increases, however, because of the large number of A projections, fixed J calculations do not fit into the node memory. To push the parallelization to a finer level it is appropriate to nest a farm construct (to
98
<«
10
^ 0
-i
1 5
1
1 10
i
I 15
i20
Number of processors
Figure 4: Limit (clashed line) and measured speedup (solid line) for the TIDEP program handle the correct stream of the total angular momentum quantum number values) with another farm, to distribute the related A calculations within each J value, provided that at each step a mechanism is developed to feed the Coriolis terms of the Hamiltonian with the adjacent fixed A blocks of matrices to be manipulated. 5
Conclusions
The parallelization of complex computational chemical applications can be pushed to a fine grain level by carefully handling parallelization libraries (such as MPI) and by introducing suitable decoupling schemes. In this paper, the decoupling has been exploited both by introducing (at least partially) direct calculations and by enforcing some dynamical constraints. This offers the advantage of leaving with the user the control of the efficiency of the parallelization while the code is being structured by inserting MPI calls. As an alternative, the use of coordination languages allowing the user to design a parallel model at an abstract level has been also experimented. This has made the investigation of different parallel models straightforward and the introduction of decoupling approximations simple. Our work has, however, singled out some critical features of SklE. Namely, the rigidity of the basic skeletons and
99
the lack of a shared memory structure. These problems will be taken care by ASSIST, a CL resulting from an evolution of SklE 1 4 . 6
Acknowledgements
Thanks are due to CINECA (Bologna, Italy), EPCC (contract ERB FMGE CT95 0051, Edinburgh), CESCA-CEPA (contract ERB FMGE CT95 0062, Barcelona) for computer time grants. We thank also CNR, ASI and MURST for funding the present research. References 1. A. Lagana, S. Crocchianti, A. Bolloni, V. Piermarini, R. Baraglia, R. Ferrini, D. Laforenza, Computational Granularity and Parallel Models to Scale up Reactive Scattering Calculations, Comp. Phys. Comm. 128, 295 (2000). 2. O. Gervasi, D. Cicoria, A. Lagana, R. Baraglia, Animazione e Calcolo Parallelo per lo Studio delle Reazioni Elementari, Pixel 10, 19 (1994); R. Baraglia, D. Laforenza, A. Lagana, A Web Based Metacomputing Problem-Solving Environment for Complex Chemical Applications, Lecture Notes in Computer Science, 1971, 111 (2000). 3. A. Lagana, Innovative Computing and Detailed Properties of Elementary Reactions Using Time Independent Aprroaches, Comp. Phys. Comm. 116, 1 (1999). 4. A. Lagana, A. Riganelli, Computational Reaction and Molecular Dynamics : from Simple Systems and Rigorous Methods to Large Systems and Approximate Methods, Lecture Notes in Chemistry 75, 1 (2000). 5. G.A. Parker, S. Crocchianti, M. Keil, Quantum Reactive Scattering for Three Particle Systems Using Hyper spherical Coordinates, Lecture Notes in Chemistry 75, 88 (2000). 6. G.G. Balint-Kurti, Time Dependent Quantum Approach to Chemical Reactivity, Lecture Notes in Chemistry 75, 74 (2000). 7. A. Lagana, S. Crocchianti, G. Ochoa de Aspuru, R. Gargano, G.A. Parker, Parallel Time-Independent Quantum Calculations of AtomDiatom Reactivity, Lecture Notes in Computer Science 1041, 361 (1995); A. Bolloni, A. Riganelli, S. Crocchianti, A. Lagana, Parallel Quantum Scattering Calculations Applied to the Dynamics of Elementary Reactions, Lecture Notes in Computer Science 1497, 331 (1998). 8. R. Baraglia, D. Laforenza, A. Lagana, Parallelization Strategies for a Reduced Dimensionality Calculation of Quantum Reactive Scattering Cross
100
9.
10.
11.
12.
13.
14.
Section on a Hypercube Machine, Lecture Notes in Computer Science 919, 554 (1995). Message Passing Interface Forum, Int. J. of Supercomputer Applications 8 ( 3 / 4 ) (1994); M. Smir, S. Otto, S. Huss-Ledermam, D. Walker, J. Dongarra, MPI: The complete reference (1996), MIT Press. A. Lagana, A. Bolloni, S. Crocchianti, Quantum Isotopic Effects and Reaction Mechanism: the Li + HF Reaction, Phys. Chem. Chem. Phys. 2, 535 (2000) V. Piermarini, L. Pacifici, S. Crocchianti, A. Lagana, G. D'Agosto, S. Tasso, Parallel Methods in Time Dependent Approaches to Reactive Scattering Calculations, Lecture Notes in Computer Science 2073, 567 (2001). V. Piermarini, G.G. Balint-Kurti, S. Gray, F. Gogtas, M.L. Hernandez, A. Lagana, Wavepacket Calculation of Cross Sections, Product State Distributions and Branching Ratios for the 0 ( : D ) + HC1 Reaction, J. Phys. Chem A 105 (24), 5743 (2001) B. Bacci, B. Cantalupo, P. Pesciullesi, R. Ravazzolo, A. Riaudo, M. Torquati, SklE-CL: User's Guide, Version 2.0; M. Vanneschi, Parallel Paradigms for Scientific Computing, Lecture Notes in Chemistry, 75, 168 (2000). P. Ciullo, M. Danelutto, L. Vaglini, M. Vanneschi, D. Guerri, M. Lettere, Progetto ASI-PQE2000, Workpackage 1, Ambiente ASSIST: modello di programmazione e linguaggio di coordinamento ASSIST-CL (versione 1.0) (2001)
PARALLEL IMAGE RECONSTRUCTION USING ENO INTERPOLATION
J.CZERWINSKA,W.E.NAGEL Center for High Performance Computing (ZHR), Dresden University of Technology, D-01062 Dresden, Germany E-mail: czerwinska,[email protected] The scope of this paper is a presentation of the method for image reconstruction. Most interpolation methods such as polynomial interpolation, with the special case of a linear interpolation, or splines, are sensitive to the sharp change in interpolated data. Essentially non-oscillatory reconstruction adapts to the function gradient in a way that deals efficiently with the sharp edges, which are quite common in average images built by discrete data. The drawback of the method is the relatively complex algorithm, which needs a lot of logical operation. Due to the parallelisation ENO, which improved the efficiency of the method, interpolation with its high accuracy shows itself to be as a reasonable alternative to other types of interpolation.
1
Introduction
A essentially non-oscillatory (ENO) method was originally proposed, to overcome problems of smoothing across the discontinuities in computational fluid dynamics by Harten 1 . At first it was used for the numerical solution of Hamilton-Jacobi equations by Oser et al. 2 . Later it was made more efficient by Shu and Osher 3 4 . The accuracy of the method and the way it deals with discontinuities can be used in the calculation of partial differential equations, as well as in a wide range of other problems. The basic concept can be moved to other fields of application, like image reconstruction by interpolation or subpixel interpolation 5 . Splines and polynomials are commonly used in computer vision and graphics applications for obtaining interpolation of discrete data. The behaviour of those methods is satisfying when the region of data is relatively smooth, but they are prone to error in the vicinity of the discontinuities. Due to this fact ENO interpolation seems to be a good alternative. The essentially non-oscillatory interpolation is based on the choice of the smoothest stencil possible, to obtain interpolation results. The selection is done by means of logical choices. Hence, the method is not as fast and efficient as the other commonly used reconstructions of the function on the discrete set of data. Parallelisation seems to be one of the ways to make it more attractive for general application.
101
-.k-
O ^ a
A
p™1 s c o n d unier neighbcwtrod itmid i n k r neighborhood forth order neighbourhood
Figure 1. Estimation of the order of the reconstruction
2
ENO interpolation
The basic concept of the ENO reconstruction used in this paper can be described as follows. Consider a domain fi with characteristic length of tessellation Ax. The expansion in the vicinity of node i has to fulfill the following conditions: • conserves the means • have a compact support ( only points in the neighbourhood are taken to the reconstruction in the vicinity of the node i ), where the size of neighbourhood depends on reconstruction accuracy (see Figure 1) • exactly reconstruct polynomials of degree < k equivalent to an accuracy of order (k+1) • ENO property- choice of the smoothest stencil used for the interpolation. For the structured grid, which was analysed in this paper, the 2D interpolation can be done by locally splitting into ID cases, which was proved to be valid for partial differential equations of a hyperbolic type by Maraquina 6 . ENO property is realised by searching for the smoothest stencil. The polynomial interpolation reconstructs data using the same type of data, for instance linear interpolation will behave better near jump of the function, but will not be accurate in other smooth regions. The parabola will interpolate data better in smooth regions, but discontinuities will cause overshoots on the interpolated
103
Figure 2. Discrete data of the Abgrall function on the coarse mesh
data. ENO property is basically an adaptation of the stencil (data used for interpolation) to the interpolated data set. 3
Results
The method, which has been described above, can be used in a variety of applications; in mathematics (function interpolation), computer science (image processing), etc. In this section, one particular case will be presented - the function proposed by Abgrall 7 8 «/* < ^cos(ny),f(x,y)
ifx > -cos{xy),f{x,y) where, u$
= «^f)(*.l/).
- u_^)(x,y)
+cos(2ny),
(0
(2)
104
0.2
0.4
0.6
0.8
1
Figure 3. Processed image - ENO reconstruction
ifr < --yU^x^y)
ifr > -,u^{x, o
if I
r
=
y) = 2r - 1 +
-rsin(-r2),
b
-rsin(3irr),
\< 2>u(x>y) =1 sin(2irr) \,
r = x +
tan(4>)y,
(3)
(4)
(5)
(6)
In Figure 2, the graph of the function in the domain [-1,1] is presented. The discrete data are obtained on mesh 51x51 points. The image reconstruction was done using the few following steps. From a starting level node, three steps of interpolation were carried out to obtain three new forward values of the function in new nodes. The distance between nodes shrinks to half of its
105
Figure 4. Processed image - linear interpolation ENO
original value in every step and the amount of the nodes is eight times the original. In Figure 3 the reconstruction for ENO interpolation for (408x408) nodes is presented. Figure 4 presents reconstruction for linear interpolation. It is easy to see the difference in the behaviuor of the solution for ENO and simple linear interpolation in the vicinity of the discontinuities. A satisfying sharpness of the discontinuities was not obtain for the latter. On the discontinuities ENO reconstruction accuracy is of the first order (same as linear interpolation), but in other region the accuracy only depends on the choice of the order of the approximation. The case presented in Figure 3 is second order of accuracy. The bad result for Figure 4 is also caused by the fact that error in the vicinity of the discontinuities has accumulated over steps of interpolations. For ENO reconstruction every step introduced only low accuracy points on the sharp discontinuity, while preserving accuracy of other regions.
106
4
Aspects of parallelisation
The parallelisation of the method was carried out using domain decomposition. A two-dimensional image was divided for non-overlapping subdomains (parallel stripes). The choice of such parallelisation method has provided no restriction on scalability. Non-overlapping subdomains were used to decrease memory usage needed to store data from the neighbourhood of the subdomain borders. This means that the values of the points are only computed once in a specific subdomain. The only values needed to compute interpolated points on the boundary are sent from neighbouring domains. After every level of reconstruction, data is collected by the main processor and if a new computation is needed, the data is resent. Due to the way the algorithm is constructed, the parallel efficiency is as presented in Figure 5. The time was estimated for the image - 166464 points low accuracy (second order method, called basic case in Figure 5) and high accuracy interpolation (forth order method, larger case needing a computational time,which is about 200 times longer). The gain in computational time is very drastic between computers with only one and those with several processors. With an increased number of processors, the advantage is not that large for a basic low accuracy case because the cost of the computation starts to be comparable with the time necessary to send and collect data. In the applications usually processed, images are not any larger than in the considered case. For that reason, it seems that, the considered parallel ENO interpolation algorithm can be used with quite good efficiency for computers with less than 10 processors. Of course in some cases the data files are much larger than the case considered above (for instance, when there is a need for an interpolation of the results of engineering computations several millions of points can be reached). In this case, then an increase in the number of processors will significantly increase the efficiency of the computations. For the pictures considered in this paper, time of calculation on 16 processors was very short, allowing to obtain enhanced picture almost immediately. Hence, there was no need to study effect of parallelisation on the higher number of processors. The algorithm of the method is very general, so there is no obstacle to calculate more complex cases with the larger number of processors. For the realisation of parallelisation, MPI was used and all calculations were done at the R10000, Onyx 3800.
107 11»
1
1
i
1
1
1
1
* larger case O basic case 0.9 0.8 0.7
-
o
-
0
-
§0.6 S
H;0.5
«
-
E
-
|0.4 0
0.3
-
# 0.2
"
o
0
* 0.1 0
* '
1
8
10
12
* 1
18
number of processors
Figure 5. Parallel efficiency of the image reconstruction algorithm. Basic case represents lower accuracy interpolation. Large case is higher order interpolation
5
Future work
There are basically two directions in which recent work can be continued. The first one is the adaptation of the existing algorithm to unstructured 2D cases. It may not have a straightforward application significance for image processing, but for large scale engineering applications it is a very desirable goal. The unstructured meshes are better at adapting to complex geometries, which are common for engineering problems, but it is also known that existing methods of interpolation and differentiation do not reach the required accuracy, which is easily obtained by structural grid calculation. Because of this the unstructured ENO interpolation will be interesting for future work .The second path could be 3D approximation. It is again important for large engineering computations and particularly interesting for computational fluid dynamics. In both cases the parallelisation issue is very important to make considered methods efficient in usage.
108
6
Conclusions
In this paper, the accurate and efficient method for interpolation was presented(ENO interpolation), which can be used for image reconstruction of a discrete set of data. Due to the parallelisation, the efficiency of the method increased and made it a good alternative to other widely used interpolation methods. The parallel algorithm seems to work especially well for large set of data,which it makes it even more promising for future applications. References 1. A. Harten. Eno schemes with subcell resolution. J. of Comp. Physics, 83:148-184, 1989. 2. C-W. Shu and S.Osher. Efficient implementation of essentially nonoscillatory shock-capturing schemes. J. of Comp. Physics, 77:439-471, 1988. 3. S. Osher and C.-W. Shu. High-order essentially non-oscillatory schemes for hamilton-jacobi equations. SI AM J. of Numer. Anal., 77:907-922, 1991. 4. B.B. Kimia K. Siddiqi and C.-W. Shu. Geometric shock-capturing eno schemes for subpixel interpolation, computation and curve evolution. Tech.Report LEMS, 142, Febuary 1995. 5. P. Haeberli and D. Voorhies. Image processing by linear interpolation and extrapolation. IRIS Universe Magazine, August 1994. 6. A. Maraquina and R. Donat. Capturing shock reflections: A nonlinear local characteristic approach. UCLA CAM Report, 31, April 1993. 7. R. Abgrall. On essentially non-oscillatory schemes on unstructured meshes: analysis and implementation. J. of Comp. Physics, 114:45-58, 1994. 8. R.Abgrall and A.Harten. Multiresolution representation in unstructured meshes. SIAM J. of Numer. Anal., 35:2128-2146, 1998.
TRAINING ON-LINE RADIAL BASIS FUNCTION NETWORKS ON A MIMD PARALLEL COMPUTER A. D'ACIERNO 8 ISA -CNR, via Roma 52 A/C, 83100 - Avellino, Italy This paper describes a parallel mapping scheme for the gradient-descent learning algorithm. The problem we are dealing with is Time-Series forecasting by means of GRBF Networks so that (i) the network has one neuron in the output layer but, at the same time, (ii) the memory of each processor can (typically) hold the whole training set. A mapping scheme recently proposed seems to be the optimal one (the only one?) for speeding the learning up. The used approach is synchronous, so that both SIMD and MIMD parallel computers can be used for actual implementations. Results on a MIMD parallel computer are shown and commented.
1
Introduction
The Time-Series forecasting problem (given a sequence x(l),x(2),....,x(N) find the continuation x(N+l),x(N+2),...) can be stated in terms of estimating an unknown function/such that: x(t) «Jc(f) = f(x(t - 1 ) , x(t -2),....,x(t-M)) (1) where M is the (unknown) memory of the (unknown) system. As it is well known, the problem of estimating an unknown function given some input-output pairs (regression problem) can be parametric and non-parametric. In parametric regression the form of the functional relationship between the dependent and independent variables is known (or guessed) but it contains parameters whose values have to be determined. In the non-parametric regression, instead, there is no a priori knowledge about the shape of the function to be estimated. Generalized Radial Basis Function (GRBF) neural networks are non-parametric models (that can be linear as well non-linear) able to approximate any reasonable continuous function mapping arbitrarily well and with the best approximation property [3]. As each neural network, the behaviour of a GRBF network depends on free parameters evaluated through a learning procedure; such a procedure tend to be very time-consuming and it is obvious to capitalize on the intrinsic parallelism of neural systems to speed the computation up. Learning algorithms typically involve only local computation but, on the other
hand, the output of each unit depends on the output of many other units so that most of the running time can easily be spent in communication rather than in actual
E-Mail: dacierno.a @isa.av. enr. it
109
110
computation. The motivation for this work was exactly to try to solve the mapping problem, with reference to a well known learning algorithm. 2
RBF: A Brief Introduction
The problem of estimating an unknown function / : 3iM —> SR given a set of data {(jr.,y(.)€ 9t" x5t|_, clearly is ill-posed, since it has an infinite number of solutions; to choose a particular solution, we need some a priori knowledge about the unknown function, typically assuming that the function is smooth (i.e. two similar outputs corresponds to two similar inputs). Since we look for a function that is, at the same time, close to the data and smooth, it is natural to choose as the solution of our regression problem the function /that minimizes the following functional:
n[f]=hf(xi)-yi)2+4Pf\\
(2)
where X (> 0) is the so called regularization factor, P is a constraint operator (usually a differential operator) and II - II is a norm (usually the L2 norm) on the function space off. The operator P clearly embodies the a priori knowledge about the solution and so depends on the problem to be solved. Minimization of functional H leads to the Euler-Lagrange equation that can be written as:
PPfi*) =T2>i:"/(*,))*(*-*,)
(3)
where P is the adjoint of the (differential) operator P and the right side comes from the functional derivative (with respect to/) of//. The solution of the partial differential equation 3 is the integral transformation of its right side with a kernel given by the Green's function of the differential operator PP, that is the function G satisfying: PPG(x;y) = S(x-y) whose solution, because of the delta functions, can be written as: f(x)=\fd(yi-f(xl))G(x;xi)
(4) (5)
By evaluating equation 5 at the W data points we easily obtain a set of equations for the coefficients c = ^' ~*^X'''A '
/A.
.
111
When the operator P is translationally invariant G will depend on the difference of its arguments ( G ( x ; x i ) = G ( x - x,)) and if P is also rotationally invariant G will be a radial function, i.e. G(x;xt) = G(|x - x,|) (e.g. a Gaussian function). The solution given by standard regularization theory is expensive in computational terms (OfN*)) and, what is of course worse, the probability of ill-conditioning the system is higher for larger and larger matrices. The generalized regularization theory approach is to expand the solution on a smaller basis, so deriving:
/Oc) = £c,.G(r,f,.)
(6)
where n«N and the f,'s are called "centres": equation 6 is topologically described in figure 1, where each Green's function is realized by means of a neuron; these networks are called Generalized Figure 1. A GRBF Network. Radial Basis Function (GRBF) networks. When the G,'s are Gaussian functions, it is of course: ( lv Output
Input
Lvytr
0(*) = £ w,. *f[exp' y-i
(*,-'„> >
(7)
Kjl
To evaluate the free parameters there are several possibilities. The simplest one selects centres by means, for example, of a clustering algorithm; then selects the o's according some heuristics and last, to evaluate the w's, solves a least square problem. Using such an approach a GRBF network behaves almost linearly. A quite different approach (that makes GRBF networks highly non-linear models) assumes that all the parameters can change during the training phase; in our experiments, for example, the centres are initialised using a competitive clustering algorithm and the learning strategy tries to minimize the error E defined as:
by using the well-known gradient-descent algorithm. This is an iterative algorithm that at each step (by considering the i-th training pattern and if Xy represents component 7 of example i) applies the following formulae: Aw
/ = ~1„ TT^" = IS, Ii e x P 2 dw,
*„f
\J 2a,
(9)
112
1 def
f^-^Vr
((to-',/)
2dalr
{
^ 2a„. )
alr
)M
n n
This procedure can of course work either on-line or in batch mode; in the on-line case parameters changes are used as they are evaluated while, in the batch mode, they are accumulated to evaluate global terms that are summed after all (or some) examples have been processed. 3. The Proposed Mapping Scheme The design of a neural predictor requires the choice of the parameter M to create the training set from the available historical sequence (this is one of the complications that make Time-Series forecasting a more difficult problem that straight classification). Analyse mis fact from the neural designer point of view; even if tricks and methods to guess the right M exist, it is a matter of fact that (theoretically) an over-sized M could be used, since not useful values should (or, maybe better, could) be neglected by the neural structure simply making a\ —» +<» for all i and for ally greater than some *. From a practical point of view, instead, the condition fffj -» +oo can be only approximated, so deriving that the selection of M is based mostly on trial and error (with some heuristics); moreover, the gradient descent algorithm gets stuck in the first local minima it meets, and thus it has to be usually re-started many times with different initial conditions. As concerns the right value of H it is a matter of fact that it should be chosen as low as possible just to save computing time since, as it should be clear from section 2, the performance does not decrease as H increases (provided that it remains « N). Summarizing, the neural designer has to perform a lot of experiments to optimally dimension and train the network, the learning phase thus being a process that has to be performed interactively; given the computational requirements of the problem at hand, the availability of fast implementations on parallel machines is mandatory to speeding the prototyping phase up. When the algorithm we are using to train our parameters is batch (or, with different words, when the batch algorithm can be used) the parallel implementation is straightforward, being of course possible to have a copy of the network on each processor together with some training patters. The problem clearly holds when we have to use the on-line version of the algorithm since, to speeding the computation up, we have to split the GRBF among processing nodes. This problem is, at least from an algorithmic point of view, almost equivalent to the parallel implementation
113
of the Back-Propagation of errors learning Algorithm (BPA) for feed-forward neural networks; this is not surprising, since (from a topological point of view) a GRBF can be viewed as a feed-forward neural network. Thus, even if (as far as the author knows) there are not (significant) papers dealing with gradient-descent algorithm for GRBF, we could try to implement it in parallel by using results suggested for the parallel BPA. Typically, the on-line BPA is seen as a sequence of matrix-vector products that are performed in parallel usually in a systolic fashion [5]. As it is well known, there are two systolic solution to the matrix-vector product problem; the first one maps, on each processor, a row of the matrix while, the second one, maps, on each processing element, a column of the matrix. A parallel implementation based on intra-layer parallelism can be obtained by using the by-row solution or, maybe better, by using a mixture of by-row and by-col solutions [6]. It has been also proposed in [4] a parallel implementation that, although still based on intra-layer parallelism, exploits the collective nature of the communication involved (see also [1]). The idea underlying many of these schemes cannot be used for the problem at hand for the following trivial reason. For a 3-layer feed-forward networks the maximum number P , ^ of processors that can be used equals the dimension of the smaller layer; in our problem the dimension of the output layer is 1, that simply means no parallelism at all. A quite different approach [2] is based on the synergistic use of neuron parallelism and synaptic parallelism to split the neural network. Suppose, for the sake of simplicity, that the number of processors to be used equals M, the generalization being straightforward; in this case we can map, on each processor, an hidden unit and, as concerns the input layer, we assume that each processor knows all the training input/output pairs. (The last of course holds for Time-Series forecasting since the training set is a simple array). If each processor knows each training pattern it can evaluate (without communication) the activation value of the handled hidden neuron. Once the components of the sum have been evaluated what it is needed is a reduction operator that allows each processor to know 0(x(). Last, we have to apply Eqns. 9, 10 and 11; by using our strategy each processor knows all it needs to evaluate the A's related to parameters handled without communication. A (maybe negligible) drawback, clearly, is that the error term e, has to be evaluated in each processor. Using the proposed schema, clearly we have that P^ equals H, and this does not seem an hard constrain when coarse-grained parallel computers are considered. If fine-grained machines are considered, on the other hand, it is highly possible that the number of processors is » H\ if this is the case a lower level parallelism has to be used to increase Pmax (this is the concern of our work in progress).
114
4. Implementations and Results Given a problem and a possible solution, the actual implementation step must be clearly addressed. When the problem is the parallel implementation of an algorithm (the solution being a mapping scheme) the implementation step typically is the porting step; to writer's mind, as better is the solution as easier will be the porting. For the problem at hand, we will try to show that the porting problem is trivial when the described solution is used. The Meiko CS-2 is a MIMD parallel computer able to provide both scalar and vector processing nodes; in our configuration it is composed by SPARK multiprocessors nodes (2 processor for each node). The communication network is a logarithmic network (a Benes network folded about its centre line) constructed from 8 way cross-point switches and bi-directional links; each node has its own communication processor, i.e. a dedicated interface to the communication network. A key point is the availability of a library of network functions which allow user to efficiently solve some communication problems without using low-level routines. For examples, very simple and efficient routines are provided for message broadcasting, for DMA accessing of any node memory, etc. Particularly interesting for the problem at hand is the gsum function; this function performs a reduce operation on a vector, and the result is broadcast to all processors. Using the gsum function and the suggested mapping the porting problem can be solved very easily; suppose to start from a sequential implementation where it is possible to set at run-time both M and H (and this does not seem too difficult). To obtain the parallel version we have to: 1. add the code to determine the number of processes involved in the current simulation; 2. add the code to determine the number of hidden neurons that each process has to simulate; 3. add a gsum call in the forward phase to complete equation. 7; 4. modify I/O routines. Since just step 3 notably affects the performance we have that: rjiSeq
TPar - - — + T g s u m (12) P where 7**"" represents the time to perform reductions. The speed-up obtained as the number P of processor vary having M as parameter and with #=32 is shown in figure 2 (left); the results obtained as P vary having H as parameter with M=16 are shown in figure 2 (right). Clearly, having fixed P, the speed-up increases as M or H increases; this is not surprising, since the communication overhead does not increases with the number of free parameters of the neural network. As concerns the load balancing, it is worth noting that, in an
115 14 •
12
/ /.
^r •^
AJf^
C
4
10
I
8
*^tf^ r^
.
«•
'
4^x£-+ IP -
6 4 2
=yg
0
8
12
1g
8 P
P
Rgure 1. Left: the speed-up having M as parameter (circle is M=8, square is M=16 and triangle is M=32). Right: the speed-up having H as parameter (circle is H=16, square is //=32 and triangle is //=64).
actual implementation, the speed-up clearly depends on
; this effect can be
well appreciated for small networks (see the case H-16). It is worth noting that, in the experiments, processors belongs to different nodes (see section 5). 7*""", clearly, depends just on P, being fixed to 1 the dimension of the vector to be combined, and it is T*sl""=0(lg2P), being the processors arranged in a tree. This, together with equation 12, suggests the following: TgSUm^TPar_i
fSeq
„* i + * 2 [lgp1
(13)
Using the available 48 data points it is possible to give a rather accurate estimates of k, and k2 solving the least square linear problem based on equation 13. When the time are in usees for 1 iteration on 1 example we have that k, = 5.3 and k2 = 22; 43 out of the 48 data points (90%) are in the range k,+ k2\\g2P'] ± 30% (see figure 3). The suggested analysis could seem too inaccurate, but it is worth remembering that we are talking of usees on a MIMD machine, with each node running its own operating system. 5. Concluding Remarks In this paper it is proposed a mapping scheme for the gradient descent algorithm; we are using this algorithm in the learning phase of GRBF Networks for Time-Series forecasting. The used approach is based on a mapping scheme whose main drawback is that each processor has to know the whole training set; such a drawback does not apply at the problem under study being the training set a simple array. The scheme is synchronous so that both MIMD and SIMD architectures can be used. Results obtained on the Meiko CS-2, a MIMD coarse-grained parallel computer, have been shown. The results demonstrate the worth of the suggested approach;
116
another useful result obtained is that coding the parallel program is almost as easy as coding the sequential one. A problem we are dealing with is that the code isn't able to efficiently use both the processors constituting the node and sharing the memory; roughly speaking, this means that, for example, the efficiency obtained using 8 processors is higher if the processors belongs to different nodes. This surprising and 180 undesired behaviour depends (i) • 160 on the fact that the gsum function 140 does not avail itself of 120 shared-memory mechanisms to /r~* • • 100 combine vectors residing in the /*— • ' ' ' so same memory and (ii) on the fact A . • i 8 «0 that the 2 processors share the 40 same communication processor so 20 that many conflicts on such a 0 critical resource exist. To solve the problem a new version of the combination function able to overcome such a substantial ss m Figure 3. T " : the model ± 30% vs. data points. deficiency is under development.
References M. Besch and H. W. Pohl, "Flexible Data Parallel Training of Neural Networks Using MIMD-Computers", In Third Euromicro Workshop on Parallel and Distributed Processing, Sanremo, Italy, January 1995. A. d'Acierno, "Back-Propagation Learning Algorithm and Parallel Computers: The CLEPSYDRA Mapping Scheme", Neurocomputing, 31 (2000), 67-85. F. Girosi and T. Poggio, "Networks and the best approximation property", Biological Cybernetics, Vol. 63 (1990) 169-176. H. Klapuri et al, "Mapping Artificial Neural Networks To a Tree Shape Neurocomputer", Microprocessors & Microsystems, 20 (5) (1992), 267-276. 5. S.Y. Kung and and J.N. Hwang, "Parallel Architectures for Artificial Neural Nets", Proc. ICNN, San Diego, CA, 1988, vol. 2, 165-172. 6. X. Zhang, M. Mckenna, J.P. Mesirov, and D. L. Waltz, "The Back-Propagation Algorithm on Grid and Hypercube Architectures", Parallel Computing, 14 (1990)317-327.
PARALLEL SIMULATION OF A CELLULAR LANDSLIDE MODEL USING CAMELOT
GIUSEPPE DATTILO AND GIANDOMENICO SPEZZANO ISI-CNR, c/o DEIS Universita delta Calabria, 87036 Rende(CS), ITALY E-mail: [email protected]. unical. it
The simulation of landslide hazards is a key point in the prevention of natural disasters, since it enables to compute risk maps and helps to design protection works. We present a parallel simulator that handles debris/mud flows developed by a problem-solving environment, called CAMELot, which allows interactive simulation and steering of parallel cellular computations. CAMELot is a system that uses the cellular automata formalism to model and simulate dynamic complex phenomena on parallel machines. It combines simulation, visualisation, control and parallel processing into one tool that allows to interactively explore a simulation, visualise the state of the computation as it progresses and change parameters, resolution or representation on the fly. In the paper, we give an overview of the CAMELot system and we show its practical use for programming and steering landslides models developed in the project. Moreover, an evaluation of the performances of the simulator on the MEKO CS-2 parallel machine is presented.
1
Introduction
The rapid development of computer technology has opened up new perspectives, and has increased the importance of computer simulation as an essential tool in research and experimentation. Today, we can build simulation models which allow to predict die behavior of real physical systems in actual or hypothetical circumstances. Moreover, the advent of parallel computing is making complex simulations in many disciplines tractable, increasing the interest in different application fields. One area of remarkable interest is the simulation of natural disasters as landslides, earthquakes and floods. In these simulations we must face the difficult scientific problem of predicting how territo