PARALLEL
COMPUTING: Software Technology, Algorithms, Architectures and Applications
ADVANCESIN PARALLELCOMPUTING VOLUME 13
Series Editor:
Gerhard R. Joubert ManagingEditor
(Technical University of Clausthal) Aquariuslaan 60 5632 BD Eindhoven,The Nmherlands
2004
ELSEVIER Amsterdam - Boston - Heidelberg - London - New Y o r k - Oxford Paris- San D i e g o - San Francisco- Singapore- S y d n e y - Tokyo
PARALLEL COMPUTING: Software Technology, Algorithms, Architectures and Applications
Edited by
G.R. Joubert
W.E. Nagel
Clausthal
Dresden
Germany
Germany
EJ. Peters
W.V. Walter
Eindhoven
Dresden
The Netherlands
Germany
20O4
ELSEVIER Amsterdam - Boston - Heidelberg - London - New York - Oxford Paris- San D i e g o - San Francisco- S i n g a p o r e - S y d n e y - Tokyo
ELSEVIER B.V. Sara Burgerhartstraat 25 P.O. Box 211, 1000 AE Amsterdam, The Netherlands
ELSEVIER Inc. 525 B Street, Suite 1900 San Diego, CA 92101-4495 USA
ELSEVIER Ltd The Boulevard, Langford Lane Kidlington,Oxford OX5 1GB UK
ELSEVIER Ltd 84 Theobalds Road LondonWC1X 8RR UK
9 2004 Elsevier B.V. All rights reserved. This work is protected under copyright by Elsevier B.V., and the following terms and conditions apply to its use: Photocopying Single photocopies of single chapters may be made for personal use as allowed by national copyright laws. Permission of the Publisher and payment of a fee is required for all other photocopying, including multiple or systematic copying, copying for advertising or promotional purposes, resale, and all forms of document delivery. Special rates are available for educational institutions that wish to make photocopies for non-profit educational classroom use. Permissions may be sought directly from Elsevier's Rights Department in Oxford, UK: phone (+44) 1865 843830, fax (+44) 1865 853333, e-mail:
[email protected]. Requests may also be completed on-line via the Elsevier homepage (http://www.elsevier.com/locate/permissions). In the USA, users may clear permissions and make payments through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; phone: (+1) (978) 7508400, fax: (+1) (978) 7504744, and in the UK through the Copyright Licensing Agency Rapid Clearance Service (CLARCS), 90 Tottenham Court Road, London W1P 0LP, UK; phone: (+44) 20 7631 5555; fax: (+44) 20 7631 5500. Other countries may have a local reprographic rights agency for payments. Derivative Works Tables of contents may be reproduced for internal circulation, but permission of the Publisher is required for external resale or distribution of such material. Permission of the Publisher is required for all other derivative works, including compilations and translations. Electronic Storage or Usage Permission of the Publisher is required to store or use electronically any material contained in this work, including any chapter or part of a chapter. Except as outlined above, no part of this work may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the Publisher. Address permissions requests to: Elsevier's Rights Department, at the fax and e-mail addresses noted above. Notice No responsibility is assumed by the Publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made.
First edition 2004 Library of Congress Cataloging in Publication Data A catalog record is available from the Library of Congress. British Library Cataloguing in Publication Data A catalogue record is available from the British Library.
ISBN: 0-444-51689-1 O The paper used in this publication meets the requirements of ANSI/NISO Z39.48-1992 (Permanence of Paper). Printed in The Netherlands.
PREFACE Dresden, a city of science and technology, of fine arts and baroque architecture, of education and invention, location of important research institutes and high tech firms in IT and biotechnology, and gateway between Western and Eastern Europe, attracted 175 scientists for the international conference on parallel computing ParCo2003 from 2 to 5 September 2003. It was the tenth in the biannual ParCo series, the longest running European conference series covering all aspects of parallel and high performance computing. ParCo2003 was once again a milestone in gauging the status quo of research and the state of the art in the development and application of parallel and high performance computing techniques, highlighting both current and future trends. The conference was hosted by the Center of High Performance Computing (ZHR) of the Technical University of Dresden. Since its foundation in 1828, the TU Dresden has undergone tremendous transformations from engineering school to technical school of higher education to full university. Today, the TU Dresden offers a broad range of subjects and specialisations in a wide variety of fields to about 33000 students. In the tradition of many inventions in the development of mechanical calculators and early computers, the Center for High Performance Computing was founded in 1997 and has played an important role in the development of modem methods and tools to support high performance computing at the university and beyond. Nowadays, many aspects of parallel computing have become part of mainstream computing. It is now commonplace to buy commodity off-the-shelf computers for home and office use that incorporate parallel techniques such as superscaIarity, hyper threading, VLIW (Very Long Instruction Word) and even cluster technologies that were considered advanced a mere decade ago. Quite apart from the speed with which new parallel technologies find their way into new products, these developments underline the importance of parallel computing research and development for the advancement of computer science and IT in general. In view of the rapid technology transfer taking place, one could be led to conclude that parallel computing research and development has passed its zenith since it has become standard computing practice. ParCo2003 showed that such a conclusion is invalid and that many complex research issues remain to be investigated. Thus it is c l e a r - and this has been the case for a number of years - that future research in parallel computing will have to concentrate increasingly on all aspects of software engineering. In addition the development of new architectures, especially those based on new technologies such as nanotechnologies, biocomputing, improved methods for performance evaluation, advanced algorithms, etc. must continue to receive appropriate attention.
Historical aspects and current trends were highlighted by three invited talks: 9 Friedel HoBfeld (Germany): Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday 9 Manfred Zorn (USA): Computational Challenges in the Genomics Era 9 Charles D. Hansen (USA): High-Performance Visualisation: So Much Data, So Little Time
vi Furthermore, there were more than 80 contributed papers broadly grouped under three main topics: Applications, Algorithms, and Software & Technology. Most of the papers received three and sometimes even four reviews, and we want to thank all members of the Programme Committee for their diligent work. In contrast to previous ParCo conferences, this conference put a strong emphasis on minisymposia, which constituted two entire parallel tracks of the conference. The topics of the seven minisymposia were: Title Organised by Franz-Josef Pfreundt Grid Computing Manfred Zom and Craig Stewart Bioinformatics, Allen Malony Performance Analysis Barbara Chapman OpenMP Tor Sorevik Parallel Applications Anne C. Elster Cluster Computing Alessandro Genco Mobile Agents We wish to thank the organisers of these minisymposia for their tremendous work and support in attracting excellent speakers and in reviewing the papers. The invited speakers, authors of contributed papers and participants in the industrial session as well as in the various minisymposia highlighted many challenging application areas of parallel computing, such as bioinformatics and genomics, visualisation, image and video processing, modelling and simulation, mobile agents and data mining. Due to the complexity of the problems encountered, parallel computing paradigms often provide the only feasible approach. The overall picture conveyed by the conference was thus one of consolidation of parallel computing technologies and transfer of these into off-the-shelf products on the one hand, and of emerging new areas of research and development on the other. The editors are greatly indebted to the members of the International Programme Committee as well as of the Steering, Organising, Finance and Exhibition Committees for the time they spent in making this conference such a successful event. Many thanks are due to the staff of the Center for High Performance Computing for their enthusiastic support. This was a major factor in making this conference a great success. Our special thanks go to Claudia Schmidt (general organisation), Heike Jagode (overall design), Thomas Blfimel (web administration), Stefan Pflfiger (exhibition), and Guido Juckeland (proceedings).
Gerhard R. Joubert Germany
February 2004
Wolfgang E. Nagel Germany
Frans J. Peters Netherlands
Wolfgang V. Walter Germany
vii
SPONSORS AMD GmbH Cray Computer Deutschland GmbH Hewlett-Packard GmbH, Gesch~iftsstelle Berlin IBM Deutschland GmbH, Fachb. Lehre u. Forschung Megware Computer GmbH NEC High Performance Computing Europe GmbH Pallas GmbH Silicon Graphics GmbH SUN Microsystems GmbH
E X H I B I T O R S / PARTICIPANTS IN THE I N D U S T R I A L T R A C K Cray Computer Deutschland GmbH Hewlett-Packard GmbH, Gesch~iftsstelle Berlin IBM Deutschland GmbH, Fachb. Lehre u. Forschung Megware Computer GmbH NEC High Performance Computing Europe GmbH Silicon Graphics GmbH SUN Microsystems GmbH
viii CONFERENCE
COMMITTEE
Gerhard R. Joubert (Conference Chair, Germany/Netherlands) Wolfgang E. Nagel (Germany) Frans J. Peters (Netherlands) Wolfgang V. Walter (Germany)
STEERING COMMITTEE Frans J. Peters (Chair, Netherlands) Friedel Hossfeld (Germany) Paul Messina (USA) Masaaki Shimasaki (Japan) Denis Trystram (France) Marco Vanneschi (Italy)
ORGANISING COMMITTEE Wolfgang V. Walter (Chair, Germany) Thomas Blfimel (Germany) Uwe Fladrich (Germany) Heike Jagode (Germany) Guido Juckeland (Germany) Claudia Schmidt (Germany) Bernd Trenkler (Germany) Andrea Walther (Germany)
EXHIBITION SUB-COMMITTEE Stefan Pflfiger (Chair, Germany) Norbert Attig (Germany) Hubert Busch (Germany) Wolf-Dietrich Harz (Germany) Matthias Mfiller (Germany)
FINANCE COMMITTEE Frans J. Peters (Chair, Netherlands)
ix
PROGRAMME COMMITTEE Wolfgang E. Nagel (Chair, Germany) Nikolaus Adams (Germany) Hamid Arabnia (USA) Norbert Attig (Germany) Eduard Ayguad6 (Spain) Achim Basermann (Germany) Christian Bischof (Germany) Petter E. Bjorstad (Norway) Arndt Bode (Germany) Thomas Brandes (Germany) Mats Brorsson (Sweden) Helmar Burkhart (Switzerland) Barbara Chapman (USA) Michel Cosnard (France) Pasqua D'Ambra (Italy) Luisa D'Amore (Italy) Erik D'Hollander (Belgium) Koen De Bosschere (Belgium) Luiz DeRose (USA) Andreas Deutsch (Germany) Beniamino Di Martino (Italy) Michael Eiermann (Germany) Rtidiger Esser (Germany) Thomas Fahringer (Austria) Afonso Ferreira (France) Salvatore Filippone (Italy) Michael Gerndt (Germany) Lucio Grandinetti (Italy) Andreas Griewank (Germany) John Gurd (UK) Volker Gtilzow (Germany) Bianca Habermann (Germany) Rolf Hempel (Germany) Hans-Christian Hoppe (Germany) Lennart Johnsson (USA) Dieter Kranzlmiiller (Austria) Norbert Kroll (Germany) Herbert Kuchen (Germany) Keqin Li (USA) Thomas Lippert (Germany) Thomas Ludwig (Germany) Allen Malony (USA)
Tomas Margalef (Spain) Djordje Maric (Switzerland) Federico Massaioli (Italy) Arndt Meyer (Germany) Bemd Mohr (Germany) Almerico Murli (Italy) Per Oster (Sweden) Jean-Louis Pazat (France) Franz-Josef Pfreundt (Germany) Wilfried Philips (Belgium) Michael J. Quinn (USA) Roll Rabenseifner (Germany) Thomas Rauber (Germany) Alexander Reinefeld (Germany) Michael Resch (Germany) Richard Reuter (Germany) Jean Roman (France) Mathilde Romberg (Germany) Dirk Roose (Belgium) Hanns Ruder (Germany) Gudula Rtinger (Germany) Tor Sorevik (Norway) Jens Simon (Germany) Horst Simon (Germany) Henk J. Sips (Netherlands) Erich Strohmaier (USA) Vaidy Sunderam (USA) Mateo Valero (Spain) Marco Vanneschi (Italy) Jeffrey Vetter (USA) Heinrich Voss (Germany) Martin Walker (Switzerland) Wolfgang Walter (Germany) Helmut Weberpals (Germany) Roland Wismtiller (Germany) Gabriel Wittum (Germany) Rtidiger Wolff (Germany) Emilio Zapata (Spain) Hans Zima (Austria) Manfred Zorn (USA)
This Page Intentionally Left Blank
xi CONTENTS Preface Sponsors, Exhibitors / Participants in the industrial track Committees
V
vii viii
Invited Papers Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday E Hossfeld So Much Data, So Little Time... C. Hansen, S. Parker, C. Gribble
Software Technology
13
21
On Compiler Support for Mixed Task and Data Parallelism T. Rauber, R. Reilein, G. Riinger
23
Distributed Process Networks - Using Half FIFO Queues in CORBA A. Amar, P. Boulet, J.-L. Dekeyser, E Theeuwen
31
An efficient data race detector backend for DIOTA M. Ronsse, B. Stougie, J. Maebe, E Cornelis, K. De Bosschere
39
Pipelined parallelism for multi-join queries on shared nothing machines M. Bamha, M. Exbrayat
47
Towards the Hierarchical Group Consistency for DSM systems : an efficient way to share data objects L. LefOvre, A. Bonhomme An operational semantics for skeletons M. Aldinucci, M. Danelutto A Programming Model for Tree Structured Parallel and Distributed Algorithms and its Implementation in a Java Environment H. Moritsch
55
63
71
A Rewriting Semantics for an Event-Oriented Functional Parallel Language E Loulergue
79
RMI-like communication for migratable software components in HARNESS M. Migliardi, R. Podesta
87
Semantics of a Functional BSP Language with Imperative Features E Gava, E Loulergue
95
xii The Use of Parallel Genetic Algorithms for Optimization in the Early Design Phases E. Slaby, W. Funk An Integrated Annotation and Compilation Framework for Task and Data Parallel Programming in Java H.J. Sips, K. van Reeuwijk
103
111
On The Use of Java Arrays for Sparse Matrix Computations G. Gundersen, T. Steihaug
119
A Calculus of Functional BSP Programs with Explicit Substitution
127
E Loulergue JToe: a Java API for Object Exchange
135
S. Chaumette, P. Grange, B. MOtrot, P. Vign&as A Modular Debugging Infrastructure for Parallel Programs D. Kranzlmiiller, Ch. Schaubschldger, M. Scarpa, J. Volkert
143
Toward a Distributed Computational Steering Environment based on CORBA O. Coulaud, M. Dussere, A. Esnard
151
Parallel Decimation of 3D Meshes for Efficient Web-Based Isosurface Extraction A. Clematis, D. D'Agostino, M. Mancini, V. Gianuzzi
159
Parallel P r o g r a m m i n g
167
MPI on a Virtual Shared Memory F. Baiardi, D. GuerrL P. Mori, L. RiccL L. Vaglini
169
OpenMP vs. MPI on a Shared Memory Multiprocessor J. Behrens, O. Haan, L. Kornblueh
177
MPI and OpenMP implementations of Branch-and-Bound Skeletons I. Dorta, C. Le6n, C. Rodriguez, A. Rojas
185
Parallel Overlapped Block-Matching Motion Compensation Using MPI and OpenMP E. Pschernig, A. Uhl
193
A comparison of OpenMP and MPI for neural network simulations on a SunFire 6800 A. Strey
201
Comparison of Parallel Implementations of Runge-Kutta Solvers: Message Passing vs. Threads M. Korch, T. Rauber
209
xiii
Scheduling
217
Extending the Divisible Task Model for Workload Balancing in Clusters U. Rerrer, O. Kao, F. Drews
219
The generalized diffusion method for the load balancing problem G. Karagiorgos, N. Missirlis, E Tzaferis
225
Delivering High Performance to Parallel Applications Using Advanced Scheduling N. Drosinos, G. Goumas, M. Athanasaki, N. Koziris
233
Algorithms
241
Multilevel Extended Algorithms in Structural Dynamics on Parallel Computers K. Elssel, H. Voss
243
Parallel Model Reduction of Large-Scale Unstable Systems P. Benner, M. Castillo, E.S. Quintana-Orti, G. Quintana-Orti
251
Parallel Decomposition Approaches for Training Support Vector Machines T. Serafini, G. Zanghirati, L. Zanni
259
Fast parallel solvers for fourth-order boundary value problems M. Jung
267
Parallel Solution of Sparse Eigenproblems by Simultaneous Rayleigh Quotient Optimization with FSAI preconditioning L. Bergamaschi, ,4. Martinez, G. Pini
275
An Accurate and Efficient Selfverifying Solver for Systems with Banded Coefficient Matrix C. H61big, W. Krdmer, T.A. Diverio
283
3D parallel calculations of dendritic growth with the lattice Boltzmann method W. Miller, E Pimentel, I. Rasin, U. Rehse
291
Distributed Negative Cycle Detection Algorithms L. Brim, 1. Cernd, L. Hejtmdnek
297
A Framework for Seamlesly Making Object Oriented Applications Distributed S. Chaumette, P. VignOras
305
Performance Evaluation of Parallel Genetic Algorithms for Optimization Problems of Different Complexity P. K6chel, M. Riedel
313
xiv Extensible and Customizable Just-In-Time Security (JITS) Management of ClientServer Communication in Java S. Chaumette, P. VignOras
Applications & Simulation An Object-Oriented Parallel Multidisciplinary Simulation System U. Tremel, F. Deister, K.A. Sorensen, H. Rieger, N.P. Weatherill
321
329 The SimServer
331
Computer Simulation of Action Potential Propagation on Cardiac Tissues: An Efficient and Scalable Parallel Approach J.M. Alonso, J.M. Ferrero (Jr.), V. Herndndez, G. Molt6, M. Monserrat, J. Saiz
339
M o D y S i m - A parallel dynamical UMTS simulator M.J. Fleuren, H. Stiiben, G.F. Zegwaard
347
apeNEXT: a Multi-TFlops Computer for Elementary Particle Physics F. Bodin, Ph. Boucaud, N. Cabibbo, F. Di Carlo, R. De Pietri, E Di Renzo, H. Kaldass, A. Lonardo, M. Lukyanov, S. de Luca, J. Micheli, V. Morenas, N. Paschedag, O. Pene, D. Pleiter, E Rapuano, L. Sartori, F. Schifano, H. Simma, R. Tripiccione, P. Vicini
355
The Parallel Model System LM-MUSCAT for Chemistry-Transport Simulations: Coupling Scheme, Parallelization and Applications R. Wolke, O. Knoth, O. Hellmuth, W. Schrdder, E. Renner
363
Real-time Visualization of Smoke through Parallelizations T. Vik, A.C. Elster, T. Hallgren
371
Parallel Simulation of Cavitated Flows in High Pressure Systems P.A. Adamidis, F. Wrona, U. Iben, R. Rabenseifner, C.-D. Munz
379
Improvements in black hole detection using parallelism F. Almeida, E. Mediavilla, A. Oscoz, E de Sande
387
High Throughput Computing for Neural Network Simulation J. Culloty, P. Walsh
395
Parallel algorithms and data assimilation for hydraulic models C. Mazauric, V.D. Tran, W. Castaings, D. Froehlich, EX. Le Dimet
403
Multimedia Applications Parallelization of VQ Codebook Generation using Lazy PNN Algorithm A. Wakatani
413 415
XV
A Scalable Parallel Video Server Based on Autonomous Network-attached Storage G. Tan, S. Wu, H. Jin, E Xian
423
Efficient Parallel Search in Video Databases with Dynamic Feature Extraction S. Geisler
431
Architectures
439
Introspection in a Massively Parallel PIM-Based Architecture H.P. Zima Time-Transparent Inter-Processor Connection Reconfiguration in Parallel Systems Based on Multiple Crossbar Switches E. Laskowski, M. Tudruj SIMD design to solve partial differential equations R. W. Schulze
441
449
457
Caches
465
Trade-offs for Skewed-Associative Caches H. Vandierendonck, K. De Bosschere
467
Cache Memory Behavior of Advanced PDE Solvers D. Wallin, H. Johansson, S. Holmgren
475
Performance
483
A Comparative Study of MPI Implementations on a Cluster of SMP Workstations G. Riinger, S. Trautmann
485
MARMOT: An MPI Analysis and Checking Tool B. Krammer, K. Bidmon, M.S. Miiller, M.M. Resch
493
BenchIT- Performance Measurement and Comparison for Scientific Applications G. Juckeland, S. Bdrner, M. Kluge, S. K6lling, WE. Nagel, S. Pfliiger, H. R6ding, S. Seidl, T. William, R. Wloch
501
Performance Issues in the Implementation of the M-VIA Communication Software Ch. Fearing, D. Hickey, P.A. Wilsey, K. Tomko
509
Performance and performance counters on the Itanium 2 study U. Andersson, P. Ekman, P. Oster
A benchmarking case
On the parallel prediction of the RNA secondary structure F. Almeida, R. Andonov, L.M. Moreno, V. Poirriez, M. Pdrez, C. Rodriguez
517
525
xvi Clusters
533
M D I C E - a MATLAB Toolbox for Efficient Cluster Computing R. Pfarrhofer, P Bachhiesl, M. Kelz, H. St6gner, A. Uhl
535
Parallelization of Krylov Subspace Methods in Multiprocessor PC Clusters D. Picinin Jr., A.L. Martinotto, R. V. Dorneles, R.L. Rizzi, C. H61big, T.A. Diverio, P. O.A. Navaux
543
First Impressions of Different Parallel Cluster File Systems T.P Boenisch, P W. Haas, M. Hess, B. Krischok
551
Fast Parallel I/O on ParaStation Clusters N. Eicker, F. Isaila, T. Lippert, T. Moschny, W.F. Tichy
559
PRFX : a runtime library for high performance programming on clusters of SMP nodes B. Cirou, M.C. Counilh, J. Roman Grids
569
577
Experiences about Job Migration on a Dynamic Grid Environment R.S. Montero, E. Huedo, I.M. Llorente
579
Security in a Peer-to-Peer Distributed Virtual Environment J. K6hnlein
587
A Grid Environment for Diesel Engine Chamber Optimization G. Aloisio, E. Blasi, M. Cafaro, I. Epicoco, S. Fiore, S. Mocavero
599
A Broker Architecture for Object-Oriented Master/Slave Computing in a Hierarchical Grid System M. Di Santo, N. Ranaldo, E. Zimeo A framework for experimenting with structured parallel programming environment design M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, M. Danelutto, P. Pesciullesi, R. Ravazzolo, M. Torquati, M. Vanneschi, C. Zoccolo M i n i s y m p o s i u m - Grid C o m p u t i n g
Considerations for Resource Brokerage and Scheduling in Grids R. Yahyapour Job Description Language and User Interface in a Grid context: The EU DataGrid experience G. Avellino, S. Beco, E Pacini, A. Maraschini, A. Terracina
609
617
625
627
635
xvii On Pattern Oriented Software Architecture for the Grid H. Prem, N.R. Srinivasa Raghavan
Minisymposium- Bioinformatics Green Destiny + mpiBLAST = Bioinfomagic W. Feng
643
651 653
Parallel Processing on Large Redundant Biological Data Sets: Protein Structures Classification with CEPAR D. Pekurovsky, I. Shindyalov, P. Bourneb
661
MDGRAPE-3: A Petaflops Special-Purpose Computer System for Molecular Dynamics Simulations M. Taiji, T. Narumi, Y. Ohno, A. Konagaya
669
Structural Protein Interactions: From Months to Minutes P. Dafas, J. Gomoluch, A. Kozlenkov, M. Schroeder
677
Spatially Realistic Computational Physiology: Past, Present and Future J.R. Stiles, W.C. Ford, J.M. Pattillo, T.E. Deerinck, M.H. Ellisman, T.M. Bartol, T.J. Sejnowski
685
Cellular automaton modeling of pattern formation in interacting cell systems A. Deutsch, U. BOrner, M. B~ir
695
Numerical Simulation for eHealth: Grid-enabled Medical Simulation Services S. Benkner, W. Backfrieder, G. Berti, J. Fingberg, G. Kohring, J.G. Schmidt, S.E. Middleton, D. Jones, J. Fenner
705
Parallel computing in biomedical research and the search for peta-scale biomedical applications C.A. Stewart, D. Hart, R. W. Sheppard, H. Li, R. Cruise, V. Moskvin, L. Papiez
Minisymposium- Performance Analysis
719
727
Big Systems and Big Reliability Challenges D. A. Reed, C. Lu, C.L. Mendes
729
Scalable Performance Analysis of Parallel Systems: Concepts and Experiences H. Brunst, W.E. Nagel
737
CrossWalk: A Tool for Performance Profiling Across the User-Kernel Boundary A. V. Mirgorodskiy, B.P. Miller
745
Hardware-Counter Based Automatic Performance Analysis of Parallel Programs F. Wolf, B. Mohr
753
xviii Online Performance Observation of Large-Scale Parallel Applications A.D. Malony, S. Shende, R. Bell
761
Deriving analytical models from a limited number of runs R.M. Badia, G. Rodriguez, J. Labarta
769
Performance Modeling of HPC Applications A. Snavely, X. Gao, C. Lee, L. Carrington, N. Wolter, J. Labarta, J. Gimenez, P. Jones
777
Minisymposium - OpenMP
785
Thread based OpenMP for nested parallelization R. Blikberg, T. Sorevik
787
OpenMP on Distributed Memory via Global Arrays L. Huang, B. Chapman, R.A. Kendall
795
Performance Simulation of a Hybrid OpenMP/MPI Application with HESSE R. Aversa, B. Di Martino, M. Rak, S. Venticinque, U. Villano
803
An environment for OpenMP code parallelization C.S. Ierotheou, 11. Jin, G. Matthews, S.P. Johnson, R. Hood
811
Hindrances in OpenMP programming E Massaioli
819
Wavelet-Based Still Image Coding Standards on SMPs using OpenMP R. Norcen, A. Uhl
827
M i n i s y m p o s i u m - Parallel A p p l i c a t i o n s
835
Parallel Solution of the Bidomain Equations with High Resolutions X. CaL G.T. Lines, A. Tveito
837
Balancing Domain Decomposition Applied to Structural Analysis Problems P. E. Bjorstad, J. Koster
845
Multiperiod Portfolio Management Using Parallel Interior Point Method L. Halada, M. Lucka, I. Melichercik
853
Performance of a parallel split operator method for the time dependent Schr6dinger equation T. Matthey, T. Sorevik
861
xix Minisymposium - Cluster Computing
869
Design and implementation of a 512 CPU cluster for general purpose supercomputing B. Vinter
871
Experiences Parallelizing, Configuring, Monitoring, and Visualizing Applications for Clusters and Multi-Clusters O.J. Anshus, J.M. Bjorndalen, L.A. Bongo
879
Cluster Computing as a Teaching Tool O.J. Anshus, A.C. Elster, B. Vinter Minisymposium- Mobile Agents
887
895
Mobile Agents Principles of Operation A. Genco
897
Mobile Agent Application Fields E Agostaro, A. Genco, S. Sorce
905
Mobile Agents and Grid Computing E Agostaro, A. Chiello, A. Genco, S. Sorce
913
Mobile Agents, Globus and Resource Discovery E Agostaro, A. Genco, S. Sorce
919
A Mobile Agent Tool for Resource Discovery E Agostaro, A. Genco, S. Sorce
927
Mobile Agents and Knowledge Discovery in Ubiquitous Computing A. Genco
935
Author & Subject Index
Author Index Subject Index
943 945 951
This Page Intentionally Left Blank
Invited Papers
This Page Intentionally Left Blank
Parallel Computing:SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday F. Hossfeld a aChair for Technical Informatics and Computer Sciences, University of Technology (RWTH) Aachen, Central Institute for Applied Mathematics, Research Centre Juelich, Germany On 28 December 2003, the scientific community will celebrate the 100-th anniversary of John von Neumann's birthday. On this occasion, we are reminded of his achievements as outstanding mathematician and creator of Game Theory, but even more that he laid the very conceptional foundations of the digital computer. His concept had the fortune - contrary to Konrad Zuse's in Germany- that the transistor was invented in 1947, just when JvN wrote his famous reports on the digital computer, at Bell Labs thus giving rise to the extraordinary technological development of microelectronics pushed further by other inventions like photolithography and integrated circuits in the years to come. Since decades, the exponential growth of the power of microchips- every 18 months, the integration density of transistors on the chips is doublingis steering the also exponential growth of computer power. The top computers have surpassed the teraflops level by far today targeting towards 100 teraflops or even petaflops. The computer has become ubiquitous, and protagonists of robotics and artificial intelligence are tempted to attribute to it omnipotent capabilities which will lead to autonomous "humanoids" (Moravec, Kurzweil) on the one hand and threatening horror scenarios on the other (Joy). The predictions of the semiconductor industry tell us that "Moore's Law" describing the exponential evolution in microelectronics might remain valid for other 10 to 15 years. Beyond Moore's Law, quantum effects will definitely end the orderly functioning of"classical" circuits. In 1982, Richard Feynman pointed out that certain quantum mechanical systems could not be simulated efficiently on a "classical" digital computer how powerful it may evolve. This led to speculations that computation in general could be done, in principle and even more efficiently, if a novel computer could make thorough use of quantum effects, thus providing a challenging option for parallel computing by exploiting the exponential speedup through quantum parallelism. Peter Shor's factorization algorithm of 1994 showed that the quantum computer is capable to solve NP-hard problems efficiently. Experimentalists work on different physical concepts to realize quantum computation; for instance, quantum dots, trapped ions, superconducting devices, and NMR technology have been shown to provide the principles and the technology to build quantum computers. 1. LEGACIES In the early 1950-s, John von Neumann once said that three scientific achievements had changed the view of the world in the 20-th century: (1) Einstein's Theory of Relativity, (2)
Heisenberg's Quantum Mechanics, and (3) G6del's Incompleteness Theorem. In the second half of the 20-th century, the history of science and technology added a then unforeseen 4-th one: von Neumann's Digital Computer. John von Neumann was born in December 1903 in Budapest as son of a Hungarian Jewish banker. JvN was a mathematical infant prodigy with kind of photographic memory. Of course, he later was disposed to study mathematics; however, his father seeking advice from the famous von Kfirmfin then at the University of Technology in Aachen, JvN was to study chemical engineering which he humbly d i d - obviously in Budapest, Berlin, and Zurich. He finished these studies with the diploma in 1926. In the same year, however, in parallel he got his PhD in mathematics with a thesis on set theory certainly influenced at his visits to G6ttingen by Hilbert and his program. After lecturing in Berlin and Hamburg, he received an invitation as visiting professor from Princeton University which he accepted in 1930. In 1933, after the foundation of the Institute of Advanced Study in Princeton in 1930, he became together with Alexander and Veblen the first professors of mathematics at the institute which attracted afterwards Einstein, G6del, Church, Dyson, and Oppenheimer and many other famous scientists as well as, in 1937, as a guest scientist also Turing. Hence, JvN was well informed not only about the progress of physics but also on the Hilbert-program deconstructing work of G6del and on Turing's and Church's ideas of computation and computability. The concept of the Turing machine and algorithmics gave definitely major impact on his later plans and projects of designing and building the digital computer. Highly awarded, JvN died of bone cancer on February 8, 1957. Already in 1921, he received the Award of the Best Mathematician in Hungary. In 1927, he published five papers: three on quantum mechanics which had just been created by Heisenberg and Schr6dinger, one on a mathematical problem, and one which is considered as the foundation of game theory. In 1932, he published the important monography - in German - on the "Mathematical Foundations of Quantum Mechanics" where he set this theory on solid mathematical ground especially with his ideas on algebras which anew gain much attention in modem theoretical physics. From 1940 to 1945, he was deeply involved in the US Government's Manhattan Project continuing his involvement as an esteemed advisor until his death in Washington. However, simultaneously, he focussed his mathematical interests on grand challenges arising from the stalemate of the analysis of partial differential equations which again guided him to the definition, design, and construction of the digital computer to overcome the stagnation of the analytic mathematical treatment of complex problems like climate and weather forecast, to indicate only a few out of his broad spectrum of interdisciplinary challenges which led him to establish computer simulation as the third category of scientific methodology in addition to theory and experiment, a discipline which has later been named Computational Science (& Engineering) by the Nobel ~rize Winner Kenneth Wilson (1982). Honoured with the Medal of Freedom, the Albert Einstein Memorative Award and the Enrico Fermi Award and the membership in the American Academy of Sciences, the American Academy of Arts and Science, the American Philosophical Society and many other academies abroad, he dies too early to complete, besides his diverse projects in mathematics and computing, his thorough analysis of the potential of the digital computer and his "organs"- as he called its various components- and the similarities of the "digital brain" with, or rather its fundamental distinction from the human brain as he elaborated in his last publication, the fragmentary analysis of "The Computer and the Brain" which he worked out for the invited Silliman Lectures in 1956 which he never could give at Yale University [1, 2, 3, 4, 5].
2. COMPUTER TECHNOLOGY AND SUPERCOMPUTER ARCHITECTURE
While Konrad Zuse in Germany [6] as the definitely, and at last also in the US accepted, first digital computer pioneer struggled since the 1930s with the limited technical and technological resources available, JvN was lucky enough to design his digital computer concept at a time when the transistor was invented by John Bardeen, Walter Brattain and William Shockley in 1947 at Bell Labs - earning them the Nobel Prize in 1956 - which gave a tremendous push to the development of digital devices encouraging the STRETCH Program at IBM in 1955 in parallel with the development of the ferrite core memory by Jay Forrester and stimulating John Backus to create the first compiler for the high-level language Fortran in 1957 to harness the IBM 704 computer with 32,768 32-bit word ferrite core memory and a magnetic drum as secondary storage. In 1959, Jack Kilby at Texas Instruments developed the first integrated circuit (then on germanium substrate) and Robert Noyce at Fairchild Camera invented photolithography. These giant technological steps created the incredible explosion of microelectronics which resulted in 1970 in the road-paving achievement of the first microprocessor Inte14004. From then on, we can follow an exponential growth of digital computer power in parallel with the exponential growth of the densitiy of transistors on the chips which is acknowledged today as "Moore's Law" allowing for the doubling of microprocessor power and memory size within about 18 months. Personal computers, workstations, servers, and supercomputers follow this "Law" since decades enhanced even more by innovations in computer architecture and operating system concepts [7] leading to vector-processing functions and parallelism in modern supercomputers which nowadays reach far beyond the teraflops performance even with large microprocessor clusters while since early 2002 the Japanese "Earth Simulator"- a powerful combination of vector-processing and parallel structures built on NEC technology- is heading the TOP-500 list with more than 40 teraflops peak and about 35 teraflops sustained Linpack performance [8], thus causing similar shock waves -"Computnik" - within the political and technocratic circles in USA as by the Sputnik launched then by the late Soviet Union. This tremendous explosion of the digital computer is in almost all aspects guiding back to roots of computer architecture and programming as layed by JvN in the 1940-s. Scanning the history since the very birthday of Computational Science and Engineering, which as mentioned may be dated back to 1946 when JvN formulated the strategic program in his famous report on the necessity and future of digital computing together with H. H. Goldstine, at that time complex systems were primarily involved with fluid dynamics. JvN expected that really efficient high-speed digital computers would "break the stalemate created by the failure of the purely analytical approach to nonlinear problems" and suggested fluid dynamics as a source of problems through which a mathematical penetration into the area of nonlinear partial differential equations could be initiated. JvN envisioned computer output as providing scientists with those heuristic hints needed in all parts of mathematics for genuine progress and to break the deadlock- the "present stalemate"- in fluid dynamics by giving clues to decisive mathematical ideas. In a sense, his arguments sound very young and familiar. As far as fluid dynamics is concerned, in his John von Neumann Lecture at the SIAM National Meeting in 1981 yet Garett Birkhoff came to the conclusion on the development of fluid dynamics that it be unlikely that computational fluid dynamics (CFD) would become a truly mathematical science in the near future, although computers might soon rival windtunnels in their capabilities; both, however, would be ever essential for research [9, 10].
The various strategic position papers in the 1980s [11, 12, 13, 14, 15] and the government technology programs in the USA, in Europe, and in Japan in the early 1990s claimed that the timely provision of supercomputers to science and engineering and the ambitious development of innovative supercomputing hardware and software architectures as well as new algorithms and effective programming tools are an urgent research-strategic response to the grand challenges arising from these huge scientific and technological barriers. The solutions of complex problems are critical to the future of science, technology, and society. Supercomputing will be a crucial factor for the industry as well in order to meet the requirements of international economic competition especially in the area of high-tech products. Despite the remarkable investments in research centers and universities in building up supercomputing power and skills and also some sporadic efforts in the industry concerning supercomputing in Europe, it took until the 1990s that the U.S. Government and the European Union as well as several national European governments started non-military strategic support programs [ 16, 17, 18]. Their goals were also to enhance supercomputing by stimulating the technology transfer from universities and research institutions into industry and by increasing the fraction of the technical community which gets the opportunity to develop the skills required to efficiently access the highperformance computing resources. In recent years, computer simulation has reached even the highest political level, since, in 1996, the United Nations voted to adopt the Comprehensive Test-Ban Treaty banning all nuclear testing for military and peaceful purposes. Banning physical nuclear testing created a need for full-physical modeling and high-confidence computer simulation and, hence, unprecedented steps in supercomputer power. DoE's Accelerated Strategic Computing Initiative (ASCI) [ 19, 20] aiming to replace physical nuclear-weapons testing by computer simulation, and NSF's Partnerships for Advanced Computational Infrastructures [21 ] in the US targeting at the advancement of new computing and communication infrastructures for grid computing [22, 23] intended to revolutionize science and engineering- as well as business- is definitely establishing computer simulation as a fundamental methodology in science and engineering. The dedication of the Nobel Prize for Chemistry in 1998 to Computational Chemistry, in addition, confirmed its significance in the scientific community as well as in industry and politics. In the early 1990s, Cray vector-supercomputers with shared memory architecture and proprietary bipolar CPU technology were foreseeable to run into difficulties to provide the giant steps in compute power needed by greedy users, then particularly in physics. Supercomputer architectures with massive parallelism and distributed memory emerged as the future, also more cost-effective, line for the very top end. During these years, nearly thirty companies were offering massively parallel systems and others were planning to enter the market with new products, although many experts predicted that the market would not be able to sustain thus many vendors [24]. In [25], a chronological compilation of high performance computer history illuminates the "Darwinistic" forces affecting the supercomputer evolution lines. It demonstrates that the expected shake-out in the computer industry took place questioning the health and the future potential of this industry in total. Some went out of the parallel computer business for quite different reasons -, others became just mergers. The dramatic survival battle in the supercomputer industry was also giving severe damage to the users and the customers in the supercomputing arena. Quite often the critical situation of parallel computing has rigorously been analyzed with respect to the possible negative impacts on the future perspectives and the progress of this scientific discipline. Already Goldstine said that the history of computing is
littered with "australopithecanes", short computer lines which do not lead anywhere [3]. Following in the wake of DoE's ASCI program, in recent years powerful new supercomputers were brought into the market by the manufacturers participating in the high-performance computing race again. It seemed that only those supercomputer manufacturing companies would have a realistic chance to survive in the years to come who had the potential, capabilities, and favour to get involved in the ASCI or other significant US Government supported programs. However, in early 2002, the Japanese took the lead with the "Earth Simulator" which was argued by the climate problems arising around the world. The parallel architectures became quite successful on the TOP 500 list recently extending the parallel computer concept towards clustered symmetric multiprocessor (SMP) nodes mainly based on commodity components tying together possibly tens of thousands of processing elements [26]. Hence, the Central Institute for Applied Mathematics at the Research Centre Juelich running the National Supercomputer Centre "John von Neumann Institute for Computing" just inaugurated its 8.9 teraflops SMP-based new supercomputer "JUMP", now No. 1 in Europe, with 41 IBM p690 nodes of 32 processors each thus adding up to 1312 processors and surpassing its MPP system CRAY T3E/1200 by more than a factor of 10 in performance [27, 28]. The simulation of extremely complex systems may also determine future large-scale computing by interconnecting supercomputers of diverse architectures as giant supercomputer complexes via the new paradigm of grid computing. These developments will challenge not only system reliability, availability and serviceability to novel levels, but also interactivity of concurrent algorithms and, in particular, adaptivity, accuracy, and stability of parallel numerical methods [29]. In any case, however, a new technology will be necessary to target at and beyond the petaflops barrier in supercomputing, since the further expansion of the floor space of several thousands of square-meters needed by today's SMP-clustered systems with only tenths of teraflops peak performance will be neither reasonable nor technically sensibly feasible. IBM's Blue Gene project may give some guidance and, more aggressively, nanotechnology may provide innovative transistor concepts which will shift the limit of Moore's Law a bit more into the next decades [30]. 3. FROM BITS TO QUBITS
Nevertheless, the end of Moore's Law has been predicted for the time around 2020, at last, since extrapolation leads into nanoscales where the quantum regime will totally govern and, thus, by noise disturb the transistor functions. While the applications of"ubiquitous" computing and its specific requirements may be satisfied by the currently available and further improved technology, high-performance computing targeting towards multi-petaflops performance may severely suffer from the ending exponential growth. Therefore, alternative computing concepts breaking those performance barriers should be intensively explored in order to provide the means to deal with complex problems in future science, industry, and society. In 1982, Richard Feynman explained that the simulation of certain quantum physical systems cannot be simulated by "classical" computers; he, however, pointed out that quantum mechanics could possibly provide the principles to develop a fundamentally new computing paradigm: quantum computing, based on technologically and logically different computer architectures [31]. In 1984 already, David Deutsch designed a quantum computer model [32] which has been elaborated since then theoretically in remarkable depth [33] yielding, as an expansion
of the Church-Turing computational concept to quantum Turing machines, deep complexitytheoretical results and quantum algorithms which meet the expectation of exponential computational speedup due to the inherent exponential superposition of quantum states which, in analogy to classical Boolean bit-based logics, are represented by "qubits", thus exploiting the so-called quantum parallelism. Like in Boolean switching algebra with the NAND and NOR gates, in quantum computing also universal gates (Qubit and Conditional NOT: CNOT) are available besides versatile gates to build up logical circuits, and remarkably- like in classical computation- the Fourier Transform turned out to be a fundamental algorithmic element to exploit the exponential speedup in quantum algorithms. On this basis, Peter Shor succeeded in developing a factorization quantum algorithm [34] which not only provides exponential speedup compared with classical sequential algorithms for this well-known computationally hard problem, but also demonstrated that quantum computing is fundamentally capable to dissolve at least for a significant subset of NP-hard problems the NP - P question by providing efficient quantum algorithms [35]. Although there are diverse physical methods which provide the principal means to establish quantum computer technology and architecture (like NMR technology, quantum dots, trapped ions, superconducting devices and others) which in some cases already have been successfully demonstrated with small numbers of qubits, it will definitely take many years to establish, if at all, a sound quantum computer technology. There are many sources for limited stability and fundamental perturbations of entangled quantum states (decoherence). However, as science history has shown elsewhere, once the idea is born, it will be definitely pursued by scientific curiosity in innovative computational concepts and by the need of greedy applications as well. Thus, quantum computing represents a grand challenge for the years to come; it opens a new space for future intellectual adventures in research and development of physics, mathematics, and computer science. 4. TOWARDS THE "DIGITAL BRAIN"? Along with considerations on the future of computation and the understanding of the human brain, in his popular books Roger Penrose also addressed quantum mechanics and possibly emerging new theories in physics like String Theory and its future extensions as potential scientific vehicles to explain human brain functions and understanding mind and consciousness [36, 37, 38]. As indicated above, JvN discussed in his preliminarily sketched Silliman Lectures, published as "The Computer and the Brain" [5], the similarities and differences of the computer, as he defined it, and the brain, as he understood it at his time. Interestingly enough, in his famous reports defining the digital computer in the 1940-s he talked about the major computer components as "organs" thus showing that he in some sense entertained the long European tradition of ideas to design and construct humanoids which boomed in the 17th and 18-th centuries giving space to many, as seen today, ridiculous appearances of fluteplaying monkeys, chess-playing fakes and other curiosities. However, as a great mathematician he certainly also stood in the tradition of occidental philosophy which for centuries up to today is challenged by the mind-body problem reaching back to the ancient philosophers Thales of Milet ("the four element S " ), Anaxagoras of Clazomenae (introducing "nous", the mind), Leukippos, Demokrit and, last not least, Aristoteles and Platon via the Renaissance with Locke , ,, ), ("Essay concerning understanding"), Leibniz ("Nouveaux essais sur l'entendement h umam de la Mettrie ("L'homme plus que machine"), Descartes ("Discours de la m&hode" and "Med-
itationes de prima philosophia") towards Eccles ("The self and its brain", together with Popper) and the brain researchers of today like Edelman, Chalmers, Singer, Roth and many others [39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51 ] pointing out that the human brain be definitely the most complex system in the - k n o w n - universe. The more one learns about the brain, the greater the distance seems to grow to the digital computer, although this statement does not match with the expectations of Artificial Intelligence (AI). The "strong" Artificial Intelligence still maintains its postulation that the exponentially growing complexity of the digital computer will automatically create mind and (self-)consciousness in the digital machine, and Ray Kurzweil [52] and Hans Moravec [53] draw dramatic pictures of a future of autonomous intelligent robots and humanoids which might take over control already during this century keeping us humans as pets if we behave. While Moravec is focussed mainly on robotics, he predicts that the digital computer has reached or is not far from the capacity and capabilitiy to simulate living beings and, pointing at recent computer chess matches with Kasparov, even compete with human intelligence. Based on simplistic calculations comparing vertebrate retina performance and robot vision requirements, he states that 100 tera-instructions per second of computer power will be sufficient to match human behaviour by simulation. This computer performance indeed will be soon available, if it is not already today. On the other hand, IT experts as Bill Joy, philosophers and social scientists as well as brain researchers consider Kurzweil's predictions as either horror scenarios or fundamental impossibilities [54]. However, as long as we do not know what it is about human intelligence and its essence and the possibly - if not, as a scientist, to say: certainly - materialistic ground of human mind and consciousness, what they are and how they emerge, there will be no final answer to the mind-body problem a n d - vice versa - no understanding of the human brain, and, thus, it is not possible to make true statements about the "intelligent" evolution of the digital computer as well. Penrose discussed- in "Shadows of the Mind" [38] - four viewpoints on the question whether computing equals thinking and thinking equals computation: 9 Position A: All thinking is computation ("computation" in the sense of Church and Turing, i.e. based on algorithms and representable by Turing machines); in particular, feelings of conscious awareness are evoked merely by executing appropriate computations. ("Strong" AI) 9 Position B: Awareness is a feature of the brain's physical action; and whereas any physical action can be simulated computationally, computational simulation cannot by itself evoke awareness. ("Weak" AI) 9 Position C: Appropriate physical action of the brain evokes awareness, but this physical action cannot be even properly simulated computationally. 9 Position D: Awareness cannot be explained by physical, computational, or any other scientific terms. It seems to be quite natural and consequent that Penrose, as a mathematician and scientist rather than a strong-AI promoter or a theologically locked esoteric, discarded Positions A, B, and D as inadequate answers to the question, and in strengthening Position C he referred to the evolution of physical theories extending quantum mechanics which will favour and finally
10 confirm Position C. In this respect, it is interesting that also Nobel Prize winner John Eccles seems to have pursued during his last scientific activities (still vague) ideas of involving quantum mechanical processes in the description of the functions of the human brain concerning the interaction of mind and matter. Looking back to JvN's first monograph defining and clarifying rigorously the mathematical foundations of quantum mechanics and to his last work comparing the computer and the brain, we must say that his early death caused a great loss also for this challenging and futuredetermining field of research. If we listen to the predictions of the Kurzweils and Moravecs, we may be justified to state that JvN as one last genius of the 20-th century would have activated his exceptional intellectual capabilities and sharp theoretical instruments to shape this field in his strong scientific manner as he did in his relatively short life in other areas, rather than by blue speculations. The science community has indeed good reasons to celebrate his 100-th birthday! REFERENCES
[ 1] [2] [3] [4] [5]
[6]
[7] [8] [9] [10] [11] [12]
[13]
[ 14] [15]
von Neumann, J., Collected Works, Vol. I-VI, Pergamon Press, 1961-1963. Aspray, W., John yon Neumann and the Origins of Modern Computing, MIT Press, 1990. Macrae, N., John yon Neumann, Pantheon Books, 1992. yon Neumann, J., Mathematical Foundations of Quantum Mechanics, Princeton University Press, 1955. von Neumann, J., The Computer and the Brain, 2nd Edition (with a Foreword by Paul. M. Churchland and Patricia S. Churchland), Yale University Press, Yale Nota Bene Book, 2000. Ceruzzi, P. E., The Early Computers of Konrad Zuse, 1935 to 1945, Ann. Hist. Comp. Vol. 3 (1981), No. 3,241-262; Ritchie, D., The Computer Pioneers, Chapter 3, Simon & Schuster, 1986. Hwang, K., Advanced Computer Architecture- Parallelism, Scalability, Programmability, McGraw-Hill, 1993. www.TOP500.org/lists/2003/11. Birkhoff, G., Numerical Fluid Dynamics, SIAM Review Vol. 25 (1983), 1-34. Roache, P. J., Fundamentals of Computational Fluid Dynamics, Hermosa Publishers, 1998. Special Double Issue: Grand Challenges to Computational Science, Future Generation Computer Systems 5 (1989), No. 2/3. Committee on Physical, Mathematical, and Engineering Sciences, Federal Coordinating Council for Science, Engineering, and Technology, Grand Challenges 1993: High Performance Computing and Communications, The FY 1993 U.S. Research and Development Program, Office of Science and Technology Policy, Washington, 1992. Board on Mathematical Sciences of the National Research Council (USA), The David II Report: Renewing U.S. Mathematics - A Plan for the 1990s, in: Notices of the American Mathematical Society, May/June 1990, 542-546; September 1990, 813-837; October 1990, 984-1004 Commission of the European Communities, Report of the EEC Working Group on HighPerformance Computing (Chairman: C. Rubbia), February 1991. Trottenberg, U., et al., Situation und Erfordernisse des wissenschaftlichen H6chstleistungsrechnens in Deutschland- Memorandum zur Initiative High Performance Scientific
11
[16] [17]
[ 18] [ 19] [20] [21 ]
[22] [23]
[24]
[25]
[26]
[27] [28] [29]
[30] [31] [32] [33] [34]
Computing (HPSC), Februar 1992; published in: Informatik-Spektrum Band 15 (1992), H. 4, 218. The Congress of the United States, Congressional Budget Office, Promoting HighPerformance Computing and Communications, Washington, June 1993. High-Performance Computing Applications Requirements Group, High-Performance Computing and Networking, Report, European Union, April 1994; - High-Performance Networking Requirements Group, Report, European Union, April 1994. Bundesministerium f'tir Forschung und Technologie, Initiative zur F6rderung des parallelen H6chstleistungsrechnens in Wissenschafl und Wirtschafl, BMFT, Bonn, Juni 1993. www.llnl.gov/asci/. www.llnl.gov/asci-pathforward/. Smarr, L., Toward the 21-st Century, Comm. ACM Vol. 40 (1997), No. 11, 28-32. - Smith, Ph. L., The NSF Partnerships and the Tradition of U.S. Science and Engineering, Comm. ACM Vol. 40 (1997), No. 11, 35-37. Foster, I., and C. Kesselman (eds.), The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, 1999. Atkins, D. E., et al., Revolutionizing Science and Engineering Through Cyberinfrastructure, National Science Foundation Report, Blue Ribbon Advisory Panel on Cyberinfrastructure, USA, January 2003. The Superperformance Computing Service, Palo Alto Management Group, Inc., Massively Parallel Processing: Computing for the 90s, SPCS Report 25, Second Edition, Mountain View, California, June 1994. Strohmaier, E., et al., The marketplace of high-performance computing, Special Anniversary Issue (G. Joubert et al., eds.), Parallel Computing Vol. 25 (1999), No. 13/14, 15171544. Hossfeld, F., et al., Gekoppelte SMP-Systeme im wissenschafllich-technischen Hochleistungsrechnen - Status und Entwicklungsbedarf- (GoSMP), Analyse im Auflrag des BMBF, F6rderkennzeichen 01 IR 903, Dezember 1999. www. fz-j uelich.de/zam. http ://jumpdoc, fz-j uelich.de/. Hossfeld, F., Teraflops Computing: A Challenge to Parallel Numerics?, in: Zinterhof, P., et al. (eds.), Parallel Computation, Proceedings of the 4-th International ACPC Conference 1999, Salzburg, Austria, pp. 1-12; Feeding Greedies on Meager Roadmaps, in: Bubak, M., et al. (eds.), Proceedings of SGI'2000, Krakow, Poland, pp. 11-22. Semiconductor Industry Association (SIA), International Technology Roadmap for Semiconductors, 2003 Edition (ITRS 2003). Feynman, R. P., Simulating physics with computers, Int. J. Theor. Phys. Vol. 21 (1982), 467. Deutsch, D., Quantum Theory, the Church-Tufing Principle and the universal quantum computer, Proc. Roy. Soc. Lond. A Vol. 400 (1985), 97. Nielsen, M. A., and Chuang, I. L., Quantum Computation and Quantum Information, Cambridge University Press, 2000. Shot, P. W., Algorithms for quantum computation: discrete logarithms and factoring, in: Proceedings 35-th Annual Symposium on Foundation of Computer Science, IEEE Press, 1994; Polynomial-time algorithms for prime factorization and discrete logarithms on a
12 quantum computer, SIAM J. Comp. Vol. 26 (1997), No. 5, 1484-1509. [35] Moret, B. M. E., and Shapiro, H. D., Algorithms from P to NP - Volume 1: Design & Efficiency, The Benjamin/Cummings Publishing Company, 1991. [36] Penrose, R., The Emperor's New Mind, Oxford University Press, 1989. [37] Penrose, R., Shadows of the M i n d - A Search for the Missing Science of Consciousness, Oxford University Press, 1994. [38] Penrose, R., The Large, the Small, and the Human Mind, Cambridge University Press, 1997. [39] Locke, J., An Essay Concerning Human Understanding, Great Books in Philosophy, Prometheus Books, 1995. [40] de La Mettrie, J. O., Machine Man (L'homme plus que machine), in: Cambridge Texts in the History of Philosophy (ed. by Ann Thomson),: La Mettrie, Machine Man and Other Writings, Cambridge University Press, 1996. [41 ] Descartes, R., Discours de la m6thode pour bien conduire sa raison, et chercher la verit6 dans les sciences; Meditationes de prima philosophia, in: Philosophische Schriften in einem Band, Felix Meiner Verlag, 1996. [42] Leibniz, G. W., Nouveaux Essais sur L'Entendement Humain, Livre I-IV, Philosophische Schriften Band III, Wissenschaftliche Buchgesellschaft, 1985; Schriften zur Logik und zur philosophischen Grundlegung von Mathematik und Naturwissenschaft (orig. latin), Philosophische Schriften Band IV, Wissenschaftliche Buchgesellschaft, 1992. [43] Popper, K., and Eccles, J., The Self and Its Brain - An Argument for Interactionism, Springer Verlag, 1977. [44] Chalmers, D. J., The Conscious Mind- In Search of a Fundamental Theory, Oxford University Press, 1996. [45] Edelman, G. M., and Tononi, G., A Universe of Consciousness - How Matter Becomes Imagination, Basic Books, 2000. [46] Pinker, St., How the Mind Works, Penguin Books, 1998. [47] Singer, R., Der Beobachter im Gehirn, Essays zur Hirnforschung, Suhrkamp Taschenbuch Wissenschaft Band 1571, Suhrkamp Verlag, 2002. [48] Roth, G., Das Gehirn und seine Wirklichkeit- Kognitive Neurobiologie und ihre philosophischen Konsequenzen, Suhrkamp Taschenbuch Wissenschaft Band 1275, Suhrkamp Verlag, 1997. [49] Pauen, M., und Roth, G. (Hrsg.), Neurowissenschaften und Philosophie - Eine Einfiihrung, Wilhelm Fink Verlag, 2001. [50] Pauen, M., Grundprobleme der Philosophic des Geistes - Eine Einf'fihrung, Fischer Taschenbuch Verlag, 2001. [51 ] Zoglauer, Th., Geist und Gehirn- Das Leib-Seele-Problem in der aktuellen Diskussion, Vandenhoek & Ruprecht, 1998. [52] Kurzweil, R., The Age of Spiritual Machines; in German: Homo s@piens, Kiepenheuer & Witsch, 1999. [53] Moravec, H. P., Mind Children: The Future of Robots and Human Intelligence, Harvard University Press, 1990; Robot: Mere Machine to Transcendent Mind, Oxford University Press, 2000. [54] Schirrmacher, F. (Hrsg.), Die Darwin A G - Wie Nanotechnologie, Biotechnologie und Computer den neuen Menschen tr~iumen, Kiepenheuer & Witsch, 2001.
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004ElsevierB.V. All rights reserved.
13
So M u c h Data, So Little Time... C. Hansen a, S. Parker a, and C. Gribble ~ aScientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT 84112 USA Massively parallel computers have been around for the past decade. With the advent of such powerful resources, scientific computation rapidly expanded the size of computational domains. With the increased amount of data, visualization software strove to keep pace through the implementation of parallel visualization tools and parallel rendering leveraging the computational resources. Tightly coupled ccNUMA parallel processors with attached graphics adapters have shifted the research of visualization to leverage the more unified memory architecture. Our research at the Scientific Computing and Imaging (SCI) Institute at the university of Utah has focused on innovative, scalable techniques for large-scale 3D visualization. Real-time ray-tracing for isosurfacing has proven to be the most interactive method for large scale scientific data. We have also investigated cluster-based volume rendering leveraging multiple nodes of commodity components. 1. INTRODUCTION In recent years, scalable architectures and algorithms have led to unprecedented growth in computational data. The effectiveness of using such advanced hardware that produces large amounts of high-resolution data will hinge upon the ability of human experts to interact with their data and extract useful information. Needless to say, the interactive analysis of such large, and sometimes unwieldy, data sets has become a computation bottleneck in its own right. To address this challenge, the field of visualization and computer graphics has made significant advances in the area of parallel methods for both scientific visualization and image generation. 2. SCALABLE ISOSURFACING Many applications generate scalar fields p(x, y, z) which can be viewed by displaying isosur.faces where p(x, y, z) = Piso" Ideally, the value for Piso is interactively controlled by the user. When the scalar field is stored as a structured set of point samples, the most common technique for generating a given isosurface is to create an explicit polygonal representation for the surface using a technique such as Marching Cubes [ 1, 10]. This surface is subsequently rendered with attached graphics hardware accelerators such as the SGI Infinite Reality. Marching Cubes can generate an extraordinary number of polygons, which take time to construct and to render. For very large (i.e., greater than several million polygons) surfaces the isosurface extraction and rendering times limit the interactivity. In this paper, we generate images of isosurfaces directly
14 with no intermediate surface representation through the use of ray tracing. Ray tracing for isosurfaces has been used in the past (e.g. [9, 12, 21 ]), but we apply it to very large datasets in an interactive setting. It is well understood that ray tracing is accelerated through two main techniques [ 18]: accelerating or eliminating ray/voxel intersection tests and parallelization. Acceleration is usually accomplished by a combination of spatial subdivision and early ray termination [8, 5, 20]. Ray tracing has been used for volume visualization in many contexts (e.g., [8, 19, 23]). Ray tracing for volume visualization naturally lends itself towards parallel implementations [11, 14]. The computation for each pixel is independent of all other pixels, and the data structures used for casting rays are usually read-only. These properties have resulted in many parallel implementations. A variety of techniques have been used to make such systems parallel, and many successful systems have been built. These techniques are surveyed by Whitman [24]. Ray tracing naturally lends itself towards parallel implementations. The computation for each pixel is independent of all other pixels, and the data structures used for casting rays are usually read-only. Simply implementations work, but achieving scalability on large parallel resources requires careful attention to synchronization costs and resource assignment. To reduce synchronization overhead we can assign groups of rays to each processor. The larger these groups are, the less synchronization is required. However, as they become larger, more time is potentially lost due to poor load balancing because all processors must wait for the last job of the frame to finish before starting the next frame. We address this through a load balancing scheme that uses a static set of variable size jobs that are dispatched in a queue where jobs linearly decrease in size. The implementation of the work queue assignment uses the hardware fetch and op counters on the Origin architecture. This allows efficient access to the central work queue resource. This approach to dividing the work between processors seems to scale very well. Rendering a scene with a large memory footprint (rendering of isosurfaces from the visible female dataset [ 17]) uses only 2.1 to 8.4 Mb/s of main memory bandwidth. These statistics were gathered using the SGI perfex utility, benchmarked with 60 processors.
2.1. Rectilinear Isosurfacing Our algorithm has three phases: traversing a ray through cells which do not contain an isosurface, analytically computing the isosurface when intersecting a voxel containing the isosurface, and shading the resulting intersection point [ 16]. This process is repeated for each pixel on the screen. Since each ray is independent, parallelization is straightforward. An additional benefit is that adding incremental features to the rendering has only incremental cost. For example, if one is visualizing multiple isosurfaces with some of them rendered transparently, the correct compositing order is guaranteed since we traverse the volume in a front-to-back order along the rays. Additional shading techniques, such as shadows and specular reflection, can easily be incorporated for enhanced visual cues. Another benefit is the ability to exploit texture maps which are much larger than texture memory (typically 32MB to 256MB). 3. PARALLEL ISOSURFACING RESULTS We applied the ray tracing isosurface extraction to interactively visualize the Visible Woman dataset. The Visible Woman dataset is available through the National Library of Medicine as part of its Visible Human Project [ 15]. We used the computed tomography (CT) data which
15 was acquired in lmm slices with varying in-slice resolution. This data is composed of 1734 slices of 512x512 images at 16 bits. The complete dataset is 910MBytes. Rather than downsample the data with a loss of resolution, we utilize the full resolution data in our experiments. As previously described, our algorithm has three phases: traversing a ray through cells that do not contain an isosurface, analytically computing the isosurface when intersecting a voxel containing the isosurface, and shading the resulting intersection point.
# processors 2 3 4 6 8 12 16 24 32 48 64 96 124
0.427/1.00 0.84/1.97 1.26/2.94 1.67/3.91 2.45/5.73 3.20/7.50 4.81 /11.26 6.38/14.93 9.54/22.33 12.65/29.61 18.85/44.13 24.73/57.90 35.38/82.82 43.06/100.79
Frame rate/Speedup 0.084 /1.00 0.155 /1.00 0.17 /1.99 0.31 /2.00 0.25 /2.95 0.46 /2.96 0.33 /3.96 0.62 /3.97 0.50 /5.97 0.93 /5.96 0.67 /7.94 1.23 /7.93 1.00 /11.89 1.84 /11.88 1.33 /15.84 2.45 /15.80 1.98 /23.54 3.65 /23.49 2.63 /31.38 4.88 /31.47 3.92 /46.72 7.30 /47.02 5.18 /61.78 9.64 /62.14 7.67 /91.38 14.28 /92.02 9.73 /115.88 18.17 /117.08
0.304 /1.00 0.60 /1.98 0.89 /2.93 1.19 /3.92 1.76 /5.77 2.32 /7.61 3.44 /11.30 4.59 /15.08 6.84 /22.48 9.12 /29.96 13.52 /44.39 17.72 /58.19 25.04 /82.23 30.28 /99.45
0.568 /1.00 1.13 /1.98 1.68 /2.96 2.24 /3.94 3.29 /5.80 4.36 /7.67 6.51 /11.47 8.64 /15.21 12.92 /22.76 17.09 /30.10 25.27 /44.50 32.25 /56.80 45.50 /80.14 57.70 /101.63
~.20 o
7
~15
m
o
~10 o
ffl
~
JO ~i~
j',i!
'
'
'
'
'i'
'
I
'
iFrame N~mber~(tihe) I
Figure l. Variation in framerate as the viewpoint and isovalue changes.
~a
I
i
I
i
I
16 The interactivity of our system allows exploration of both the data by interactively changing the isovalue or viewpoint. For example, one could view the entire skeleton and interactively zoom in and modify the isovalue to examine the detail in the toes all at about ten FPS. The variation in framerate is shown in Figure 1. 4. INTERACTIVE VOLUME RENDERING WITH SIMIAN Simian is a scientific visualization tool that utilizes the texture processing capabilities of consumer graphics accelerators to produce direct volume rendered images of scientific datasets. The true power of Simian is its rich user interface. Simian employs direct manipulation widgets for defining multi-dimensional transfer functions; for probing, clipping, and classifying the data; and for shading and coloring the resulting visualization [6]. A complete discussion of direct volume rendering using commodity graphics hardware is given in [2]. For more on using multi-dimensional transfer functions in interactive volume visualization, see [7]. All of the direct manipulation widgets provided by Simian are described thoroughly in [6]. The size of a volumetric dataset that Simian can visualize interactively is largely dependent on the size of the texture memory provided by the local graphics hardware. For typical commodity graphics accelerators, the size of this memory ranges from 32MB to 256MB. However, even small scientific datasets can consume hundreds of megabytes, and these datasets are rapidly growing larger as time progresses. Although Simian provides mechanisms for swapping smaller portions of a large dataset between the available texture memory and the system's main memory (a process that is similar to virtual memory paging), performance drops significantly and interactivity disappears. Moreover, because the size of the texture memory on commodity graphics hardware is not growing as quickly as the size of scientific datasets, using the graphics accelerators of many nodes in a cluster-based system is necessary to interactively visualize large-scale datasets. Naturally, cluster-based visualization introduces many challenges that are of little or no concern when rendering a dataset on a single node, and there are many techniques for dealing with the problems that arise. Our goal was to create an interactive volume rendering tool that provides a full-featured interface for navigating and visualizing large-scale scientific datasets. Using Simian, we examine two approaches to cluster-based interactive volume rendering: (1) a "cluster-aware" version of the application that makes explicit use of remote nodes through a message-passing interface (MPI), and (2) the unmodified application running atop the Chromium clustered rendering framework. 5. CHROMIUM CLUSTERED RENDERING F R A M E W O R K Chromium is a system for manipulating streams of OpenGL graphics commands on commodity-based visualization clusters [3]. For Linux-based systems, Chromium is implemented as a set of shared-object libraries that export a large subset of the OpenGL application programming interface (API). Extensions for parallel synchronization are also included [4]. Chromium's crappfaker libraries operate as the client-side stub in a simple client-server model. The stub intercepts graphics calls made by an OpenGL application, filters them through a userdefined chain of stream processing units (SPUs) that may modify the command stream, and finally redirects the stream to designated rendering servers. The Chromium rendering servers,
17 or crservers, process the OpenGL commands using the locally available graphics hardware, the results of which may be delivered to a tiled display wall, returned to the client for further processing, or composited and displayed using specialized image composition hardware such as Lightning-2 [22]. Depending on the particular system configuration, it is possible to implement a wide variety of parallel rendering systems, including the common sort-first and sort-last architectures [13]. For the details of the Chromium clustered rendering framework, see [3]. In principle, Chromium provides a very simple mechanism for hiding the distributed nature of clustered rendering from OpenGL applications. By simply loading the Chromium libraries rather than the system's native OpenGL implementation, graphics commands can be processed by remote hardware without modifying the calling application. However, for the Simian volume rendering application, Chromium does not currently provide features that sufficiently mask the underlying distributed operation and still enable the application to realize the extended functionality that we seek. OpenGL applications may still require significant modifications to effectively utilize a cluster's resources, even when employing the Chromium framework. It was necessary to implement some OpenGL functionality that was not supported by Chromium. First, we implemented the subset of OpenGL calls related to 3D textures, including glTexImage3D, which is the workhorse of the Simian volume rendering application. In addition, we also added limited support for the NVIDIA GL NV t e x t u r e s h a d e r and G L _ N ' V _ t e x t u r e s h a d e r 2 extensions, implementing only those particular features of the extensions that are explicitly used by Simian. 6. CLUSTER-BASED VOLUME RENDERING RESULTS We experimented with two C-SAFE fire-spread datasets simulating a heptane pool fire, h300_0075 and h300_0130, that were produced by LES codes. Both of the original datasets were 302x302x302 byte volumes storing scalar values as u n s i g n e d c h a r s . Simian is capable of rendering volumes using gradient and Hessian components in addition to scalar values. Therefore, each of the fire-spread datasets was pre-processed to included these additional components and then padded to a power of 2, resulting in two 512x512x512x3 byte volumes that store the value, gradient, and Hessian components ("vgh") as u n s i g n e d c h a r s . 1 We first attempted to visualize both the original and vgh versions of each fire-spread dataset using the stand-alone version of Simian and a fairly powerful workstation. This workstation was equipped with an Intel Xeon 2.66GHz processor, 1GB of physical memory, and an NVIDIA GeForce4 Quadro4 900 XGL graphics accelerator with 128MB of texture memory. On this machine, Simian was able to load and render the original datasets using the swapping mechanisms inherent in OpenGL. As expected, the need to swap severely penalized the application's performance, resulting in rates of 0.55 frames per second. However, the OpenGL drivers could not properly handle the large-scale vgh datasets, crashing the application as result. Having firmly established the need for a cluster-based approach, we tested both the clusteraware version and the Simian/Chromium combination with the vgh datasets using 8, 16, and 32 of the C-SAFE cluster nodes. The results are summarized in Table 1. Note that, with 8 and 16 rendering nodes, the vgh subvolumes distributed by the clusteraware version are 256x256x256 bytes and 256x256x128 bytes, respectively. These subvol1Forthe remainder of this discussion, it is assumedthat the numberof bytes consumedby a raw volumetric dataset is the given size multiplied by three, accounting for each of the three, 1-bytecomponentsin a vgh dataset.
18 Rendering Approach
Number of Nodes
Cluster-aware Simian
8 16
32 Simian/Chromium combination
8 16
h300_0075 h300_0130
vgh dataset
vgh dataset
0.52
0.58 2.15 3.44 0.05 0.04 0.02
1.47 2.87 0.07 0.05 0.03
32 Table 1 Average frame rates (in frames per second) using various cluster configurations
umes exceed the maximum "no-swap" volume size permitted by the cluster's graphics hardware (128x128x256 bytes), so in either of these configurations, even the cluster-aware Simian must invoke its texture swapping facilities. However, with 16 rendering nodes, the swapping is much less frequent, resulting in reasonably interactive frame rates. With 32 rendering nodes, the subvolume size is reduced to 128x128x256 bytes, so no texture swapping occurs and interactive frame rates are restored. Chromium is not able to distribute subvolumes among the rendering nodes. As a result, Simian must rely on its texture swapping mechanism. When the application calls for a new block to be swapped into texture memory, Chromium must transmit the block to the appropriate rendering nodes. The resulting delays impose severe performance penalties that grow with the number of rendering nodes. This behavior is reflected in the low frame rates given in Table 1. 7. CONCLUSIONS The architecture of the parallel machine plays an important role in the success of visualization techniques. In parallel isosurfacing since any processor can randomly access the entire dataset, the dataset must be available to each processor. Nonetheless, there is fairly high locality in the dataset for any particular processor. As a result, a shared memory or distributed shared memory machine, such as the SGI Origin, is ideally suited for this application. The load balancing mechanism also requires a fine-grained low-latency communication mechanism for synchronizing work assignments and returning completed image tiles. With an attached graphics engine, we can display images at high frame rates without network bottlenecks. We have implemented a similar technique on a distributed memory machine that proved challenging. Frame-rates were, of course, lower and scalability was reduced. We have shown that ray tracing can be a practical alternative to explicit isosurface extraction for very large datasets. As data sets get larger, and as general purpose processing hardware becomes more powerful, we expect this to become a very attractive method for visualizing large scale scalar data both in terms of speed and rendering accuracy. For cluster-based volume rendering, we have demonstrated that a cluster-aware MPI version performs far better than simply replacing the underlying shared graphics library with a parallelized library. There are several advantages of developing cluster-aware visualization tools. First, these tools can exploit application-specific knowledge to reduce overhead and enhance performance and efficiency when running in a clustered environment. Second, cluster-aware applications that are built upon open standards such as MPI are readily ported to a wide variety
19 of hardware and software platforms. While frameworks that mask the clustered environment may provide certain advantages, using standard interfaces allows an application's components to be reused or combined in new, more flexible ways. Third, cluster-aware applications are not dependent upon the functionality provided by a clustered rendering package. Finally, because they do not access remote resources via a lower-level framework, cluster-aware applications can exploit the capabilities of the underlying system directly.
REFERENCES
[1]
B. Wyvill G. Wyvill, C. McPheeters. Data structures for soft objects. The Visual Com-
puter, 2:227-234, 1986. [2] [3]
[4] [5] [6]
[7]
[8] [9] [ 10]
[11 ] [ 12] [13] [14]
[15]
[16]
M. Hadwiger et al. High-quality volume graphics on consumer pc hardware. IEEE Visualization 2002 Course Notes. G. Humphreys, Mike Houston, Yi-Ren Ng, Randall Frank, Sean Ahem Peter Kirchner, and James T. Klosowski. Chromium: A stream-processing framework for interactive graphics on clusters. In Proceedings of SIGGRAPH, 2002. H. Igehy, G. Stoll, and P. Hanrahan. The design of a parallel graphics interface. In SIGGRAPH Computer Graphics, 1998. Arie Kaufman. Volume Visualization. IEEE CS Press, 1991. J. Kniss, G. Kindlmann, and C. Hansen. Interactive volume rendering using multidimensional transfer functions and direct manipulation widgets. In Proceedings oflEEE Visualization, 2001. J. Kniss, G. Kindlmann, and C. Hansen. Multi-dimensional transfer functions for interactive volume rendering. IEEE Transactions on Visualization and Computer Graphics, pages 270-285, July 2002. Mark Levoy. Display of surfaces from volume data. IEEE Computer Graphics & Applications, 8(3):29-37, 1988. Chyi-Cheng Lin and Yu-Tai Ching. An efficient volume-rendering algorithm with an analytic approach. The Visual Computer, 12(10):515-526, 1996. William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. Computer Graphics, 21 (4): 163-169, July 1987. ACM Siggraph '87 Conference Proceedings. K.L. Ma, J.S. Painter, C.D. Hansen, and M.F. Krogh. Parallel Volume Rendering using Binary-Swap Compositing. IEEE Comput. Graphics and Appl., 14(4):59-68, July 1993. Stephen Marschner and Richard Lobb. An evaluation of reconstruction filters for volume rendering. In Proceedings of Visualization '94, pages 100-107, October 1994. S. Molnar, M. Cox, D. Ellsworth, and H. Fuchs. A sorting classification of parallel rendering. 1EEE Computer Graphics and Applications, July 1994. Michael J. Muuss. Rt and remrt - shared memory parllel and network distributed raytracing programs. In USENIX: Proceedings of the Fourth Computer Graphics Workshop, October 1987. National Library of Medicine (U.S.) Board of Regents. Electronic imaging: Report of the board of regents, u.s. department of health and human services, public health service, national institutes of health. NIH Publication 90-2197, 1990. Steven Parker, Michael Parker, Yarden Livnat, Peter-Pike Sloan, Charles Hansen, and
20
[ 17]
[ 18] [ 19] [20] [21 ] [22]
[23] [24]
Peter Shirley. Interactive Ray Tracing for Volume Visualization. IEEE Transactions on Visualization and Computer Graphics, 5(3):238-250, July 1999. Steven Parker, Peter Shirley, Yarden Livnat, Charles Hansen, and Peter-Pike Sloan. Interactive ray tracing for isosurface rendering. In Proceedings of Visualization '98, October 1998. E. Reinhard, A.G. Chalmers, and F.W. Jansen. Overview of parallel photo-realistic graphics. In Eurographics "98, 1998. Paolo Sabella. A rendering algorithm for visualizing 3d scalar fields. Computer Graphics, 22(4):51-58, July 1988. ACM Siggraph '88 Conference Proceedings. Lisa Sobierajski and Arie Kaufman. Volumetric Ray Tracing. 1994 Workshop on Volume Visualization, pages 11-18, October 1994. Milos Sramek. Fast surface rendering from raster data by voxel traversal using chessboard distance. In Proceedings of Visualization '94, pages 188-195, October 1994. G. Stoll, M. Eldridge, D. Patterson, A. Webb, S. Berman, R. Levy, C. Caywood, M. Taveira, S. Hunt, and P. Hanrahan. Lighming-2: A high-performance display subsystem for PC clusters. In SIGGRAPH Computer Graphics, 2001. Craig Upson and Micheal Keeler. V-buffer: Visible volume rendering. Computer Graphics, 22(4):59-64, July 1988. ACM Siggraph '88 Conference Proceedings. Scott Whitman. A Survey of Parallel Algorithms for Graphics and Visualization. In High Performance Computing for Computer Graphics and Visualization, pages 3-22, 1995. Swansea, July 3-4.
Software Technology
This Page Intentionally Left Blank
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
23
On Compiler Support for Mixed Task and Data Parallelism T. Rauber ~, R. Reilein b, and G. Rfingerb a Department of Mathematics, Physics, and Computer Science University of Bayreuth E-mail: rauber@uni-bayreuth, de b Department of Computer Science Chemnitz University of Technology E-mail: {reilein, ruenger}@cs, tu-chemnitz, de The combination of task and data parallelism can lead to an improvement of speedup and scalability for parallel applications on distributed memory machines. To support a systematic design of mixed task and data parallel programs the TwoL model has been introduced. A key feature of this model is the development support for applications using multiprocessor tasks on top of data parallel modules. In this paper we discuss implementation issues of the TwoL model as an open framework. We focus on the design of the framework and its internal algorithms and data structures. As examples fast parallel matrix multiplication algorithms are presented to illustrate the applicability of our approach. 1. INTRODUCTION Parallel applications in the area of scientific computing are often designed in a data parallel SPMD (Single Program Multiple Data) style based on the MPI standard. The advantage of this method is a clear programming model but on large parallel platforms or cluster systems the speedup and scalability can be limited especially when collective communication operations are used frequently. The combination of task and data parallelism can improve the scalability of many applications but requires a more intricate program development. The adaption of complex program code to the characteristics of a specific parallel machine may be quite time consuming and often results in a code structure which causes a high reprogramming effort when porting the software to another parallel system. To support the systematic development of mixed task and data parallel programs the TwoL model has been introduced. The model provides a stepwise development process which is subdivided into several phases. Applications are hierarchically composed of predefined or user-supplied basic data parallel modules. The development starts with a specification of the parallelism inherent in the algorithm to be implemented. The specification is transformed stepwise into a coordination program by applying scheduling, static load balancing, and data distribution algorithms. During this transformation process the code is adapted to the characteristics of a specific parallel system. In this paper we present an implementation of the TwoL model as an open compiler framework (TwoL-OF). The open framework implements the core concepts of the TwoL model and additionally provides several intermediate program representations resulting within the trans-
24 formation process. The intermediate representations are produced in specific TwoL-OF formats to provide access interfaces for compiler users. The advantage is that the framework can be used by both, application programmers and algorithm developers. Application programmers can use the framework as transformation tool by providing a specification of the problem which is transformed into an efficient parallel program. Developers can test new scheduling and load balancing techniques by exploiting the access interfaces of the intermediate formats. The remaining paper is structured as follows. Section 2 gives a short overview of the TwoL model. In Section 3 the TwoL-OF compiler is introduced and the compilation process is illustrated. Section 4 presents first results with fast matrix multiplication algorithms and Section 5 concludes. 2. THE TwoL MODEL The TwoL (Two Level) model has been developed to exploit the combination of task and data parallelism [6, 7]. It defines a transformation process from a specification of the parallelism inherent in an algorithm into a coordination program for a specific parallel system. The parallelism is exploited on two levels, an upper task parallel level consisting of hierarchically structured multiprocessor tasks and a lower data parallel level. On the lower level the user provides module specifications and function implementations which declare and realize multiprocessor tasks. On the upper level consecutive transformation steps generate a coordination program from the specification of potential task parallelism. Figure 1 illustrates the derivation of the coordination program. non-executable specification coordination program ~...@.,,,.,~ stepwise derivation --~ double *a,*b; ~
@/Q~@
@ ; ~ ~
.Transformation
9Annotation
9 Selection
Level ~'~ ~
MPI Comm comml,comm2; MPI--Comm_split(&comml,color pid,&comm2) ; if (gpid==0) { ...
~ typea distrib b var x:a ,..
Declaration of types,
m o d u l e s and variables,
Parallel library and/~
user-defined functions.
I dmm dmm-fox std [
II
dmm can
Figure 1. Derivation of the parallel coordination program. The specification program declares multiprocessor tasks as modules and defines their data and control dependencies. It uses a special type concept to declare data types and data distribution types. The transformation applies scheduling, load balancing, and data distribution techniques and the decisions made are included as annotations. The final coordination program contains all information about execution order, processor group sizes, and data distributions. The entire transformation process is accompanied by a parallel cost model to guide the transformations and to obtain an efficient parallel program for a specific parallel machine. 3. TwoL-OF C O M P I L E R STRUCTURE AND IMPLEMENTATION The TwoL open framework compiler implements the core concepts of the TwoL model, supports the guided development of mixed task and data parallel programs, and the porting to
25 different parallel machines. The emphasis lays on fast and easy coding support utilizing the activation of basic modules written in different imperative languages especially C or Fortran. It offers specific interfaces to control the transformation process and to manually revise decisions made by the transformation algorithms. Therefore the intermediate code representation is explicitly stored and open for transformations and annotations.
3.1. Short overview The compilation process starts with a specification language file describing the inherent parallelism of an algorithm. The basic components of a specification are modules which can either be declared (basic modules) or defined (coordination modules). For basic modules a module name and three different types for each parameter to be passed have to be declared. Each module parameter has a data type which corresponds to a data type in the high level target language and a data distribution type which declares how the data are distributed over a processor group and which is used by redistribution functions to establish the data distribution required before starting the execution of a basic module. The third type is the I/O access mode which defines the kind of access to the parameters. Basic modules can be implemented in C or Fortran and are linked to the generated program code in the final compilation step. Coordination modules have to be declared before they can be defined by a coordinating expression which consists of module calls, data dependence operators, and control statements. These expressions are translated into syntax trees which are extended and transformed in the following compilation phases. Based on the coordinating expressions the coordination program is generated by the compiler framework as C code augmented with MPI functions and calls to a data redistribution library interface. The resulting code can be compiled and linked with the basic module implementations using a standard C compiler. The TwoL-OF compiler consists of two consecutive compiler stages. The first stage reads a specification program, generates the intermediate code representation, and outputs two control files for the next stage, one for type definitions and a second file to control tree transformations and annotations. The intermediate code representation comprises different kinds of symbol tables, specification syntax trees for the coordinating expressions, and parallel data dependence graphs to attach data flow information to the syntax trees. Between both stages the control files are augmented with additional specifications supplied by automatic tools or by the user. In the second stage after the intermediate code is read, the type definition and transformation control files are processed. The contents are used to amend symbol table entries and to transform the specification syntax trees into coordination syntax trees. C code with calls to MPI and a data redistribution library is generated after compiler passes to update the data flow information and to introduce variable copying and data redistribution. The compilation process is accompanied by the output of internal data structures as program files which can be postprocessed using the graphviz package by AT&T to generate a graphical visualization. In the next subsections we introduce the specification language and give an example for a parallel specification to illustrate the compilation process.
3.2. Specification language for multiproeessor task programs The specification language allows the expression of data dependencies between module calls by two operators, the ~--operator for data dependence and the II-operator for data independence. Both operators can be used in infix form or as k-ary operator. Furthermore there are several statements to control the program flow especially loops and conditions. The following grammar
26 defines the essential parts for expressing data dependencies and control flow. cexpr stat cexpr_list
--+ I
cexpr >- cexpr I cexpr II cexpr ] II (cexpr_list) I >- (cexpr_list) stat ( cond ) { cexpr } I call loop ] while I if cexpr_list, cexpr ] cexpr
A coordination module defines one coordinating expression (cexpr) which is recursively built up of sub-expressions. To illustrate the compilation process the standard four block matrix multiplication is used here as an example. The algorithm is defined by the following equation:
( all a12 ) ( b l l x hi2 ) _ _ a21 a22 b21 b22
( a 1 1 • b11nt-q12 x b21 all >
To specify the inherent parallelism a specification program containing the following definition of a coordination module (tram) has to be supplied by the user. The declarations of the parameter types for the basic modules used are omitted here. c o m p m o d tmm (a, b, c, n) { m p a r t (a, all, a12, a21, a22, n) >- m p a r t ( b , b l l , b 1 2 , b 2 1 , b 2 2 , n ) ]l (dram(all, bll, cll, n/2) , dram (a12, b21, dll, n/2) , dram (all, b12, c12, n/2) , dmm (a12, b22, d12, n/2) , dmm (a21, bll, c21, n/2) , dmm (a22, b21, d21, n/2) , dmm(a21,b12,c22,n/2) , dmm(a22,b22,d22,n/2)) ~- II ( m m a ( c l l , d l l , c l l , n / 2 ) , m m a ( c l 2 , d l 2 , c l 2 , n / 2 ) , mma(c21,d21,c21,n/2) , mma(c22,d22,c22,n/2)) >- rejoin (cll, c12, c21, c22, c, n)
The module m p a r t subdivides a matrix into four quadratic submatrices and mj o in reverses this process. Addition and multiplication of submatrices are performed by mma and dram, respectively. This specification is processed by the first compiler stage and the coordinating expression is stored as a specification syntax tree with a linked parallel data dependence graph for expressing data flow. Figure 2 illustrates the parallel data dependence graph for the specification example as output from the framework. Together with the intermediate representation which is stored as binary file by the first stage two text files for type definitions and to control transformations are generated. The type definition file is used to specify a C or Fortran data type for the abstract types used in the specification program. If memory allocation is required for a specific data type the corresponding statements can be supplied. Within the second file the user can control transformations and annotations of the specification syntax trees. In its unmodified form the transformation control file expresses the maximum degree of parallelism specified. The following code fragment shows a part of the transformation control file for the block matrix multiplication example: par[ID34] [4] { [0] :ID30 # m m a ( c l l , d l l , c l l , n / 2 ) [i] :ID31 # m m a ( c l 2 , d l 2 , c l 2 , n / 2 ) [2] :ID32 # m m a ( c 2 1 , d 2 1 , c 2 1 , n / 2 ) [3] :ID33 # m m a ( c 2 2 , d 2 2 , c 2 2 , n / 2 )
}
27
m
\
\
I
\
/
Figure 2. Part of the Parallel data dependence graph for block matrix multiplication. The labels enclosed in brackets and prefixed with 'ID' are unique identifiers for the nodes of the corresponding specification syntax tree. They are also displayed in the visualization of the trees and data dependence graphs. The second parameter of the p a r statement specifies the number of processor groups or multiprocessor tasks, respectively. The t a s k statement groups tasks together to be executed consecutively on a specific processor group given as first parameter. Tasks would be separated by commas in that case. The number sign (#) marks the start of a comment which are generated here for the sake of clarity. The following subsection presents the internal structure of the compiler framework in more detail. 3.3. Algorithms and data structures The first compiler stage of the TwoL open framework comprises a syntax-oriented translation scheme to built up specification syntax trees, the construction of symbol tables for data types, data distribution types, variables, and modules, and the creation of parallel data dependence graphs as representations of the data flow. The second stage updates the symbol tables and transforms the syntax trees using information in specific control files provided by automatic tools or directly by the user. After the insertion of the data redistribution operations required the final coordination code is generated. A specification program which contains definitions of coordination modules by coordinating expressions is translated into syntax trees which are realized as C data structures. A tree node comprises a type, a unique identifier, pointers to child nodes, and a union to maintain type specific data. Leaf nodes, which represent module calls, have an additional pointer to a data structure for the parallel data dependence graph. The nodes of this graph represent parameters of module calls and a vertex of the graph defines a true data dependence between two accesses to a specific variable. A parallel data dependence graph is constructed by a depth first sweep over the corresponding specification syntax tree. If the algorithm detects an input parameter reading a particular variable it creates a new node for the parallel data dependence graph and traverses backwards a list of previously visited nodes to find an appropriate output for that variable. The search procedure takes the data dependence information stored in the specification syntax tree into account. When an output is found which does not already have a graph node a new one is constructed and linked with the current one by pointers. These links define a true data dependency between two accesses to a variable. Specification syntax trees are also used to generate a text representation of transformation
28 control structures that is output to the transformation control file. For each ll-operator node the file contains a specific program structure, see for example the control file in Section 3.2. According to these structures the following transformation of the syntax trees is applied by the translation scheme of the second compiler stage. Each II-operator node is replaced by a fork node that is used to insert activations of multiprocessor tasks. The data independence expressed by the tl-operator is then replaced by a fork into multiprocessor tasks. A fork node stores the number of processor groups and has new task nodes as children to define multiprocessor tasks. The former children of the II-operator node have been linked to the particular task nodes representing the processor group which will execute them in the final code. After this transformation the syntax tree defines the parallel execution order of a coordination function. The second compiler stage maintains variables and introduces copies if a variable is accessed in parallel. Based on the information contained in the parallel data dependence graphs data redistribution operations are inserted in the trees as additional nodes. The resulting trees comprise all information to generate the final coordination code. Figure 3 illustrates the internal structure of the TwoL-OF compiler and the main compilation steps. l a Parsing the specification program to built up specification syntax trees and symbol tables.
Specification language program
ISymbol tables Specification syntax tree modules ~,~ data t y p e s Specification syntax tree with linked distribution types paralleldata dependence graph I* variables Type definition Intermediatecode Transformationcontrol program representation program
I
.,.
Updated symbol tables
.....
|
......
Coordination syntax tree with linked paralleldata dependence graph ( ~ C code coordination function
l b Construction of the parallel data dependence graphs. 1c Generation of control files and write out of intermediate code. 2a Processing of intermediate code and parsing of transformation control and type definition files to transform syntax trees and to update symbol tables entries. 2b Maintenance of variable copies and data exchange operations. 2c Generation of coordination program as C code.
Figure 3. TwoL-OF structure (left) and main compilation steps (fight).
4. EXPERIMENTS
To evaluate the TwoL-OF compiler two versions of four block matrix multiplication C = A x B have been implemented, the standard algorithm with 8 matrix multiplies and 4 additions and the Strassen scheme with 7 multiplies and 18 additions [8]. The inner multiplications of two block matrices are done using the Fox algorithm. To determine the overhead both schemes have been implemented by hand and as specification program supplemented with appropriate basic modules. All four versions have been measured on a Linux cluster (CLiC) built of 800 MHz
29 Pentium III PCs interconnected with switched Fast Ethernet. Figure 4 shows the overhead of the compiler-generated version which is given as the ratio between the runtime and the runtime of the hand-tuned version.
Overhead factor for block matrix multiplication on CLiC Matrix size 2400 , 3600 ........... 4800 ............
1.15 1.1 1.05
Overhead factor for Strassen matrix multiplication on CLiC Matrix size 2400 , 3600 ........... 4800 ............
1.15 1.1
7_. . . . . . . . . . . . . . . . . . . . . . . . . . . .
16 36
64
100
5-~_-~..........................
144
196
256
processor number
"-,..
1.05
16 36
64
100
144
196
256
processor number
Figure 4. Overhead factors for the standard scheme (left) and the Strassen scheme (fight) on CLiC. The results show a maximum overhead factor of 1.12 for this examples. This corresponds to first experiences made with other codes and applications. It has been observed that handtuned programs usually require less variable copies and can organize data redistribution more compact. But on the other hand they require a higher development effort and cannot benefit from the automatic detection and insertion of data redistribution operations.
5. RELATED W O R K AND CONCLUSION Many research groups have developed compiler tools and parallel environments which can integrate task and data parallelism. Most of them extent Fortran or High Performance Fortran (HPF) with task parallel constructs [3, 9, 2], see [1] for a good overview. Newer approaches include the coordination of data parallel programs written in skeleton-like or data parallel languages [5], parallel Haskell programming [10], or task and data parallel programming in Java
[4].
In this paper the TwoL open framework compiler as an implementation of the core concepts of the TwoL model was introduced as an open platform for algorithm and application developers. We gave a closer look on the structure of the framework and presented early compilation results. The average overhead factor of compiler generated programs is reasonably small and caused by a higher amount of variable copies required in combination with a slight communication overhead. This factor is expected to be reduced by the development of optimization algorithms for message-packing and for an efficient placement of data redistribution operations. Future work will include the definition of an interface to the binary intermediate representation and support for runtime prediction.
30 REFERENCES
[1] [2]
[3] [4] [5] [6]
[7] [8] [9] [ 10]
H. Bal and M. Haines. Approaches for Integrating Task and Data Parallelism. IEEE Concurrency, 6(3): 74-84, 1998. P. Banerjee, J. Chandy, M. Gupta, E. Hodge, J. Holm, A. Lain, D. Palermo, S. Ramaswamy, and E. Su. The Paradigm Compiler for Distributed-Memory Multicomputers. IEEE Computers, 28(10):37-47, 1995. I. Foster and K.M. Chandy. Fortran M: A Language for Modular Parallel Programming. Journal of Parallel and Distributed Computing, 25(1):24-35, 1995. F. Kuijlman, H.J. Sips, C. van Reeuwijk, and W.J.A. Denissen. A Unified Compiler Framework for Work and Data Placement. In Proc. ofASC12002, pages 109-115, 2002. S. Pelagatti and D.S. Skillicorn. Coordinating Programs in the Network of Tasks Model. Journal of Systems Integration, 10(2): 107-126, 2001. T. Rauber and G. R/inger. The Compiler TwoL for the Design of Parallel Implementations. In In Proc. of 4th International Conference on Parallel Architecture and Compilations Techniques (PACT'96), pages 292-301. IEEE Computer Society Press, 1996. T. Rauber and G. Riinger. A Transformation Approach to Derive Efficient Parallel Implementations. IEEE Transactions on Software Engineering, 26(4):315-339, 2000. V. Strassen. Gaussian Optimization is not Optimal. Numerische Mathematik, (13):354356, 1969. J. Subhlok and B. Yang. A New Model for Integrated Nested Task and Data Parallel Programming. In Proc. ofACM SIGPLANPPoPP 97, pages 1-12, 1997. P.W. Trinder, H-W. Loidl, and R.F. Pointon. Parallel and Distributed Haskells. Journal of Functional Programming, 2(4/5):469-510, 2002.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
31
Distributed Process Networks - Using Half FIFO Queues in CORBA A. Amar a*, R Boulet a, J.-L. Dekeyser a, and F. Theeuwen b aLaboratoire d'Informatique Fondamentale de Lille, Lille 1 Cit6 Scientifique, Bat. M3, 59655 Villeneuve d'Ascq cedex, France b Philips ED & T / Synthesis WAY 3.13, Prof. Holstlaan 4 5656 AA Eindhoven, The Netherlands Process networks are networks of sequential processes connected by channels behaving like FIFO queues. These are used in signal and image processing applications that need to run in bounded memory for infinitely long periods of time dealing with possibly infinite streams of data. This paper is about a distributed implementation of this computation model. We present the implementation of a distributed process network by using distributed FIFOs to build the distributed application. The platform used to support this is the CORBA middleware. 1. INTRODUCTION Kahn Process Networks [6, 7] are well adapted to model many parallel applications, specially dataflow applications (signal processing, image processing). In this model, processes communicate only via unbounded first-in first-out (FIFO) queues. This model has a dataflow flavor and can express a high degree of concurrency which makes it particularly well suited to model intensive signal processing applications or complex scientific applications. This model makes no assumption on the computation load of the different processes and thus is heterogeneous by nature. Distributed architectures provide an attractive alternative to supercomputers in terms of computation power and cost to execute such complex and computation intensive applications. The two main weak points of these architectures are their communication capabilities (relatively high latency) and often the heterogeneity of their hardware. We present in this paper a distributed implementation of the process network model on heterogeneous distributed hardware. The different processing power of the connected computers is a good support for the different computation needs of the networked processes. We have chosen to use the Common Object Request Broker Architecture (CORBA) [9] middleware to handle the communications for its interoperability properties. Indeed, each process of the process network can be written in a different language and run on a different hardware, provided that these are supported by the chosen Object Request Broker (ORB). In addition of the heterogeneity, our implementation presents the following characteristics: *This work has been supported by the ITEA 99038 project, Sophocles.
32 9 automation of data transfer between distributed processes 9 dynamic and interactive linking of the processes to form the data flow 9 hybrid data-driven, demand-driven data transfer protocol, with thresholds for load balancing 9 the implementation was carried out such as to enable a distributed or local execution without any change to the program source. This paper is organized as follows. In section 2, we motivate our approach and we present our implementation. Section 3 describes a process network deployment and distributed execution. The transfer strategies (demand and data driven) are detailed in section 4. And we finally outline our conclusions and plans for future work in section 5. 2. DESIGN AND IMPLEMENTATION 2.1. Related work
The Kahn process network model has been proposed by Kahn and MacQueen [6, 7] to easily express concurrent applications. Processes communicate only through unidirectional FIFO queues. Read operations are blocking. The number of tokens produced and their values are completely determined by the definition of the network and do not depend on the scheduling of the processes. The choice of a scheduling of a process network only determines if the computation terminates and the sizes of the FIFO queues. Some networks do not allow a bounded execution. Parks [ 10] studies these scheduling problems in depth. He compares three classes of dynamic scheduling: data-driven, demand-driven or a combination of both with respect to two requirements: 1. Complete execution (the application should execute completely, in particular if the program is non-terminating, it should execute forever). 2. Bounded execution (only a bounded number of tokens should accumulate on any of the queues). These two properties are shown undecidable by Buck [4] on boolean dataflow graph which are a special case of process networks. Thus they are also undecidable for the general case of process networks. Data-driven schedules respect the first requirement, but not always the second one. Demand-driven schedules may cause artificial deadlocks. A combination of the two is proposed by Parks [ 10] to allow a complete, unbounded execution of process networks when possible. In the context of a distributed execution of a process network, the process execution is inherently asynchronous. We have thus chosen a completely asynchronous scheduling: each process runs in its own thread that is scheduled by the underlying operating system. As explained by Parks and Roberts in [ 11 ], who use a similar scheduling technique, using bounded communication queues allow for a fair execution of the process network. This blocking write when the output queue is full can lead to deadlocks. Determining a priori if the queue length is large enough to avoid such deadlocks is undecidable. We provide a way for the user to modify this length at runtime.
33 Several implementations of process networks are used for different purposes: for heterogeneous modeling with PtolemyII [8], for signal processing application modeling with YAPI [5] and for metacomputing in the domain of Geographical Information Systems with Jade/PAGIS [13]. To our knowledge, only the Jade/PAGIS implementation and the one by Parks and Roberts [ 11 ] are distributed. Parks and Roberts use the Java Object Serialization to automate the distribution of the network processes while we use a central console to deploy the processes. In Jade, all communications proceed through a central communication manager while in Parks and Roberts' and our implementation the processes communicate directly. This allows a greater scalability. Only our implementation allow the coupling of processes written in different languages. 2.2. Design directions The design of our distributed process network implementation was done so as to: 9 enable users to simulate their network model quickly and effectively, 9 keep the source compatibility with the Yapi library: this library developed by Philips [5] implements the process network model for a local execution, Yapi is a C++ library which focuses on signal processing applications, 9 enable the distributed and the local execution without any change to the program source. The idea of the Yapi syntax is to group processes into process networks. The processes communicate via ports. These ports are linked by point-to-point unidirectional FIFO queues. A process network can be seen as a process and used in the same way. This hierarchical construction allows an easy modeling of complex applications. For this study, we have completely reengineered Yapi to be able to distribute any application over a CORBA bus without any change to the application code. In our implementation the communications are hidden to the programmer who can though configure the data transfer parameters. The reader can refer to [3] for more implementation details. 3. PROCESS NETWORK DISTRIBUTION 3.1. Deployment To control the distributed process networks, a console has been developed. It consists of a program which controls the processes connection and execution by the use of a simple language. It also provides a frontend used for monitoring. The presence of a manager program is contrary to the peer-to-peer character of component systems. However, the console is minimal and serves only as collaboration control. All the communications between the components are done without involving the console through the distributed FIFO queues presented in section 3.2. The FIFO links that form the process network are made interactively (or via a script) by this console. The use of a console allows more flexibility in the connection choice and a dynamic control of the components and the communication parameters. The use of an interactive console and the fact that the FIFOs are bounded also allow for an incremental development where computations can start even if the application is not complete. When all the output queues are full, the computation is suspended and can resume as soon as a consuming component is attached to the not-yet-connected output queues.
34
3.2. Distributed FIFOs The FIFO queues are completely distributed, and distributed process networks communicate directly, without a central point contrarily to what is done in Jade [ 13]. These queues implement the blocking read needed by the process network model but, as they are bounded, the write may block also if the FIFO queue is full. A deadlock can appear, but as the execution is fully distributed, deadlock detection is difficult and has not been implemented. To guarantee the code reuse with our implementation, the implementation of the distributed FIFOs must be done without programmer intervention or code change. This was done by encapsulating the distributed FIFOs (the CORBA object) in the FIFOs. The figure 1 shows the structure of the FIFO objects. For the programmer, no difference exists between the local and the distributed FIFO queue. To determine if the FIFO is a distributed FIFO or not, the runtime uses its ports. When a FIFO is local to the program, it has an input and output port. On the other hand, the distributed FIFO is represented by two half FIFOs, the output FIFO queue (producer side) and the input FIFO queue (consumer side). Each half FIFO has one port (an input port for the input FIFO, and an output port for the output FIFO) and should be linked to the other half FIFO. The runtime uses this property to activate the CORBA object only on the distributed FIFOs, and thus, the FIFO distribution is transparent for the programmer.
F,FO
_•
OutputPort
I
write
Data
write
CorbaObjRef
~,FO
I Da-~a 11 writ~Corba readi~__
le ad
o
Distributed FIFO ask
offer
full satisfyRequest sync_ask empty last_request
|
o~.~ ~ - -
(.1
getLength unlink
1
Eo.Oo ~u.
noUfy4Send
link
Distributed FIFO ask
~-<--
~_ read
InputPort
offer
full satisfyRequest sync_ask getLength unlink
empty last_request notify4Send
ilink
Figure 1. Distributed FIFOs implementation
To exchange data between distributed FIFO, we implement a communication protocol based on thresholds. This protocol which will be detailed in section 4 is a hybrid of data-driven and demand-driven transfers.
4. TRANSFER STRATEGIES In this section, we are interested in the data transmission between FIFOs. The transfer strategies were implemented having in mind the minimization of communications between FIFOs, the overlapping of communications by computations, and the load balancing.
35
4.1. Threshold based protocol The communication protocol governs the exchange of data between the FIFOs. A communication can be triggered by any of the two FIFOs in a hybrid data-driven, demand-driven protocol. This protocol is completely distributed, no central authority directs the communication. Each FIFO queue handles its data transmission independently of the rest of the process network. To manage the communications, two thresholds on the number of elements of the FIFO queue have been defined: 9 a maximum threshold (for the producer FIFO queues) which indicate that offering a part of its tokens is necessary to avoid overflowing, 9 a minimum threshold (for the consumer FIFO queues) which indicates that it is necessary to ask the linked producer half-queue for some tokens to avoid idle time. The distributed FIFO IDL interface described bellow defines the methods used for the initialization, the interaction with the other FIFOs, and the recovery of informations. This interface which is implemented by the distributedFifo class is not accessible to the user, but used implicitly by the distributed FIFO queue to communicate with the other distributed FIFO queue and the manager. interface
fifo
base
interface
{ void link(in string refFifo); void unlink(); u n s i g n e d long g e t L e n g t h ( ) ;
}; interface
fifo
int
: fifo
base
interface
{ oneway void a s k ( ) ; oneway v o i d f u l l ( ) ; boolean offer(in eltSequence buffer); oneway v o i d s a t i s f y R e q u e s t (in eltSequence buffer); void empty(); void sync_ask(inout eltSequence buffer); bool notify4Send();
}; The 1 i n k and u n l i n k permit to create or remove a link between two FIFOs. The a s k and s a t i s f y ' R e q u e s t methods are used respectively by the input FIFO to ask for data, and the output FIFO to response to an a s k . s y n c _ a s k is a synchronous version of the a s k method. The f u l l and e m p t y methods are used to indicate the FIFO state, while the o f f e r and n o t 2 f y 4 S e n d are used by the output FIFO to send data to the following FIFO respectively with and without event notification. It should be noticed that communications are vectorized. Indeed we transfer token sequences together and not each token at a time.
36 4.2. Transfer policies The two next paragraphs present the two transfer policies. It can be demand driven with event notification, and data driven without event notification. The other cases are implemented but not used by the runtime (demand driven without event notification by s y n c a s k method and data driven with event notification by n o t i f y 4 S e n d / s y n c _ a s k methods). However, the programmer can modify the transfer strategy by calling the c h a n g e P o l i c y method. Below, we suppose that the two FIFOs F_out and F_in are linked, the first one being the output FIFO and the second one, the input FIFO. Demand driven
When the transfer is demand driven, it is with event notification. When the token number becomes lower than the minimum threshold, the F_in FIFO notifies ( a s k ) the F_out FIFO which responds by sending data ( s a t 2 s f y R e q u e s t ) . The demand driven without event notification protocol was not used because such requests are synchronous and the process would be blocked until the availability of the requested data. With event notification, the process can continue computing if there are still some tokens in its input FIFO (F_in). Data driven
When the transfer is data driven, it is without event notification. When the token number in the FIFO F_out exceeds its maximum threshold, the FIFO sends data ( o f f e r ) to the F_in FIFO. To avoid the overloading of the F_in FIFO, the request returns a result. This result indicates if the receiver FIFO can receive or not other requests from the FIFO F_out. If the F_in FIFO is full, the F_out FIFO will not send any data until it receives a request for data from F_in.The offer requests being asynchronous (return a result), a separate thread manages this communication mode. A data driven protocol with event notification would involves two remote requests, and the use of this mode together with the demand driven protocol with event notification does not guarantee the order of the data reception. All the communications are hidden to the programmer, only the links between the distributed FIFOs determine the data exchange between processes. For the distributed FIFOs, the communications can be triggered in two ways: 1. Inside the process" according to whether the distributed FIFO is an input or an output FIFO, the communication triggering is done in two ways: 9 when a read operation is executed in an input FIFO (figure 2), a pre-read processing is started to ask eventually for data (if the number of tokens in the FIFO is below the minimum threshold). 9 on the other hand, when a write operation is executed in an output FIFO (figure 3), post-write processing is started to eventually send data to the following FIFO (if the number of tokens in the FIFO is above the maximum threshold). 2. Outside the process: this is done directly by the remote invocations from the linked FIFOs which ask for data or send data. However, to adapt the communications to particular applications, some methods have been implemented to give to the programmer the possibility to configure the communications, by
37
.... --
,- . . . . . . . . . . . . . .
..LI .............. ~ (~
..... ~ . . . . . . .
-
- - basic_read
~I
-~ ~I
I i
I
Figure 2. Read and pre-read operations in the distributed FIFO
_,
o.,.
....
~ ,
-
>
(~
:(')
_ _(2)_.....
.... :: 1
Figure 3. Write and post-write operations in the distributed FIFO
setting some communication parameters: minimum and maximum threshold, maximal FIFO capacity, exchange buffer's maximal size. 5. CONCLUSION We have described here our distributed implementation of process networks using CORBA. This implementation is based on the assembly of software components to form a distributed application. Each software component represents a process network which can be hierarchical, and thus exploits the parallelism of the process network model. The CORBA architecture choice is motivated by the handling of the hardware and software heterogeneity. The mapping and the control of the scheduling of the processes is carried out explicitly via a console interface. That makes it feasible to start an application respecting the process network model before its complete implementation. There are a lot of important issues we plan to investigate in the l~ear future. The most important is the dynamicity aspect of the network. Our previous implelhentation of a subcase of the process network model [1, 2] provides more dynamicity, where it is possible to migrate a process or to replace it by another, and our goal is to support these two dynamicity aspects in the general model. To achieve this, two difficulties must be overcome: 9 How to retrieve the internal state of the processes, and how to resume the computation. 9 How to retrieve the contents of the local FIFO queues. Moreover, we are interested in the use of the process network model as an execution model for the applications of multidimensional signal processing based on Array-OL [12], which is a programming language dedicated to systematic signal processing. REFERENCES
[1] Abdelkader Amar, Pierre Boulet, and Jean-Luc Dekeyser. Assembling dynamic com[2]
ponents for metacomputing using CORBA. In Parallel Computing 2001, Naples, Italy, September 2001. Lecture Notes in Computer Science. Abdelkader Amar, Pierre Boulet, and Jean-Luc Dekeyser. Towards distributed process networks with CORBA. Parallel and Distributed Computing Practice on Algorithms, 2003. Special Issue on Parallel and Distributed Computing Practice on Algorithms, to appear.
38 [3]
[4] [5]
[6]
[7]
[8] [9]
[ 10] [ 11 ] [12]
[13]
Abdelkader Amar, Pierre Boulet, Jean-Luc, and Frans Theeuwen. Distributed process networks using half FIFO queues in CORBA. Research Report RR-4765, INRIA, March 2003. Joseph T. Buck. Scheduling Dynamic Dataflow Graphs with Bounded Memory Using the Token Flow Model. PhD thesis, University of California at Berkeley, 1993. E.A. de Kock, G. Essink, W. J. M. Smits, P. van der Wolf, J.-Y. Brunel, W. M. Kruijtzer, P. Lieverse, and K. A. Vissers. YAPI: Application modeling for signal processing systems. In 37th Design Automation Conference, Los Angeles, CA, June 2000. ACM Press. Gilles Kahn. The semantics of a simple language for parallel programming. In Jack L. Rosenfeld, editor, Information Processing 74: Proceedings of the IFIP Congress 74, pages 471-475. IFIP, North-Holland Publishing Co., August 1974. Gilles Kahn and David B. MacQueen. Coroutines and networks of parallel processes. In B. Gilchrist, editor, Information Processing 77, pages 993-998. North-Holland, 1977. Proc.IFIP Congress. Edward A. Lee. Overview of the Ptolemy Project. University of California, Berkeley, March 2001. Object Management Group, Inc., editor. Common Object Request Broker Architecture (CORBA), Version 2.6. http://www.omg.org/technology/documents/ formal/corba iiop. htm, December 2001. Thomas M. Parks. Bounded Scheduling of Process Networks. PhD thesis, University of California at Berkeley, 1995. Thomas M. Parks and David Roberts. Distributed process networks in java. In International Workshop on Java for Parallel and Distributed Computing, Nice, April 2003. Julien Soula, Philippe Marquet, Jean-Luc Dekeyser, and Alain Demeure. Compilation principle of a specification language dedicated to signal processing. In Sixth International Conference on Parallel Computing Technologies, PaCT 2001, pages 358-370, Novosibirsk, Russia, September 2001. Lecture Notes in Computer Science vol. 2127. Darren Webb, Andrew Wendelborn, and Kevin Maciunas. Process networks as a highlevel notation for metacomputing. In Workshop on Java for Parallel and Distributed Computing (IPPS), Puerto Rico, April 1999.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
39
An efficient data race detector backend for DIOTA M. Ronsse a, B. Stougie b, J. Maebe ~, F. Cornelis ~, and K. De Bosschere ~ aELIS Department, Ghent University, Belgium bparallel and Distributed Systems, Delft University of Technology, The Netherlands In this paper, we describe a data race backend we developed for DIOTA. DIOTA (Dynamic Instrumentation, Optimisation and Transformation of Applications) is our generic instrumentation tool, and this tool uses so-called backends to use the information gathered using the instrumentation. Our novel data race backend uses innovative technologies like multilevel bitmaps, snooped matrix clocks and segment merging in order to limit the amount of memory used. The tool was implemented for Linux systems running on IA32 processors and is fully operational. 1. I N T R O D U C T I O N The never ending urge for faster and more robust computers, combined with the existence of cheap processors causes a proliferation of inexpensive multiprocessor machines. Multithreaded applications are needed to exploit the full processing power of these machines, causing a widespread use of parallel applications. Even most contemporary applications that are not CPU-intensive are multithreaded because the multithreaded paradigm makes it easier to develop servers and applications with an MDI (multiple document interface) such as Windows programs to improve responsiveness to actions done by the user, etc. However, developing multithreaded programs for these machines is not easy as it is harder to get a good view on the state of a parallel program. This is caused by the fact that there are a number of threads 1 running simultaneously. Moreover, the very fact that the computation is split into simultaneously executing parts can introduce errors that do not exist in sequential programs. These synchronization errors show up because parallel programs are developed in order to let a number of threads work together on the same problem, hence they will share data. It is of paramount importance that the accesses performed by the different threads are properly synchronised. A lack of synchronisation will lead to data races. More specifically, such a data race occurs when two or more concurrently executing threads access the same shared memory location in an unsynchronised way, and at least one of the threads modifies the contents of the location. As data races are (most of the time) considered bugs, they should be removed. Unfortunately, data races are difficult to find because their occurrence depends on small timing variations. Although it is possible to detect data races using a static approach this is not feasible for nontrivial programs. Therefore, most data race detection tools detect data races dynamically during an actual program execution. 1in this paper we will consider an execution of a parallel program as being a process consisting of N threads executing on a machine with N processors.
40 2. DATA RACE DETECTION As mentioned before, a dynamic data race detector finds data races that occur during a particular execution. In order to detect the conflicting memory operations, one should collect these operations during a particular execution together with information about their concurrency. The concurrency depends on the (order of the) executed synchronisation operations. Basically two (dual) methods exist for detecting data races using collected memory and concurrency information: 9 for each access to a global variable, the access is compared against previous accesses by other threads to the same variable. This requires, for each global variable, information about past accesses. It is obvious that this will lead to a huge memory consumption, especially if each memory location is a potential global variable with life time equal to that of the program itself, as is the case for e.g. programs written in the C language. This method can however be applied for programs with a rigorous object model that is adhered to, such as Java programs as is shown in [4]. 9 for all parallel pieces of code, the memory operations are collected and compared. This method is based on the fact that all memory operations between two successive synchronisation operations (called segments from now on) satisfy the same concurrency relation: they are either all concurrent or not concurrent with a given operation and therefore with the segment containing the latter operation. Given the sets L(i) and S(i) containing the addresses used by the load and store operations in segment i, the concurrent segments i
((L(j)U S(j))N S(i)) r O. Therefore, data race detection basically boils down to collecting the sets L(i) and S(i) for all segments executed and comparing parallel segments. Note that (contrary and j contain racing operations if and only if
((L(i)US(i))AS(j))
U
2"
to the first method) a data race will be detected at the end of the segment containing the second racing instruction, possibly a long time (e.g. at the end of the program) after the actual occurrence of the data race. It is clear that the second method is better suited for programming languages with unconstrained life time of and access to shared variables. As we wanted to develop a tool that is programming language independent, we opted for this method. The remainder of this papers is organised as follows: first we will give a general overview of the infrastructure that supports our implementation of the data race detection tool that is being discussed here. Next, we will explain the paradigms and techniques that we use to detect data races and to make sure that the resulting tool can be used on wide range of real-world applications and with a reasonable slowdown factor. Finally, we will present some experimental evaluation results and the conclusions of this paper, along with our future plans. 3. DIOTA 3.1. General overview
DIOTA (DynamicInstrumentation, Optimizationand Transformationof Applications) is implemented as a shared library for the Linux/80x86 platform. It instruments programs at the machine code level, so it is completely compiler- and language-agnostic and can also cope with
41 hand-written assembler code. It has support for extensions to the 80x86 ISA such as MMX, 3DNow! and SSE and is written in an extensible and modular way so that adding support for new instructions and new types of instrumentation is easy. An environment variable is used to tell the dynamic linker to load the DIOTA library whenever a dynamically linked application is started 2. An i n i t routine allows DIOTA to be activated before the main program is started, after which it can gain full control and start instrumenting. The instrumentation happens gradually as more code of the program (and the libraries that it uses) is executed, so there is no need for complex analysis of the machine code to construct control-flow graphs or to detect code-in-data. The instrumented version of the code is placed in a separate memory region (called the "clone"), so the original program code is not touched and as such neither data-in-code nor the variable-length property of 80x86 instructions pose a problem. Once DIOTA encounters an instruction of which the successor is unknown (such as a jump with a target address that lies in a register or a return instruction), it inserts a trampoline in the clone. Such a trampoline will pass the actual target address to DIOTA every time the instrumented version of this instruction is executed. This way, DIOTA can check whether the code there is already instrumented and if not, do it now. Next, the instrumented version of the target code is executed. Since every exit from an instrumented block leads either to a trampoline or to another instrumented block, DIOTA remains in control. What is extremely important in the light of data race detection however, is that during this process no data is relocated and as such all addresses used by the program remain the same as in an uninstrumented execution, plus or minus a certain delta due to memory occupied by the DIOTA library. As such, all accessed memory addresses can be easily recorded without the need for performing any kind of relocation afterwards. The behaviour of DIOTA can be influenced by using so-called backends. These are small (and not so small) dynamic libraries that link against DIOTA and tell it what kind of instrumentation should be performed. They can ask for any dynamically linked routine to be intercepted and replaced with a routine of their own, ask to be notified of each memory access, of each basic block that is executed and of each system call that is performed (both before and after their execution, so their behaviour can be modified as well as analysed). DIOTA can also handle multi-threaded programs and supports the instrumentation of exception handlers. Further technical details and in-depth discussion on the issues surrounding the interception of dynamically linked routines, can be found in [8]. 3.2. Data race detection b a c k e n d
As described above, the data race detection backend is implemented as a shared library that uses the services provided by DIOTA. The backend request DIOTA to instrument all memory operations and to intercept all p t h r e a d synchronisation operations. The memory operations are intercepted to construct the L(i) and S(i) lists for all segments and the synchronisation operations are intercepted to detect concurrent segments. The problems faced by most contemporary data race detectors is the huge overhead they introduce. The fact that an execution will slow down considerably is primarily caused by the interception of all memory accesses. Indeed, for each memory access a number of operations 2This is the only prerequisite for DIOTA to be able to instrument a program: the program must be dynamically linked. Fortunately, this is the case for all but a few system-maintenanceapplications.
42 will have to be performed by the backend. There exist tools that try to reduce the number of memory operation that have to be checked; e.g. static analysis could reveal that certain memory regions will only be used by one thread. This is of course not possible if we perform data race detection at the lowest level (the machine instruction level, where we could encounter memory operation with unbound addresses, e.g. as a result of the use of a pointer in a C-program), but it is viable for languages were access to variables is strictly controlled and there are no pointers to bypass this protection (e.g. Java [4]). Our tool throws away a small portion of all memory operations: the memory operations executed by the first thread and before a second thread is created are not checked, as they can not be part of a data race. A second reason for the large overhead is the huge memory consumption exhibited by most data race detectors. This is caused by the fact that memory operations have to be compared with memory operations executed in parallel with and in the past of this operation. Keeping track of the old memory operations and the concurrency information requires a lot of memory and we will go to great length to lower the memory consumption, as will be explained in detail in the next few paragraphs.
Collecting and comparing the addresses of the memory operations As mentioned before, two lists of addresses are used for every segment i: the load operations are collected in L(i) and the store operations in S(i). These addresses are collected in a bitmap (one for each list). Such a bitmap contains a 1 on place i if address i was used. As a linear bitmap would require 232/8 = 512 MiB, a multilevel bitmap is used. At this moment we use a 9-level bitmap: the first 8 levels contain page tables while the last level contains the actual bitmap. The highest level is addressed using the 3 highest bits of the addresses. This level contains pointers to page tables at the second level. These page tables are indexed using the next 3 bits and contain pointers to the actual bitmaps. This continues for the next 6 levels, each level indexed using the next 3 bits. The actual bitmaps are indexed using the remaining 8 bits of the address. Although the use of a bitmap means that some information is lost - two or more accesses to the same address count as one - this is not a problem for data race detection. Indeed, if one of these accesses is involved in a race, the subsequent accesses are also involved in a data race. The use of multilevel bitmaps makes it possible to compare the segments in an efficient way.
Detecting and comparing parallel segments In our race detection tool, we use a classical logical vector clock [9] to detect concurrent segments. As vector clocks are able to represent the happened-before [6] relation (they are strongly consistent), two vector clocks that are not ordered must belong to concurrent segments. This gives us an easy way to detect concurrent segments.
Detecting Obsolete Segments For each segment i encountered during the program execution, we build two lists: L(i) and Of course, if we keep all these lists in memory, we will run out of memory. However, it is possible to throw away old information: if it is impossible that a new segment will be parallel with an 'old' segment i, we can throw away the lists L(i) and S(i), effectively freeing memory. Detecting the 'old' segments can be done using matrix clocks.
S(i).
43 Logical matrix clocks are traditionally used to discard information in a distributed environment: the componentwise minimum of the columns yields the maximum number of segments per thread that can be discarded [ 14]. However, in practice we can discard more information than is indicated by logical matrix clocks as logical clocks capture causality, which is one of the weakest forms of event ordering. In a particular execution, all events are executed in some order (not specified by the program), even if they do not have a causal relationship. Classical logical clocks are not able to capture this kind of additional execution-specific ordering. Our data race detection method uses clock snooping [5]. Instead of maintaining matrix clocks, a snooped matrix clock is built each time a segment ends, using the latest vector clock of the processors. All segments that have a vector clock smaller than the componentwise minimum of the snooped matrix clock can be discarded as they will not be needed in the future anymore.
Merging Segments For most parallel applications, logical matrix clocks succeed in throwing away obsolete information. However, there are applications where the segment lists never become obsolete, e.g. because there is little synchronisation between the threads. The lack of synchronisation will preclude segments from becoming causally ordered before new segments, preventing memory from being freed. It is however possible (under strict cirumstances) to lower the memory consumption somewhat by combining segments. This technique [3] is based on the fact that, if all new segments will be ordered equally (parallel with or ordered after) with two segments i and j, we can merge the segments i and j. Merging boils down to removing the segments i and j from the list and adding a new segment j' with L(i') = L(i)U L(j) and S(i') = S(i)U S(j). Of course, we lose some information with merging: if a data race is detected, we no longer know if segment i or j is the cause. Although merging segments belonging to two different threads is possible, at this moment DIOTA only merges successive segments belonging to the same thread.
4. EXPERIMENTAL EVALUATION During the development of our data race backend we used a multitude of small parallel programs and it turned out that our tool was able to find the data races we intentionally introduced. Until now, only a small number of large parallel applications have been benchmarked. One of the benchmarks we used is the Mozilla web browser, a very large, complex and multithreaded application. The table below show some results we obtained with some benchmarks on a dual Celeron processor (500MHZ, 128KiB cache) system running Linux kernel 2.4.19. Four different executions were tested: a normal execution, an execution under control of DIOTA but without adding instrumentation, the same execution but with instrumentation of all memory operations, and finally an execution with the full blown data race detection. Maximum memory overhead was 3.4 x. It is clear that the slowdown is rather high. Fortunately, this is not a real problem since the data race detector can run unattended, e.g. overnight. The effort invested in methods for limiting the memory consumption clearly pays off: the memory consumption is moderate, considering the information that is needed to perform data race detection.
44 program
normal exec. time mozilla 7.50 LU.cont -p4 8.06 fft-p4-m22 11.47 radix-p4-n4194304 6.96 cholesky-p4 inputs/tk29.o 1 0 . 4 3 ocean-p4-n514 15.59 radiosity -p 4 -batch -room 27.50 water-spatial < input4 30.70
no instrum, time slowd, 35.00 4.67x 9.59 1.19x 27.59 2.41x 11.74 1.69x 12.84 1.23x 17.56 1.13x 90.14 3.28x 52.51 1.71x
diota memory instrum, time slowd, 169.00 22.53x 54.15 6.47x 200.37 17.48x 137.39 19.73x 310.74 29.79x 339.06 21.75x 1157.45 42.09x 742.27 24.18x
data race detection time slowd. 401.00 53.47x 85.74 10.64x 393.36 34.31x 244.18 35.07x 581.97 55.80x 667.14 42.80x 6 8 0 5 . 6 1 247.49x 1 5 6 6 . 0 4 51.01x
Table 1 Benchmark results.
5. RELATED W O R K
Although much theoretical work has been done in the field of data race detection [ 1, 2] few implementations for general systems have been proposed. Tools proposed in the past had limited capabilities: they were targeted at programs using one semaphore [7], programs using only post/wait synchronisation [ 11 ] or programs with nested fork-join parallelism [2, 10]. In previous work, we developed RECPLAY [ 12] for Solaris running on the SPARC architecture. This tool was geared exclusively at data race detection and contained an instrumentation engine because this was necessary to gather the accessed memory locations. Later, the instrumentation engine was separated under the name JiTI (Just-in-Time Instrumentation), with the data race functionality remaining as a special purpose backend. Recently, JiTI has been ported to the 80x86 architecture and has since been greatly enhanced and extended, so that another name change was advisable. The DIOTA name captures the (both current and potential) capabilities of the new framework quite well. A major difference between DIOTA and JiTI is that DIOTA also instruments code in dynamic linked libraries. Also, the data race backend of DIOTA is much more powerful than the JiTI version, e.g. segment merging was added recently. The backend is still being actively developed and improved regularly. Most of the previous work, and also our tool, is based on Lamport's happened-before relation. This relation is a partial order on all synchronisation events in a particular parallel execution. Therefore, by checking the ordering of all events and by monitoring all memory accesses, data races can be detected for one particular program execution. Another approach is taken by a more recent race detector: Eraser [ 13]. It goes slightly beyond work based on the happened-before relation. Eraser checks that a locking discipline is used to access shared variables: for each variable it keeps a list of locks that were held while accessing the variable. Each time a variable is accessed, the list attached to the variable is intersected with the list of locks currently held and the intersection is attached to the variable. If this list becomes empty, the locking discipline is violated, meaning that a data race occurred. Unfortunately, Eraser detects many false data races: as Eraser is not based on the happened-before relation it has no timing information whatsoever. For instance, in theory there is no need to synchronise shared variables before multiple threads are created.
45 6. C O N C L U S I O N S A N D F U T U R E W O R K
In this paper, we presented a tool called DIOTA and its data race backend. The tool is able to instrument complex programs and with the help of the backend it can do data race detection on them. The data race detection occurs by dividing the execution into segments and by keeping a history of accessed memory locations for each segment. The segments themselves are ordered through the use logical vector clocks, which are updated by occurrences of synchronisation operations. In the future, we want to improve the speed of the data race backend in several ways. One technique is to remove redundant checks of addresses: once an address has been written to once in a segment, all subsequent read and write operations to the same address are irrelevant as far as data race detection is concerned. Therefore, it is possible to greatly optimise the instrumentation. Simple testing has shown that in e.g. the Mozilla web browser, 60% of the memory accesses use the same address as the previous memory access. Also when looking at loops, it is possible to move a lot of instrumentation code out of the loop body, e.g. the instrumentation of the write operations to loop counters. Another enhancement we propose, is to treat exception handlers properly with regard to data race detection. Currently, memory operations inside exception handlers are considered as being part of the thread in which the exception occurred. However, since exceptions are raised asynchronously, they should be regarded as very short lived separate threads instead. The latest version of DIOTA can be downloaded from h t t p : / / w w w . e 1 i s . U G e n t . b e /
diota/.
REFERENCES [1]
S.V. Adve, M.D. Hill, and R.H.B. Netzer. Detecting data races on weak memory systems. In Proceedings of the 18th Annual Symposium on Computer Architectures, pages 234-243, May 1991. [2] K. Audenaert and L. Levrouw. Space efficient data race detection for parallel programs with series-parallel task graphs. In Proceedings of the third Euromicro Workshop on Parallel and Distributed Processing, pages 508-515, San Remo, January 1995. IEEE Computer Society Press. [3] M. Christiaens, M. Ronsse, and K. De Bosschere. Bounding the number of segment histories during data race detection. Parallel Computing, 28(9):1221-1238, 9 2002. [4] Mark Christiaens and Koen De Bosschere. Trade, a topological approach to on-the-fly race detection in java programs. In Java Virtual Machine Research and Technology Symposium (JVM'O1), pages 105-116. USENIX, April 2001. [s] Koen De Bosschere and Michiel Ronsse. Clock snooping and its application in on-thefly data race detection. In Proceedings of the 1997 International Symposium on Parallel Algorithms and Networks (I-SPAN'97), pages 324-330, Taipei, December 1997. IEEE Computer Society. [6] Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21 (7):558-565, July 1978. [7] H.I. Lu, P.N. Klein, and R.H.B. Netzer. Detecting race conditions in parallel programs that use one semaphore. Technical report, Brown University, 1993. [8] J. Maebe, M. Ronsse, and K. De Bosschere. DIOTA: Dynamic instrumentation, opti-
46
[9]
[10]
[ 11 ]
[ 12] [13]
[14]
mization and transformation of applications. In M. Chamey and D. Kaeli, editors, Compendium of Workshops and Tutorials Held in conjunction with PACT'02: Intl. Conference on Parallel Architectures and Compilation Techniques, Charlottesville, VA, 9 2002. Friedemann Mattem. Virtual time and global states of distributed systems. In Cosnard, Quinton, Raynal, and Roberts, editors, Proceedings of the Intl. Workshop on Parallel and Distributed Algorithms, pages 215-226. Elsevier Science Publishers B.V., North-Holland, 1989. John M. Mellor-Crummey. On-the-fly detection of data races for programs with nested fork-join parallelism. In Proceedings of Supercomputing '91, pages 24-33, November 1991. Robert H. B. Netzer and Barton P. Miller. On the complexity of event ordering for sharedmemory parallel program executions. Intl. Conference on Parallel Processing, pages 9397, August 1990. Michiel Ronsse and Koen De Bosschere. Recplay: A fully integrated practical record/replay system. ACM Transactions on Computer Systems, 17(2):133-152, May 1999. Stefan Savage, Michael Burrows, Greg Nelson, Patrick Sobalvarro, and Thomas Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 15(4):391-411, November 1997. G.T.J. Wuu and A.J. Bemstein. Efficient solutions to the replicated log and dictionary problems. Proc. 3rd ACM Symp. Principles Distributed Computing, pages 233-242, New York, 1984. ACM Press.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
47
Pipelined parallelism for multi-join queries on shared nothing machines M. Bamha a and M. Exbrayat ~ aLaboratoire d'Informatique Fondamentale d'Orlrans, Universit6 d'Orlrans, BP 6759, 45067 Orlrans cedex 2, France {mostafa.bamha,matthieu, exbrayat }@lifo.univ-orleans. fr The development of scalable parallel database systems requires the design of efficient algorithms, especially for join, which is the most frequent and expensive operation in relational database systems. Join is also the most vulnerable operation to data skew and to the high cost of communication in distributed architectures. Moreover, for multi-join queries, the problem of data-skew is more complicated because the imbalance of intermediate results is unknown during static query optimization. In this paper, we show that the join algorithms we presented in our earlier papers can be applied efficiently in various parallel execution strategies making it possible to exploit not only intra-operator parallelism but also inter-operator parallelism. These algorithms minimize the communication and synchronization costs while guaranteeing a perfect load balancing during each stage of join computation even for highly skewed data. 1. INTRODUCTION Join is the most frequent operation of parallel relational database management systems (PDBMS). It is also the most expensive one, due to its vulnerability to data skew and to the high cost of communications in shared nothing architectures. Research has shown that join is parallelizable with near-linear speed-up on Shared Nothing machines only under ideal balancing conditions. Data skew can have a disastrous effect on performance [ 1] due to the high costs of communications and synchronizations in this architecture. Many algorithms have been proposed to handle data skew for a simple join operation, but little is known for the case of complex queries leading to multi-joins [2, 3, 4]. In particular, the performance of PDBMS has generally been estimated on queries involving only one or two joins [5]. However the problem of data skew is more acute with multi-joins because the imbalance of intermediate results is unknown during static query optimization [6]. Such algorithms are not efficient for many reasons. First, they are not scalable (and thus cannot guarantee linear speedup) because their routing decisions are generally performed by a coordinator processor while the others are idle. Second, they cannot solve load imbalance problems as they base their routing decisions on incomplete or statistical information. Finally, they poorly handle data skew because data redistribution is generally based on hashing data into buckets, while hashing is known to be inefficient in the presence of high frequencies [2]. On the contrary, the join algorithms we presented in [7, 8, 9], use a total data-distribution information in the form of histograms. In this paper, we show that these join algorithms can be
48 applied efficiently in various parallel execution strategies, making it possible to exploit not only intra-operator parallelism but also inter-operator parallelism. These algorithms are used in order to avoid the effect of load imbalance due to data skew, and to reduce the communication costs due to the redistribution of the intermediate results, which can lead to a significant degradation of the performance. The organization of this paper is as follows: we first recall the notions of PDBMS, join operations and load balancing in section 2. We then present parallel execution strategies for multi-queries in section 3, and propose and discuss the use of pipelining strategies together with the Osfa_j oin algorithm in section 4. We finally sum up in section 5. 2. PDBMS : JOIN OPERATIONS AND LOAD BALANCING
2.1. PDBMS, join operations and data skew The join (or equi-join) of two relations R and S on attribute A of R and attribute B of S (A and B of same domain) is the relation T, written R N S, containing the pairs of tuples from R and S verifying R.A = S.B. The sere• - j o • of S by R is the relation S ~< R composed of the tuples of S which occur in the join of R and S. Semi-join reduces the size of relations to be joined and R N S = R N (S K R) = (R ~< S) N (S K R). In PDBMS, relations are generally partitioned among nodes by horizontal fragmentation according to the values of a given attribute. Three methods are used [10]: hash partitioning, block-partitioning (also called range partitioning) and cyclic (also called round-robin) partitioning. The main parallel join algorithms are: Sort-merge join, Simple-hash join, Grace-hash join and Hybrid-hash join [11]. All of them are based on hashing functions which redistribute relations so that tuples having the same attribute value are forwarded to the same node. Local joins are then computed and their union provides the output relation. Their main disadvantage is to be vulnerable to both attribute mlue skew (imbalance of the output of the redistribution phase) and join product skew (imbalance of the output of local joins) [4, 6]. The former affects immediate performance and the latter affects the efficiency of output of pipelined operations in the case of a multi-join. 2.2. Load balancing in parallel join operations The authors of [4] have identified the two best proposed solutions in conventional and sampling-based parallel join algorithms. All are based on the notion of parallel hashing. In the first category, the extended adaptive load balancing parallel hash join [3] sends all tuples with the same attribute value to the same node. As a result, the algorithm may fail to balance the load when few values have a large weight. Moreover, the algorithm ignores the attribute value skew (AVS) of the probe relation and the join product skew UPS). In the second category of algorithms, virtual process partitioning [2] improves on the previous category but only handles AVS for the build relation. It also ignores JPS. To improve on these methods, the authors of [4] have introduced an algorithm, that minimizes the expected AVS by statistically predicting the result of hashing. However, it still fails to correct JPS. The algorithm is based on hashing, and therefore sensitive to the same effect as earlier methods (an important skew will anyway lead to a JPS). We conclude that all existing methods are sensitive to imbalance when applied multiple times because of JPS. To address this problem, we introduced in [7] a deterministic data redistribution algorithm
49 with near-perfect balancing properties, that dynamically computes exact frequency histograms to avoid JPS. A scalable and portable cost analysis was made with the BSP model [12], leading to general predictions about the effect of relation histograms on performance. The analysis suggests a hybrid frequency-adaptive algorithm (f a _ j o i n algorithm) [7], dynamically combining histogram-based balancing with standard hashing methods. The f a _ j o i n algorithm avoids the slowdown usually caused by AVS and the imbalance of the size of local joins processed by the standard algorithms. It handles skews at the cost of extra processing time. We analyzed this overhead both theoretically and experimentally and concluded that it never penalizes overall performance, even in the absence of skew. However, the performance of f a _ j o i n is sub-optimal when computing the join of highly skewed relations because of unnecessary redistribution and communication costs. We introduced in [9] a new parallel algorithm called Osf a _ j o i n (Optimal symmetric frequency adaptive join algorithm) to perform such joins. O s f a _ j o i n improves on f a _ j o i n by its optimal complexity and by the use of semi-joins. Its predictably low JPS and AVS make it suitable for multi-join queries. Its analyze through BSP predicts a linear speedup and an optimal complexity even for highly skewed data. 3. PARALLEL EXECUTION STRATEGIES FOR MULTI-JOIN QUERIES Several strategies were proposed for the evaluation of multi-join queries [5]. These strategies generally depend on the parallel query execution plan. They can use both intra-operator and inter-operator parallelism, and mostly differ in the way they allocate simple joins to different processors and the way they choose an appropriate degree of parallelism (i.e. the number of processors used to compute each simple join). They can be divided into four main categories presented thereafter.
3.1. Sequential parallel execution Sequential parallel execution is the simplest strategy for evaluating, in parallel, a multi-join query. This strategy does not require inter-operator parallelism since simple joins of the query are evaluated one after the other in a parallel way. It does not induce any pipelined parallelism owing to the fact that a simple join cannot be started until all its operands are entirely available. The execution time of each join is then the execution time of the slowest processor. To reach acceptable performance, the join algorithms used in this strategy should reduce the load imbalance among all the processors. O s f a _ j o i n can be used there to reduce the idle time, owing to the fact that it is insensitive to AVS and JPS, and also proved to guarantee a linear speedup. 3.2. Parallel synchronous execution In addition to intra-operator parallelism, parallel synchronous execution uses inter-operator parallelism [13]: several simple joins can be computed simultaneously on disjoint sets of processors. The main difficulty then lies in the allocation of simple joins to the available processors and in the choice of an appropriate degree of parallelism for each join, so that i) the global execution time of each operator should be of the same order to avoid the latencies induced by the imbalance in computations of the different groups of processors, and ii) the execution time of each operator of a given group of processors (i.e. of a given parallel join) must also be of the same order, as the execution time for each join is the execution time of its slowest processor.
50 The load balancing between the groups of processors depends on the effectiveness of the dynamic balancing strategy used at the time of the query optimization, whereas the load balance of the processors in the same group can be carried out by using the join algorithms we introduced.
3.3. Segmented right-deep execution Contrary to a parallel synchronous strategy, a segmented right-deep execution [14] uses, in addition to intra-operator parallelism, pipelined inter-operator parallelism. Pipelined interoperator parallelism is used in the evaluation of the right-branches of the query tree. O s f a _ j o i n could still be used there. Nevertheless, the initial version of O s f a _ j o i n computes joins sequentially, one after the other, and thus has to be modified to handle pipelining efficiently. 3.4. Full parallel execution Full parallel execution [5] is simply a combination of the preceding strategies, that uses intra-operator, inter-operator, and pipelined inter-operator parallelism. In this strategy, all the simple joins, that compose the multi-join query, are computed simultaneously in parallel using disjoint sets of processors. Inter-operator parallelism and pipelined inter-operator parallelism are exploited according to the type of the query tree. Its effectiveness depends on the quality of the execution plan. The initial version of Osfa_j oin is not really adapted to full parallelism Oust like it was not really adapted to segmented right-deep execution), and thus has to be modified to handle pipelining. In the following section we will present a detailed approach to introduce pipelining in Osfa_j oin. 4. PIPELINING OSFA JOIN: CAN WE PIPELINE A DYNAMIC A L G O R I T H M ? . - -
4.1. Principle Pipelining has been successfully implemented with many classical join algorithms. Nevertheless, these algorithms are generally based on hash join techniques and are thus sensitive to AVS and JPS, which are very likely to occur in a multi-join query. For this reason we propose to adapt O s f a _ j o i n (which is insensitive to data skew) to pipelined joins. This task in fact appears to be difficult. Most of pipeline strategies use a static execution plan. The way data is distributed for each operator is fixed at query compilation time. When one processor produces a result tuple, this latter can be immediately transmitted to and used by the next operator, which does not need to have a global view of the whole set of manipulated tuples. This can be repeated over several consecutive join operators. Long pipeline chains can be constituted, their length being mainly limited by processors' availability. On the contrary, O s f a _ j o i n is a dynamic operator; the definitive distribution of data is determined at execution time, once histograms have been built, i.e. after the former operator has finished. This strongly limits the use of pipelines. We thus propose to use two-operator pipeline chains. The first enhancement consists in parallelizing the creation of histograms for source relations (i.e. using inter-operator parallelism). This means that we organize the execution plan so that as many join operators as possible apply either on two source tables, or on one source table and an intermediate result. In this case we can build the histograms for source tables as soon as the
51 query execution begins. This strategy is very similar to the parallelized build phase that is used for classical right-deep trees. We then build the histograms for computed (intermediate) relations as their tuples are produced. In other words, we interlace the tuple production phase of a join operator with the building phase of the next. By the way we do not need to re-scan the results from the disk. Once the histograms are produced for both tables, we can compute the communication template, then distribute data, and finally compute the join. Unfortunately, computing the communication template is the implicit barrier within the execution flow, that prohibits the use of pipeline chains. From the implementation point of view, we must notice that we do not really compute the local histogram on-the-fly (i.e. each time a tuple is produced), but rather wait for a sufficient number of tuples to be available to compute the corresponding histogram.
4.2. Detailed algorithm Let us consider each join is of the form Ri N Si, where Ri is a source relation and S~ is either a source relation (which should only occur for the first join) or an intermediate result. The execution tree consists of t join operators; the final result is stored as St+l. Let us call Rij the fragment of Ri which is stored on processor j. We use n nodes numbered 1 to n. The global algorithm is then: Algorithm 1 Pipelined Osfa_join For each join Ji - ( R i N Si) Par (on each node) j E [1, n] create local histograms for Rij Hist(Rij) broadcast Hist(Rij) to other nodes, and collect all Hist(Rik), k 7~ j build the corresponding part of the global histogram (each node being assigned a given set of values) for Ri Histj (Ri) If i= 1 (first join) Then create local histograms for Sij Hist(Sij) Else wait for Hist(Sij) to be completed by the preceding join phase Endif broadcast Hist(Sij) to other nodes, and collect all Hist(S~k), k ~ j build global histogram for 5'/Histj (Si) compute communication templates according to Histj(S~) and Histj(Ri) (see [7, 8, 9]) broadcast (and collect) the communication templates exchange data across nodes according to the communication template If i=t (last join) Then compute S(i+l)j = (Rij N Sij) Else compute S(~+I)y = (R~j N S~j) and build Hist(S(~+l)j) on the fly Endif Endpar Endfor
52
4.3. Example Figures 1 and 2 represent an annotated query execution tree of a 3-join query T1 N T2 M T3 t~ T4. Source tables are placed on leaves. Numbers indicate the relative order of operations. Referring to the general algorithm, we can identify TJ, T3 and T4 respectively to R1, R2 and R3, and T2 to $1. Figure 1 illustrates an execution where operators are executed sequentially (i.e. a basic use of Os f a _ j o i n , where we do not use inter-operator parallelism). We first build the histograms for T 1 and T2, then compute the communication templates, distribute data, and execute the join. Once the join is over, we build the histograms for T3 and T1 N T2, and so on. Figure 2 illustrates a pipelined execution of the query tree. Building the histograms of all source tables is executed in parallel. As soon as histograms are available for T1 and T2, the communication template of the first join is computed, then data is transfered, and finally the join is executed. During this last operation, result tuples are scanned in order to produce the histogram of T1 N T2. Once this phase is over, we can compute the communication template for the second join, and so on.
6. Compute comm. templates, exchange data, join T4 and $3 into $4 (result)
5. build Hist(T4) T4
5. build Hist(S3) 4. Compute comm. templates, exchange data, join T3 and $2 into $3
3. build Hist(T3) T3
3. build Hist(S2) 2. Compute comm. templates, exchange data, join T1 and T2 into $2
/2
1. build Hist(T1) T1
1. build Hist(T2) T2
Figure 1. Standard execution of a multi-join query using Os f a _ j o i n
4. Compute comm. templates, exchange data, join T4 and $3 into $4 (result)
1. build Hist(T4) T4
3. Compute comm. templates, exchange data, join T3 and $2 into $3, and build Hist(S3) on the fly
T3
2. Compute comm. templates, exchange data, join T1 and T2 into $2, and build Hist(S2) on the fly
/<
1. build Hist(Tl) Tl
1. build Hist(T2) T2
Figure 2. Pipelined execution of a multi-join query using Os f a _ j o i n
4.4. Discussion We showed in section 4.1 that Os f a _ j o i n could hardly be pipelined in a long chain. Nevertheless our two-operator pipeline achieves several enhancements. We first use a parallel construction of histograms for source relations. Moreover, we combine the actual computation of joins together with the production of the next-join histogram, thus limiting the number of accesses to data (and to disks).
53 Conceming the parallel construction of histograms for source relations, we can notice that the degree of parallelism might be limited by two factors: the total number of nodes available, and the original distribution of data. A simultaneous construction of two histograms on the same node (which occurs when two relations are distributed, at least partially, over the same nodes) would not be really interesting compared to a sequential construction. This inter-processor parallelism does not bring acceleration, but should not induce noticeable slowdown: histograms use to be small, and having several histograms in memory would not necessitate swapping. On the other hand, as relations use to be much bigger than the available memory, we have to access them by blocks. As a consequence, accessing one or several relations does not really matter. Our pipeline strategy will really be efficient if join operators are executed on disjoint (or partially disjoint) sets of processors. Intra-operator parallelism is thus limited, and we have to segment our query trees, similarly to segmented fight-deep trees, each segment (i.e. a set of successive joins) being started when the former is over. These two limitations (degree of parallelism of operators and number of "pipelined" operators) would naturally suppose some minor modifications of the general algorithm presented above. 5. CONCLUSION In this paper we overviewed the existing parallel join algorithms, and the execution strategies of multi-joins queries on shared-nothing machines. We have shown that the algorithms, we introduced in earlier papers can be applied efficiently in different parallel strategies. We proposed a pipelined version of the Os f a _ j o i n algorithm. Our approach consists of two elements: parallelizing the scan of source tables and scanning intermediate results at production time, thus limiting disk accesses. We show that long pipeline chains can hardly be constructed due to the dynamic nature of O s f a _ j o in. We are currently conducting tests that tend to validate the superiority of our approach compared to standard hash-join pipeline algorithms. The pipelined version of O s f a _ j o i n is currently implemented within a parallel database prototype, in the framework of the CARAML ~ Grid project. REFERENCES
[1] L. Harada and M. Kitsuregawa, Dynamic Join Product Skew Handling for Hash-Joins in [2]
[3] [4]
Shared-Nothing Database Systems, in Proc. of the 4th Int. Conf. on Database Systems for Advanced Applications, Singapore, 1995,246-255. D. J. DeWitt, J. F. Naughton, D. A. Schneider and S. Seshadri, Practical Skew Handling in Parallel Joins, in Proc. of the 18th Int. Conf. on Very Large Databases, Vancouver, Canada, 1992, 27-40. K. A. Hua and C. Lee, Handling data skew in multiprocessor database computers using partition tuning, in Proc. of the 17th Int. Conf. on Very Large Databases, Barcelona, Spain, 1991, 525-535. V. Poosala and Y. E. Ioannidis, Estimation of Query-Result Distribution and its Application in Parallel-Join Load Balancing, in Proc. of the 22th Int. Conf. on Very Large Databases, Bombay, India, 1996, 448-459.
1CARAMLproject URL http://www.caraml.org
54 [5]
[6] [7] [8]
[9] [10] [ 11]
[12] [13]
[14]
A.N. Wilschut, J. Flokstra and P. M. G. Apers, Parallel evaluation of multi-join queries, in Proc. of the 1995 ACM SIGMOD Int. Conf. on Management of Data, San Jose, USA, 1995, 115-126. H. Lu, B. C. Ooi and K. L. Tan, Query Processing in Parallel Relational Database Systems (IEEE Computer Society Press, Los Alamos, USA, 1994). M. Bamha and G. Hains, A frequency adaptive join algorithm for Shared Nothing machines, Parallel and Distributed Computing Practices 3(3) (1999) 333-345. M. Bamha and G. Hains, A skew insensitive algorithm for join and multi-join operation on Shared Nothing machines, in Proc. 1 l th Int. Conf. on Database and Expert Systems Applications, London, UK, 2000, 644-653. M. Bamha, An optimal and skew-insensitive join algorithm for Shared Nothing machines, Research Report RR-LIFO-2000-05, LIFO, Universit6 d'Orl6ans, France, 2000. D. J. DeWitt and J. Gray, Parallel database systems : The future of high performance database systems, Communications of the ACM 35(6) (1992) 85-98. D. Schneider and D. DeWitt, A Performance Evaluation of Four Parallel Join Algorithms in a Shared-Nothing Multiprocessor Environment, in Proc. of the 1989 ACM SIGMOD Int. Conf. on Management of Data, Portland, USA, 1989, 110-121. D. B. Skillicorn, J. M. D. Hill and W. F. McColl, Questions and Answers about BSP, Scientific Programming 6(3) (1997) 249-274. M.S. Chen, P.S. Yu and K.L. Wu, Scheduling and Processor Allocation for the Execution of Multi-Join Queries, in Proc. of the Int. Conf. on Data Engineering, Los Alamos, USA, 1992, 58-67. M. S. Chen, M. L. Lo, P. S. Yu and H. C. Young, Using Segmented Right-Deep Trees for the Execution of Pipelined Hash Joins, in Proc. of the 18th Int. Conf. on Very Large Databases, Vancouver, Canada, 1992, 15-26.
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
55
T o w a r d s the H i e r a r c h i c a l G r o u p C o n s i s t e n c y for D S M s y s t e m s : an efficient w a y to share data objects L. Lef6vre a and A. Bonhomme a aINRIA / LIP (UMR CNRS, INRIA, ENS, UCB) Ecole Normale Sup6rieure - Lyon, France [email protected] We present a formal and graphical comparison of consistency models for Distributed Shared Memory systems, based on the programmer point of view. We conclude that most consistencies can be categorized into 3 models depending on their flexibility degree (none, 1 or 2). We propose a new consistency model that provides these 3 flexibility degrees : the Hierarchical Group Consistency (HGC) and present its deployment inside the DOSMOS DSM system. 1. INTRODUCTION Distributed Shared Memory (DSM) systems are now a well recognized alternative for the deployment of a large class of applications. Their main challenge is to manage data consistency while keeping good performances. During last decade, a lot of consistencies have been proposed [2, 3, 7, 10]. However, their definition is made from the designer or hardware point of view. Each definition is thereby often dependent on the DSM designer context. By using large scale clusters (hundreds or thousands of nodes), DSM have to face the scalability problem. How to provide scalable solutions for applications needing a virtual shared space with a large number of computing nodes? In particular, one issue that DSM systems have to address concerns minimizing replication (i.e. reducing the number of messages required to keep the memory consistent) and maximizing availability (i.e. increasing the number of local accesses) of shared data (objects). Our goal is to propose a new taxonomy for consistency models, that can be used by a programmer that deals with the implementation of a distributed application on top of a cluster of machines. This taxonomy is programmer-centric. For this, we use the framework introduced by Hu et al. [9], and we show that from the programmer's point of view, there are only 3 main consistency models. The other ones are just various implementations of those models. Based on this observation, we propose a graphical representation of those models. This representation is based on different degrees of flexibility of the consistency : flexibility about consistent moment (like Release Consistency) or about consistent data (like Entry Consistency). This representation outlines that so far, there is no consistency model with flexibility about consistent processes. Hence, we suggest the Hierarchical Group Consistency (HGC). Consistency is only maintained inside a group of processes rather than between all processes. Furthermore, for each group, various consistency rules can be applied, depending on the type of sharing performed
56 inside a group. This group structure is particularly interesting for heterogeneous clusters and allows the programmer to adapt the consistency management to the application (depending on the sharing degree, for example) and to the execution cluster (depending on the communication performances, for example). The Hierarchical Group Consistency has been implemented in the DOSMOS system [5, 12]. Like in most DSM systems based on weak consistency, DOSMOS provides Acquire and Release operations to set critical sections. However, with HGC, DOSMOS also allows the programmer to specify which data to share between which processes. This paper is organized as follows : Section 2 briefly presents the programmer-centric framework used to define consistency models, and introduces a formal comparison of existing models. Then Section 3 is devoted to a graphical 3D representation of the 3 consistency models. Next, Section 4 presents the Hierarchical Group Consistency model. Section 5 describes the hierarchical group deployment in the DOSMOS system. Section 6 concludes this paper and presents future directions. 2. THREE MEMORY CONSISTENCY MODELS FROM THE PROGRAMMER'S POINT OF VIEW
Research on data consistency has always been a hot topic because of its central position in areas like parallelism, operating systems, distributed systems, etc. However, if a lot of consistencies have been proposed for DSM systems, the formal context of the consistency concept has not been clearly specified. As a consequence, comparing consistencies remains a difficult challenge [1, 8]. In this section, we focus on the consistency concept based on the programmer's point of view. That means that a consistency model is defined based on how it is perceived by the programmer rather than how it is managed or implemented by the system. We base our work on the formal framework introduced by Hu et aL [9] in order to compare different models. Section 2.1 introduces this framework and defines the best known consistency within this framework. Section 2.2 shows how to compare these models with this framework. 2.1. Formal definition of consistency models In their article [9], Hu et all define a memory model as follows : Definition 1 A memory consistency model M is a pair (CM, SYNM) where CM is the set of possible memory accesses (read, write, synchronization) and S Y N M is an inter-process synchronization mechanism to order the execution of operations from different processes. The execution order of synchronization accesses determines the order in which memory accesses are perceived by a process. Accordingly, for each program, there are several possible executions. A program execution is defined as follows.
An execution of the program PRG under consistency model M, denoted as EM(PRG), is defined as an ordering of synchronization operations of the program. With the ordering of synchronization operations, the execution of all related operations are also ordered. Thus, we define the synchronization order of an execution.
Definition 2
The synchronization order of an execution EM(PRG) under consistency model M, denoted as SOM(EM(PRG)), is defined as the set of ordinary operation pairs ordered by the synchronization mechanism S Y N M of M. Definition 3
57 Hence, for any consistency model M, we can define CM and SOM(EM(PRG)). CM deals with how the programmer has to program, and SOM gives the rules used to generate the result. The basic Atomic Consistency (AC), the Sequential Consistency (SC) [11], the Release Consistency (RC) [7], the Lazy Release Consistency (LRC) [ 1], the Eager Release Consistency (ERe) [6], the Entry Consistency (EC) [3] and the Scope Consistency (SsC) [10] can all be defined within this framework. Furthermore, based on those definitions, we easily outline that RC, LRC and ERC have the same definition. That means that LRC and ERC are different implementation of the RC model. In effect, from the programmer's point of view, the results are the same for both implementations. Similarly, we state that SsC is a particular case of EC. As a result, in the following, we only consider AC, SC, RC and EC models. Finally, Hu et all. define a correct program as follows :
Definition 4 A program PRG is said to be correct for the consistency model M, iff for any possible execution EM(PRG), all ordinary conflicting access pairs are ordered by either the program order (PO) or by the synchronization order of execution SOM.
2.2. Formal comparison of consistency models In order to compare consistency models, we define the concept of model equivalence. Definition 5 M1 and M2 are said to be equivalent iff :
a) CM~ = CM~, b) a correct program PRG for M1 is also correct for M2, c) and 2 compatible executions EM1 (PRG) and EM2 (PRG) give the same result. EM1 (PRG) and EM2(PRG) are said to be compatible executions if there do not exist 2 synchronization operations (u, v) such that (u, v) E EM1 (PRG) and (v, u) r EM2 (PRG). Theorem 1 The Atomic Consistency model and the Sequential Consistency model are equivalent. Proof : By definition, CAC -- C s c . Furthermore, AC and SC provide a global order for all read/write operations. Consequently, they are both correct for any program. Finally, for any execution Eac, all accesses are ordered. Yet, Esc only orders operations concerning data accessed several times during the execution. Thus, Esc is included in Eac. For the remaining operations that are in EAC and not in Esc, they concern distinct data. The result is, therefore, the same whatever the execution order. AC and EC are equivalent. More generally speaking, we refer to them as the Strong Consistency Model.
Theorem 2 The Release Consistency model and the Entry Consistency model are not equiva-
lent. Proof : Let us give a counterexample. Let us consider the program and execution shown here. If for EC, x is not associated to l l, then PRG is correct for EC (accesses to x are ordered by the synchronization order), and gives
58 the result (a = 0, b - 1). However, for RC, PRG is also correct, but the result is (a = 1, b - 1). Thus EC and RC are not equivalent.
Corollary 1 The Strong Consistency model the Release Consistency model and the Entry Consistency model are three different consistency models. Proof : The strong consistency model does not define any synchronization variables. It is consequently not equivalent to EC or RC. 3. G R A P H I C A L COMPARISON OF THE THREE CONSISTENCY MODELS Once we have outlined 3 distinct models, we suggest to compare them with a graphical taxonomy based on the user's point of view. Considering a correct program for a model M, this taxonomy allows us to answer the following question : At each moment t of a program execution, if a given process Pi accesses a shared data d, will he read the last value written in the shared memory by another process pj ? In case of a positive answer, we say that at moment t, process pi is consistent with the process pj about the data d. Then, we represent each model with a 3D Accesses between visualization of the answer to this question barriers Time remaining a ~ All the Time by using various parameters (t, d, p). Hence, When Critic each model is represented by a volume made between Acquire and Release up of the triplets (t, d, p) for which the answer All Processes is yes. Basically, (t, d, p) is defined with reWho v v Processes linked Other processes spect to 3 axes: with synchronization When (t) : for time intervals of running apAll MemorySpace plication; What k---v----~ v ) W h a t (d) : for shared memory space; Memory Objects linked Memory Space with synchronization Who (iv) : for application processes. As shown in Fig.l, the axis discretization is Figure 1" When, who, what axes. based on shared objects' access patterns. Accordingly, if we normalize the axes, a plain cube means that a process, for all running execution, for all shared data, reads last value written by any other process. The strong consistency model perfectly fits in that definition and graphical representation (Fig. 2). Figures 3 and 4 graphically represent weak consistencies. With Release Consistency, an access performed between Acquire and Release operations is consistent with other processes on all accessible data. In a barrier case, all the processes have the same view of the shared memory. For the remaining execution, there are some conflict risks for any accessible data. The Entry Consistency model gives a slightly different result from the previous one. In effect, if for the barriers, the behavior is the same, it is rather different for an access in a critical section surrounded by an Acquire and Release operation. The process is consistent with the other processes only for the shared data associated with this synchronization.
59 Who
Who
When
/
WhaI
Figure 2. Strong consistency
]
When
/
What
Figure 3. Release consistency Figure 4. Entry consistency
4. TOWARDS A N E W C O N S I S T E N C Y M O D E L ? THE H I E R A R C H I C A L GROUP C O N S I S T E N C Y (HGC)
who Moving from a strong towards a weak consistency implies a decrease in the When axis decrease. The Entry Consistency model also relaxes the What axis. At this stage, only one dimension remains fixed" the processes one. It would be pertinent to [ make this axis flexible. For an application, all processes do not l Whe, use all shared data. Moreover, in order to provide scalable solutions, it could be interesting to synchronize with barriers a / What subset of running processes. Thus, we propose a new consistency model which groups Figure 5: Graphical representogether processes working on the same synchronization vail- tation of a new model with 3 ables. Such a model has the graphical representation shown in flexible axes Figure 5. Basically, the HGC model is similar to EC since data are associated to a synchronization variable. Furthermore, processes are also associated to this synchronization variable. Thus, let us consider a data modification performed inside a critical section managed by the synchronization variable l. Then, those modifications are forwarded only if they concern data associated to l, and only to processes associated to 1. Furthermore, in the HGC model, it is possible to perform some synchronization barrier for only a subset of the processes. Thus, the HGC model can be defined as follows : The Hierarchical Group Consistency model is defined by : - CHCC : {read(x), write(x), Acq(1), Rel(l), Sync(1)} - (u, v) E SOHcc(EHcc(PRG)) iff 3 a synchronization variable I to which u and v are associated such that: u is performed before Rel(1) and v is performed after Rel(1). OR u is performed before Sync(1) and v is performed after Sync(1). Definition 6
Theorem 3 The Hierarchical Group Consistency model is not equivalent to the Release Consistency model nor to the Entry Consistency model Proof : HGC introduces a new synchronization operation (the barrier restricted to a synchronization variable). Consequently, CHCC # CRc and CHcc # Czc. Thus, those models are not equivalent.
60 HGC limits coherence management costs of a shared data to some dedicated and explicitly associated processes. Global communications are restricted to processes really needing to have up to date copies of shared data. In this way, the Hierarchical Group Consistency allows to mix high availability of shared data combined with weak replication [4]. 5. IMPLEMENTATION ASPECT We implement the Hierarchical Group Consistency inside the DOSMOS framework (Distributed Shared Objects MemOry System) [5, 12]. DOSMOS is based on 3 kinds of processes : Application Processes (run application code), Memory Processes (manage memory access and consistency inside a group) and Link Processes (manage memory consistency between groups).. The policy of the DOSMOS system is to manage the consistency of an object only within the group of processes that frequently use this object. Thus DOSMOS introduces a hierarchical view of one's application processes by creating groups of processes that frequently share the same object. We define a group as a pair (G, O) where O is a set of objects and G a set of Application Processes sharing those objects. Each group has a group manager (Link Process : LP) responsible for the inter-group communications. Each group management is independent from the others. Concerning read and write operations to shared data, we classify the accesses into two categories : the intra-group accesses (the accessing process belongs to object's group) and the intergroup accesses between two distinct groups. Figure 6 shows an example of object accesses using group consistency. In this example, we have 2 sites A and B. In each site, processes share the same object (Y for site A and X for site B). The objects X and Y do not have the same consistency management : Release Consistency for Y and Lazy Release Consistency for X. In this context, we describe three different actions : First, one process of site A writes Y, then this process sends the new value of Y to the other members of the site A. The second action illustrates an inter-group access. One process of site/3 wants to read the object Y. Since it does not belong to Y's group, it sends its request to its group manager that forwards it to the group manager of site A. This one forwards again the request to the right process in the group that sends the Y value to the requesting process. The last action concerns an acquire of X : since X is managed in Lazy Release Consistency, the acquirer asks the last acquirer for the new value of X. For an intra-group access, we distinguish two cases : 9 If it is the first access for the process to that object, the group manager sends it a copy of this object; 9 Else, the process already has a local copy. Then, depending on the consistency implementation of the group, the process will directly work on this copy or not. For an inter-group access, the accessing process doesn't have a copy and will never have one during the whole execution of the application. It simply asks for the value of this object (for a read) or sends the new value (for a write) to the group manager.
61 SITE A
SITE B
Action 1 /'
Action 3 .. Acion 2
\
.
Object Y Release Consistency
Object X Lazy Release Consistency
Figure 6. Three actions supporting the group consistency : an intra-write in Release Consistency (action 1), an inter-read (action2), and an acquire in Lazy Release Consistency (action 3)
6. CONCLUSION This paper presents an original taxonomy of memory consistencies based on the programmer approach. This classification shows the interest of a new consistency relaxing un-useful memory management costs. The Hierarchical Group Consistency proposes to group together processes that frequently access the same shared data. This model allows a gain in performance without denying DSM programmability. The Group Consistency model implemented in DOSMOS is an original approach for improving DSM Systems. It allows the system to be adapted to the environment under which it performs (type of processors, communication network, application patterns, object size .... ) Depending on the environment, the policy concerning replication of objects will not be the same and consequently the group structure will be different. The hierarchical Group Consistency is well suited for large scale systems and could perfectly fit for multi-cluster and Grid applications. This model fits perfectly the requirements of various kind of networks. Depending on the latency, the bandwidth or other criteria, we can design an accurate group structure in order to reach the best compromise between replication and availability suited for the studied network. REFERENCES
[1]
[2]
S. V. Adve, A. L. Cox, S. Dwarkadas, R. Rajamony, and W. Zwaenepoel. A comparison of entry consistency and lazy release consistency implementations. In Proc. of the 2nd IEEE Symp. on High-Performance Computer Architecture (HPCA-2), pages 26-37, February 1996. S. V. Adve and K. Gharachorloo. Shared memory consistency models: A tutorial. IEEE Computer, 29(12):66-76, December 1996.
62 [3]
[4]
[5]
[6] [7]
[8]
[9] [ 10]
[ 11] [ 12]
Brian N. Bershad, Matthew J. Zekauskas, and W. A. Sawdon. The midway distributed shared memory system. In 38th IEEE International Computer Conference (COMPCON Spring'93), pages 528-537, February 1993. Alice Bonhomme and Laurent Lef6vre. How to combine strong availability with weak replication of objects ? In ECOOP98 : 12th Conference on Object Oriented Programming : Workshop on Mobility and Replication, Bruxelles, Belgium, July 1998. Lionel Brunie, Laurent Lef6vre, and Olivier Reymann. High performance distributed objects for cluster computing. In 1st IEEE International Workshop on Cluster Computing (IWCC '99), pages 229-236, Melbourne, Australia, dec 1999. IEEE Computer Society Press. John B. Carter, John K. Bennet, and Willy Zwaenepoel. Implementation and performance of MUNIN. ACM- Operating Systems Review, 25(5): 152-164, 1991. K. Gharachorloo, D. Lenoski, J. Laudon, P. Gibbons, A. Gupta, and J. Hennessy. Memory consistency and event ordering in scalable shared-memory multiprocessors. In 16th Annual Symposium on Computer Architecture, pages 15-26, May 1989. L. Higham, J. Kawash, and N. Verwaal. Defining and comparing memory consistency models. In Proc. of the 10th Int'l Conf. on Parallel and Distributed Computing Systems (PDCS-97), October 1997. W. Hu, W. Shi, and Z. Tang. A framework of memory consistency models. Journal of Computer Science and Technology, 13(2), March 1998. L. Iftode, J. E Singh, and K. Li. Scope consistency: A bridge between release consistency and entry consistency. In Proc. of the 8th ACMAnnual Symp. on Parallel Algorithms and Architectures (SPAA "96), pages 277-287, June 1996. L. Lamport. How to make a multiprocessor that correctly executes multiprocess programs. IEEE Trans. on Computers, C-28(9):690-691, September 1979. Laurent Lef6vre and Olivier Reymann. Combining low-latency communication protocols with multithreading for high performance dsm systems on clusters. In 8th Euromicro Workshop on Parallel and Distributed Processing, pages 333-340, Rhodes, Greece, Jan 2000. IEEE Computer Society Press.
Parallel Computing: SoftwareTechnology,Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
63
An operational semantics for skeletons* M. AldinuccP and M. Danelutto b ~Institute of Information Science and Technologies (ISTI)- National Research Council (CNR), Via Moruzzi 1, 1-56124 Pisa, Italy bDepartment of Computer Science, University of Pisa, Via Buonarroti 2, 1-56127 Pisa, Italy A major weakness of the current programming systems based on skeletons is that parallel semantics is usually provided in an informal way, thus preventing any formal comparison about program behavior. We describe a schema suitable for the description of both functional and parallel semantics of skeletal languages which is aimed at filling this gap. The proposed schema of semantics represents a handy framework to prove the correctness and validate different rewriting rules. These can be used to transform a skeleton program into a functionally equivalent but possibly faster version. 1. INTRODUCTION Skeletons have been originally conceived by Cole [8] and then used by different research groups to design high-performance structured parallel programming environments [5, 6, 12]. A skeleton may be modeled as an higher-order function taking one or more other skeletons or portions of sequential code as parameters, and modeling a parallel computation out of them. A skeletal program is a composition of skeletons. Skeletons can be provided to the programmer either as language constructs [5, 6, 7] or as libraries [3, 9, 11 ]. The formal description of a parallel, skeletal language involves at least two key issues: 1) the description of the input-output relationship of skeletons (functional semantics); 2) the description of the parallel behavior of skeletons. The functional semantics enables the definition of semantics-preserving program transformations [2, 4, 1, 10]. These transformations can also be driven by some kind of analytical performance models associated with skeletons [ 13], in such a way that only those rewritings leading to efficient implementations of the skeleton code are considered [2, 1]. Almost all frameworks previously cited have a formal functional semantics. But none of them provide a complete and uniform description of the parallel semantics. In this work we present a schema of operational semantics suitable for skeletal languages exploiting both data and stream parallel skeletons. The operational semantics is defined in term of a labeled transition system (LTS). It describes both functional and parallel behavior in a *This work has been partially supported by the Italian MIUR Strategic Project "legge 449/97" year 1999 No. 02.00470.ST97 and year 2000 No. 02.00640.ST97, and by the Italian MIUR FIRB Project GRID.it No. RBNE01KNFE
64
1. seqf<x,'r>g
e e..seqf<7_>e
2. farm A {x, ~-)e e A {x}o(e,x) "" farm A @}o(e,x) 3. pipe A 1 A 2 (X, T}e
g > A2 "~o(g,x) A1 (X)g "' pipe A 1 A 2
(T)g
4. comp A1 A2 (x, ~-)g g, A2 A1 (x)e "" comp A1 A2 (~-)e 5. map f~ A fd (X, 7"}g
g
' fc (Oz A) fd (X)g "" map f~ A fd (r)g
6. d&c ftc fc A fd <x, 7->e
g
d&c ftc fc A fd (Y>e :
7. whileftcA(x,T)e
h "" •
, d&c ftc f~ A fd <x>e "" d&c ftc fc A fd <7-)e a
e
i# (A~ v)
fc (a (dac ftc fc A/d)) fd (Y)e otherwise
e { whileftcA (A(x)e::@}e) > (x)e :: while ftc /k (7-}g
join
(e>• "" (T>e2 _t_ e2 joinc
A (x>e~ e2> (Y)e3 7Ze A<x>h
g2
, g
fd (X>g g> ~ g, "'" g g>
iff (ft~x) otherwise
relabel
E1 A E1
fc ~
L (~) f~ <x>e e (z>e
Ei - - ~ E~
Vil
<~>~- El
..... En
A " r
3 i , j 1 < i , j < n, gi = ~j :=v i = j ~>
<~>.--E~ ..... E"-.
e E2
> A E2
context
.... e
l..n
@
sp
r
Figure 1. Lithium operational semantics, x, y E value; a, 7- E values; u C values U {e}; E E exp; F C exp*; g, g i , . . . E label; 0 :label • value ~ label.
uniform and general way. We use a subset of the Lithium language as a test-bed to describe the methodology [3]. 2. L i t h i u m F O R M A L DEFINITION
Lithium extends the Java language by providing the programmer with both task parallel and data parallel skeletons [3] All the skeletons process a stream (finite or infinite) of input tasks to produce a stream of results. All the skeletons are assumed to be stateless, static variables are forbidden in Lithium code. No concept of "global state" is supported by the implementation, but the ones explicitly programmed by the user 2. The Lithium skeletons are fully nestable.
2Such as RMI servers encapsulating shared data structures.
65 Each skeleton has one or more parameters that model the computations encapsulated in the related parallelism exploitation pattern. Lithium manages two new types in addition to Java types: streams and tuples that are denoted by angled braces and "~ ~" braces respectively: value ::= A Java value values ::= value I value, values
stream ::= ( values ) ] (c) tuplek ::=~ streamb . . . , streamk
A stream represents a sequence (finite or infinite) of values of the same type, whereas the tuple is a parametric type that represents a (finite, ordered) set of streams. Actually, streams appearing in tuples are always singleton streams, i.e. streams holding a single value. The set of skeletons (A) provided by Lithium are defined as follows:
A::= seq f I farm A I pipe A~ A21 comp A~ A21 map fa A f~ I d&c ft~ fa A f~ I while ftc A where sequential Java functions (f, 9) with no index have type Object ---+ Object, and indexed functions (fc, fd, tic) have the following types: fr :tuplek ---+ stream; fd :stream ~ tuplek; ftc :value ---+ boolean. In particular, fr fd represent families of functions that enable the splitting of a stream in k-tuples of singleton streams and vice-versa. Lithium skeletons can be considered as a pre-defined higher-order functions. Intuitively, seq skeleton just integrates sequential Java code chunks within the structured parallel framework; farm and pipe skeletons model embarrassingly parallel and pipeline computations, respectively; comp models pipelines with stages serialized on the same processing element (PE); map models data parallel computations: fd decomposes the input data into a set of possibly overlapping data subsets, the inner skeleton computes a result out of each subset and the f~ function rebuilds a unique result out of these results; d&c models Divide&Conquer computations: input data is divided into subsets by fd and each subset is computed recursively and concurrently until the ft~ condition does not hold true. At this point results of sub-computations are conquered via the fc function, while skeleton model indefinite iteration. A skeleton applied to a stream is called a skeletal expression. Expressions are defined as follows: exp ::= A stream I/x exp I exp ] fr (c~ A) fd stream. The execution of a Lithium program consists in the evaluation of a A stream expression. 3. Lithium O P E R A T I O N A L S E M A N T I C S
We describe Lithium semantics by means of a LTS. We define the label set as the string set augmented with the special label "_L". We rely on labels to distinguish both streams and transitions. Input streams have no label, output stream are 2_ labeled. Labels on streams describe where data items are mapped within the system, while labels on transitions describe where they are computed. The Lithium operational semantics is described in Figure 1. The rules of the Lithium semantics may be grouped in two main categories, corresponding to the two halves of the figure. Rules 1-7 describe how skeletons behave with respect to a stream. These rules spring from the following common schema: Skel params (x, T)el
e~ >,T(x)e 2 .. Skel params (T)e 3
where Skel E [seq, farm, pipe, comp, map, d&c, while], and the infix stream constructor (a)e~ :: (~-)g~ is a non strict operator that sequences skeletal expressions. In general, f is a function
66 appearing in the params list. For each of these rules a couple of twin rules exists (not shown in Figure 1):
o~) Skel params (x, "r) _L> (C>_L"" Skel params (x, "r)z
~) Skel params <x>e~
> ~<x>~ 2
these rules manage the first and the last element of the stream respectively. Each triple of rules manages a stream as follows: the stream is first labeled by a rule of the kind c~). Then the stream is unfolded in a sequence of singleton streams and the nested skeleton is applied to each item in the sequence. During the unfolding, singleton streams are labeled according to the particular rule policy, while the transition is labeled with the label of the stream before the transition (in this case ~1). The last element of the stream is managed by a rule of the kind ~). Eventually resulting skeletal expressions are joined back by means of the :: operator. Let us show how rules 1-7 work with an example. We evaluate farm (seq f) on the input stream (xl,x2, x3). At the very beginning only the 2~ can be applied. It marks the begin of stream by introducing (c)x (empty stream) and labels the input stream with _k. Then rule 2 can be applied: (c)• "" farm (seq f) (x,,x2,x3)•
_L
, (c)• .. seq f (Xl>0 "" farm (seq f) (x2, x3>0
The head of the stream has been separated from the rest and has been re-labeled (from _L to 0) according to the 69(1, x) function. Inner skeleton (seq) has been applied to this singleton stream, while the initial skeleton has been applied to the rest of the stream in a recursive fashion. The re-labeling function (_9 : label x value --, label (namely the oracle) is an external function with respect to the LTS. It would represent the (user-defined) data mapping policy. Let us adopt a two-PEs round-robin policy. An oracle function for this policy would cyclically return a label in a set of two labels. In this case the repeated application of rule 1 proceeds as follows: (c)• :: seq f (X,>o :: seq f (X3) 0
farm (seq f) (x2, x3>o
0
,
1
> (c)• :: seq f (xx)0 :: seq f <x2>1
::
The oracle may have an internal state, and it may implement several policies of label transformation. As an example the oracle might always return a fresh label, or it might make decisions about the label to return on the basis of the x value. As we shall see, in the former case the semantics models the maximally parallel computation of the skeletal expression. Observe that using only rules 1-7 (and their twins) the initial skeleton expression cannot be completely reduced (up to the output stream). Applied in all the possible ways, they lead to an aggregate of expressions (exp) glued by the :: operator. The rest of the work is carried out by the six rules in the bottom half of figure 1. There are two main rules (sp and dp) and four auxiliary rules (context, relabel, joint and join). Such rules define the order of reduction along aggregates of skeletal expressions. Let us describe each rule:
sp (stream parallel) rule describes evaluation order of skeletal expressions along sequences separated by :: operator. The meaning of the rule is the following: suppose that each skeletal expression in the sequence may be rewritten in another skeletal expression with a certain labeled transformation. Then all such skeletal expressions can be transformed
67
farm (pipe (seq fl ) (seq f2)) (Xl, x2, x3, x4, x5, x6, x7 )
(1)
(C)_L "" farm (pipe (seq f l ) (seq f2)) (Xl,X2, X3,X4,X5,X6,X7)_L
(2)
(e)_L "" pipe (seq f l ) ( s e q f 2 ) ( x l ) o "" pipe (seq f l ) ( s e q f2)(X2)l "" pipe (seq f l ) ( s e q f2)(x3)o "" pipe (seq f l ) ( s e q f2)(X4)l "" pipe (seq f l ) ( s e q f2)(x5)o "" pipe (seq f l ) ( s e q f2)(x6}1 "" pipe (seq fz)(seq f2)(xT)o
(3) (e)• -" seq f2 7~02 seq fl (Xl)O "" seq f2 7~12 seq fl (x2)1 "" seq f2 TZ02 seq fl (x3)o "" s e q f 2 72,.12s e q f l (x4)1 "" seq f2 R02 seq fz (x5)o "" seq f2 7Z12 seq fz @6)1 "" seq f2 7~02 seq fl (x7)o
(4)
(e)_L "" s e q f 2 (fl Xl)02 "" s e q f 2 (fl x2)12 "" s e q f 2 7%2 s e q f l (x3)o "" s e q f 2 7~12 s e q f l (x4)1 "" seq f2 R02 seq fl (xs)o "" seq f2 7Z12 seq fl @6)1 "" seq f2 7~02 seq fl (x7)o
(5)
A_ "" 02 "" 02 "" (f2 (fl m4) >12 "" 02 ""
(6) (f2 (fl Xl), f2 (fl x2) f2 (fl x3), f2 (fl x4) f2 (fl x5), f2 (fl x6), f2 (fl x7) )..L
(7)
Figure 2. The semantic of a Lithium program: a complete example.
in parallel, provided that they are adjacent, that the first expression of the sequence is a stream of values, and that all the transformation labels involved are pairwise different.
dp (data parallel) rule describes the evaluation order for the fc (c~ A) fd stream expression. Basically the rule creates a tuple by means of the fd function, then requires the evaluation of all expressions composed by applying A onto all elements of the tuple. All such expression are evaluated in one step by the rule (apply-to-all). Finally, the fc gathers all the elements of the evaluated tuple in a singleton stream. Labels are not an issue for this rule.
relabel rule provides a relabeling facility by evaluating the meta-skeleton 7~e. The rule does nothing but changing the stream label. Pragmatically, the rule imposes to a PE to send the result of a computation to another PE (along with the function to compute).
context rules establishes the evaluation order among nested expressions, in all other cases but the ones treated by dp and relabel. The rule imposes a strict evaluation of nested expressions (i.e. evaluated the arguments first). The rule leaves unchanged both transition and stream labels with respect to the nested expression.
join rule does the housekeeping work, joining back all the singleton streams of values to a single output stream of values, join, does the same work on the first element of the stream. Let us consider a more complex example. Consider the semantics of f a r m (pipe fl f2) evaluated on the input stream (xl,x2, x3, x4, x5, x6, xT). Let us suppose that the oracle function returns a label chosen from a set of two labels. Pragmatically, since farm skeleton
68 represent the replication paradigm and pipe skeleton the pipeline paradigm, the nested form farm (pipe fl f2) basically matches the idea of a multiple pipeline (or a pipeline with multiple independent channels). The oracle function defines the parallelism degree of each paradigm: in our case two pipes, each having two stages. As shown in Figure 2, the initial expression is unfolded by rule 2~ ((1) --~ (2)) then reduced by many applications of rule 2 ((2) ---~* (3)). Afterward the term can be rewritten by rule 3 ((3) --~* (4)). At this point, we can reduce the formula using the sp rule. sp requires a sequence of adjacent expressions that can be reduced with differently labeled transitions. In this case we can find just two different labels (0, 1), thus we apply sp to the leftmost pair of the previuos expressions : (E)_L "" seqf2 ~02 seq fl (Xl)0 "" seqf2 ~12 seq fl ( X 2 ) l "" seqf2 7~02seq fl (x3)0 ..... (e)_t_ :: seq f2(fl Xl)02 :: seqf2 (fl X2)12 :: seqf2 7Z02seq fl (X3)o : : ' ' " Observe that due to the previous reduction ((4) 4 (5) in Figure 2) two new stream labels appear (02 and 12). Now, we can repeat the application of sp rule as in the previous step. This time, it is possible to find four adjacent expression that can be rewritten with (pairwise) different labels (0, 1, 02, 12). Notice that even on a longer stream this oracle function never produces more than four different labels, thus the maximum number of skeletal expressions reduced in one step by sp is four. Repeating the reasoning the we can completely reduce the formula to a sequence of singleton streams ((5) 4 " (6)), that, in turn can be transformed by many application of the join rule ((6) 4 " (7)). Let us analyze the whole reduction process: from the initial expression to the final stream of values we applied three times the sp rule. In the first application of sp we transformed two skeletal expression in one step; in the second application four skeletal expressions have been involved; while in the last one just one expression has been reduced 3. The succession of sp applications in the transition system matches the expected behavior for the double pipeline paradigm. The first and last applications match pipeline start-up and end-up phases. As expected the parallelism exploited is reduced with respect to the steady state phase, that is matched by the second application. A longer input stream would rise the number of sp applications, in fact expanding the steady state phase of modeled system. 4. PARALLELISM AND LABELS The first relevant aspect of the proposed schema is that the functional semantics is independent of the labeling function. Changing the oracle function (i.e. how data and computations are distributed across the system) may change the number of transitions needed to reduce the input stream to the output stream, but it cannot change the output stream itself. The second aspect concerns parallel behavior. It can be completely understood looking at the application of two rules: dp and sp that respectively control data and stream parallelism. We call the evaluation of either dp or sp rule apar-step, dp rule acts as an apply-to-all on a tuple of data items. Such data items are generated partitioning a single task of the stream by means of an user-defined function. The parallelism comes from the reduction of all elements in a tuple (actually singleton streams) in a single par-step. A single instance of the sp rule enable the parallel evolution of adjacent terms with different labels (i.e. computations running on distinct PEs). The converse implication also holds: many transitions of adjacent terms with different 3Second and third sp application are not shown in the example.
69 labels might be reduced with a single sp application. However, notice that sp rule may be applied in many different ways even to the same term. In particular the number of expressions reduced in a single par-step may vary from 1 to n (i.e. the maximum number of adjacent terms exploiting different labels). These different applications lead to both different proofs and parallel (but functional) semantics for the same term. This degree of freedom enables the language designer to define a (functionally confluent) family of semantics for the same language coveting different aspects of the language. For example, it is possible to define the semantics exploiting the maximum available parallelism or the one that never uses more than k PEs. At any time, the effective parallelism degree in the evaluation of a given term in a par-step can be counted by inducing in a structural way on the proof of the term. Parallelism degree in the conclusion of a rule is the sum of parallelism degrees of the transitions appearing in the premise (assuming one the parallelism degree in rules 1-7). The parallelism degree counting may be easily formalized in the LTS by using an additional label on transitions. The LTS proof machinery subsumes a generic skeleton implementation: the input of skeleton program comes from a single entity (e.g. channel, cable, etc.) and at discrete time steps. To exploit parallelism on different tasks of the stream, tasks are spread out in PEs following a given discipline. Stream labels trace tasks in their journey, and sp establishes that tasks with the same label, i.e. on the same PE, cannot be computed in parallel. Labels are assigned by the oracle function by rewriting a label into another one using its own internal policy. The oracle abstracts the mapping of data onto PEs, and it can be viewed as a parameter of the transition system used to model several policies in data mapping. As an example a farm may take items from the stream and spread them in a round-robin fashion to a pool of workers. Alternatively, the farm may manage the pool of workers by divination, always mapping a data task to a free worker (such kind of policy may be used to establish an upper bound in the parallelism exploited). The label set effectively used in a computation depends on the oracle function. It can be a statically fixed set or it can change cardinality during the evaluation. On this ground, a large class of implementations may be modeled. Labels on transformations are derived from label on streams. Quite intuitively, a processing element must known a data item to elaborate it. The relabeling mechanism enables to describe a data item re-mapping. In a par-step, transformation labels point out which are the PEs currently computing the task. 5. CONCLUSION We propose an operational semantics schema that can be used to describe both functional and parallel behavior of skeletal programs in a uniform way. This schema is basically a LTS, which is parametric with respect to an oracle function. The oracle function provides the LTS with a configurable label generator that establishes a mapping between data and computation and system resources. We use the Lithium - a skeletal language exploiting both task and data parallelism- as a test-bed for the semantics schema. We show how the semantics (built according to the schema) enables the analysis of several facets of Lithium programs such as: the description of functional semantics, the comparison in performance and resource usage between functionally equivalent programs, the analysis of maximum parallelism achievable with infinite or finite resources. To our knowledge, there is no other work discussing a semantics with the same features in parallel skeleton language framework (and, in general, in the structured parallel programming world as well).
70 REFERENCES
[ 1]
[2]
[3] [4]
[5]
[6] [7] [8] [9]
[10]
[ 11 ] [12] [13]
M. Aldinucci. Automatic program transformation: The Meta tool for skeleton-based languages. In S. Gorlatch and C. Lengauer, editors, Constructive Methods for Parallel Programming, Advances in Computation: Theory and Practice, chapter 5, pages 59-78. Nova Science Publishers, NY, USA, 2002. M. Aldinucci and M. Danelutto. Stream parallel skeleton optimization. In Proc. of the 11th lASTED lntl. Conference on Parallel and Distributed Computing and Systems (PDCS'99), pages 955-962, Cambridge, Massachusetts, USA, November 1999. IASTED/ACTA press. M. Aldinucci, M. Danelutto, and P. Teti. An advanced environment supporting structured parallel programming in Java. Future Generation Computer Systems, 19(5):611-626, 2003. M. Aldinucci, S. Gorlatch, C. Lengauer, and S. Pelagatti. Towards parallel programming by transformation: The FAN skeleton framework. Parallel Algorithms and Applications, 16(2-3):87-122, 2001. P. Au, J. Darlington, M. Ghanem, Y. Guo, H.W. To, and J. Yang. Co-ordinating heterogeneous parallel computation. In L. Bouge, P. Fraigniaud, A. Mignotte, and Y. Robert, editors, Proc. of Euro-Par 1996, pages 601-614. Springer-Verlag, 1996. B. Bacci, M. Danelutto, S. Pelagatti, and M. Vanneschi. SklE: a heterogeneous environment for HPC applications. Parallel Computing, 25(13-14): 1827-1852, December 1999. G.H. Botorog and H. Kuchen. Efficient high-level parallel programming. Theoretical Computer Science, 196(1-2):71-107, April 1998. M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computations. Research Monographs in Parallel and Distributed Computing. Pitman, 1989. M. Danelutto and M. Stigliani. SKElib: parallel programming with skeletons in C. In A. Bode, T. Ludwing, W. Karl, and R. Wismiiller, editors, Proc. of Euro-Par 2000, number 1900 in LNCS, pages 1175-1184. Springer-Verlag, September 2000. S. Gorlatch, C. Lengauer, and C. Wedler. Optimization rules for programming with collective operations. In Proc. of the 13th Intl. Parallel Processing Symposium & l Oth Symposium on Parallel and Distributed Processing (IPPS/SPDP'99), IEEE Computer Society Press, pages 492-499, 1999. H. Kuchen. A skeleton library. In B. Monien and R. Feldmann, editors, Proc. of Euro-Par 2002, number 2400 in LNCS, pages 620-629. Springer-Verlag, 2002. J. Srrot and D. Ginhac. Skeletons for parallel image processing: an overview of the SKIPPER project. Parallel Computing, 28(12):1685-1708, December 2002. D. B. Skillicom and W. Cai. A cost calculus for parallel functional programming. Journal of Parallel and Distributed Computing, 28:65-83, 1995.
Parallel Computing: SoftwareTechnology,Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
71
A Programming Model for Tree Structured Parallel and Distributed Algorithms and its Implementation in a Java Environment* H. Moritsch a a School of Business, Economics, and Computer Science, University of Vienna, Brfinner Strage 72, A-1210 Vienna, Austria [email protected] We describe the Distributed Active Tree, a programming model which allows for an objectoriented, data parallel formulation of a whole class of parallel and distributed tree structured algorithms. It provides collective communication with varying synchronization conditions and shared address space through Accessible objects, representing the nodes accessed within a communication operation. We developed a Java implementation and employed it in the parallelization of nested Benders decomposition, a solution technique for large stochastic programming problems such as multistage decision problems. The optimization algorithm is part of a high performance decision support tool for asset and liability management, the Aurora Financial Management System. We discuss the architecture and coordination layer of our implementation and present experimental results obtained on a Beowulf SMP-cluster and on a network of workstations, including a comparison of a synchronous and an asynchronous version of the algorithm. 1. INTRODUCTION Tree structured problems occur in various applications areas, for example in portfolio management and reservoir optimization. In fact, they represent an important class of the problems dealt with in finance. The Aurora Financial Management System under development at the University of Vienna is a decision support tool for asset and liability management employing parallel solvers for large multistage stochastic optimization problems [11]. Using decomposition techniques, the whole optimization problem is represented as a set of small problems defined at the nodes of a scenario tree. By iteratively solving their local problems and updating them using information from neighbor nodes, all nodes contribute to the overall solution. The decomposition method has been implemented in parallel as a distributed active tree structure. The programming model behind allows to formulate a wide class of tree structured algorithms at a high level with transparent remote accesses, communication and synchronization. The remainder of this paper is organized as follows. Section 2 describes the distributed active tree model, and section 3 discusses the implementation as a Java class library on top of Remote Method Invocation. Section 4 presents an application in the field of stochastic optimization, *The work described in this paper was supported by the Austrian Science Fund as part of the Special Research Program No. F011 AURORA "Advanced Models, Applications and Software Systems for High Performance Computing".
72 a tree structured decomposition algorithm, and reports on initial experiments conducted on a network of Sun workstations and on a Beowulf SMP-cluster. Section 5 concludes the paper. 2. T H E DISTRIBUTED ACTIVE TREE M O D E L The Distributed Active Tree (DAT) is a structure which executes a parallel program. The program is defined as a set of cooperating algorithms associated with nodes of a the tree. The DAT is a distributed data structure containing active tree node objects, i.e. nodes have their own threads of control. Whereas in principle they could execute different programs, in this model they execute the same (eventually parameterized with particular properties of the node). As they operate on different data, the DAT programming model is a Single Program, Multiple Data (SPMD) model. The set of tree nodes is distributed onto a set of compute nodes (physical processors), according to a specified mapping defining the distribution of the tree data. The nodes altogether perform a parallel algorithm, thus the whole tree can be as a distributed processing unit (an abstract machine). Tree nodes can be mapped onto compute nodes individually or at the level of complete subtrees. With emphasis on iterative tree structured algorithms, the node activity is modeled as a while-loop. It can contain additional nested loops. Conceptually, all nodes are simultaneously active and exchange data with other nodes. Due to synchronization requirements, a node can be in a wait state. For the coordination among tree nodes, the model provides the abstraction Accessible. It represents the set of tree nodes (element nodes) which are involved in a communication operation, e.g. the successors, the predecessor, or the union of these. Data accesses to other nodes are performed through calling methods of the respective Accessible objects. Data transfered to another node is buffered at the target node. The target.put(data) method copies the object data into buffers at the element nodes of the Accessible target. The source.get(allOrAny) method, called at a node, retrieves data (put by the element nodes of the Accessible source) from the buffer and copies it into local data structures of the node. The get method either operates, depending on the value of its parameter, in a mode in which it waits until data has arrived from all element nodes, or in a mode in which it waits for data from at least one node; this allows for diverse synchronization constraints. Accessing a set of nodes in a single operation provides a kind of collective communication. The communication primitives are transparent with respect to the distribution of the tree: nodes residing on the same compute node (local nodes) are addressed in the same way as nodes residing at a different compute node (remote nodes); the programmer maintains a shared memory view of the whole tree. For the execution of many node activities on one compute node, different scheduling strategies are possible. If every tree node is associated with a thread of the underlying runtime environment, the scheduling is just delegated. Following a different strategy, one single thread cyclically executes one iteration on all tree nodes, then the next iteration, and so on. Combinations of these strategies are possible, e.g. there are m threads at a compute node, each of which is responsible for n tree nodes. 3. I M P L E M E N T A T I O N IN JAVA We developed an implemention in Java because of Java's portability and other features such as the object-oriented programming model, robustness, automatic memory management and support for multithreading and network programming. In particular, we assume the financial
73 application to be used in a heterogenous computing environment. The Java implementation also serves as a basis for a grid-enabled version. At the level of the Java language, the DAT model is defined as a set of interfaces, organized in several packages: a u r o r a , d a t (specification of the tree structure), a u r o r a , c-tat, a l g (specification of the algorithm), a u r o r a . d a t , c o o r d (coordination layer implementation), a u r o r a , d a t . d i s t (tree distribution specification), and a u r o r a , d a t . s c h e d (loop scheduling method); the Java classes implementing these interfaces altogether build a concrete DAT. Every tree node is an instance of a class which implements the A l g o r i t h m N o d e interface and overrides the methods i t e r a t i o n and c o n d i t i o n of the L o o p A c t i v i t y interface, thus specifying the body and termination condition of the main loop. Associated to every tree node there is a coordination object (an instance of a class implementing the Coordination interface), which maintains a buffer for incoming data and provides Accessible objects to the tree node. The scheduling of loops is specified in an implementation of the L o o p S c h e d u l e r interface which operationally defines a mapping from the set of loops to be executed on a compute node onto a set of Java threads. An instance of this class creates the threads, assigns loop iterations to them, and starts them. The tree, including its distribution, is specified in an implementation of the D i s t r i b u t e d T r e e S p e c i f i c a t i o n interface. Using Java code, arbitrary mappings can be defined. At runtime, the DAT comprises, at each compute node, a set of A l g o r i t h m N o d e objects with their corresponding C o o r d i n a t i o n objects, a L o o p S c h e d u l e r - , and a DistributedTreeSpecification object. In the m a i n function at a compute node, these objects are created and combined within a C o m p u t e N o d e object: new C o m p u t e N o d e (number, AlgorithmNode-class-name, Coordinationclass-name, new DistributedTreeSpecification-class (..) ) . r u n (new LoopSchedulerclass (..) ). The implementation of the coordination forms a separate layer in the DAT below the algorithm layer, dealing with all intra- and inter-processor communication and-synchronization. It will employ underlying mechanisms such as Sockets, RMI, CORBA, and even MPI (via the Java Native Interface). Via the C o o r d i n a t i o n interface, different coordination implementations can be used by an algorithm. We developed an implementation on top of Java Remote Method Invocation (RMI). The class R D C (RMI-based DAT Coordination) has a s y n c h r o n i z e d method add, which appends new data to the input buffer. Within the p u t operation, local nodes are accessed through shared memory via calls of the a d d method at the coordination objects of the target nodes. Remote nodes are accessed through calling (via RMI) a remote method of the target C o m p u t e N o d e object, which then locally calls the a d d method at the target coordination objects. We describe the synchronization of tree nodes and its implementation as a collection of state dependent actions using Java's built-in synchronization features [7]. Every tree node performs its activity in a separate thread. With regard to coordination, the logical state space of a node (more precisely, of the coordination object) consists of the following states. In the Active state, the node is performing computations of the node algorithm. When it starts waiting for data from other nodes, it enters the Wait state. When new data has arrived, it enters the Transfer
74
add ~ [
_~rI
i
add ( ] [element][" TransferJ( :"~ [element&&
Active a)
la~176 >
public class RDC implements Coordination { 9. . public class RDA implements Accessible { 9. . p u b l i c v o i d get ( b o o l e a n a l l O r A n y ) { s t a t e C h a n g e d (this, a l l O r A n y ) ; s y n c h r o n i z e d (RDC. this) { if (state == WAIT) try { RDC.this.wait(); } c a t c h ...
} do {
t r a n s f e r (this) ; s t a t e C h a n g e d (this, false) ; } w h i l e (state == T R A N S F E R ) ; }//get } }
b)
Figure 1. a) Logical states of the coordination object b) Synchronization within the get: operation
state. In this state, the buffer contents are copied into local data structures. States are left either when the activity associated with a state is finished, or when the a d d method has been called, i.e. another node has added data to the buffer. The statechart diagram in Figure 1 shows the additional substates ActiveNew and TransferNew, which are entered when new data has arrived during the state activity. This allows for optimizations: in the case of Active, Wait can be skipped, and in the case of Transfer, the new data will just be considered during the current g e t operation. In addition, two guard conditions control the state transitions. Incoming data is regarded relevant only if the node, which has put it into the buffer, is an element of the Accessible performing the g e t operation. In the case of a l 10rAny== t r u e , data is regarded available only if arrived from all element nodes of the Accessible. The transition diagram is implemented in the synchronized method R D C . s t a t e C h a n g e d , which writes the instance variable s t a t e ; it is called in the g e t method, and, from the source node thread, at the end of the a d d method. The code excerpt in Figure 1 shows the implementation of the g e t operation. If the state has changed to Wait, the node thread is caused to wait; s t a t e C h a n g e d calls n o t i f y just in the case that Wait has been left. The s y n c h r o n i z e d method RDC. t r a n s f e r copies the buffered data objects into the local data structures of the A l g o r i t h m N o d e object. These are not visible to the coordination object, because they are in the algorithm layer; the copy operations are methods of the algorithm data objects and, via a conversion mechanism, called by t r a n s f e r .
4. APPLICATION EXAMPLE: STOCHASTIC OPTIMIZATION A multistage decision problem, for example in portfolio optimization, is defined as a set of node problems, each of which is associated with a node of a scenario tree describing the possible developments of the environment. At every node, based on the scenario, a node specific objective function is formulated. The optimization of a combination of all node specific functions is the ultimate goal:
75
class BendersNode implements AlgorithmNode {
[.................. nil. C updateR,Q ~'~
+
I when(newCutornewSolution) l
LP ip;
+
C so've'P ) [infeasible]+
[feasible]
+ ,
master
i
l ........... > . . . . .
compute C feasibi
S o l u t i o n master; Vector Q[], R[]; Coordination coord; Accessible from; boolean synchronous; boolean forward;
- [objVal> master.objVal]3~
d
~SIRV~
(forward ? coord.getPredecessor() : coord.getSuccessors())
- coord.getAll()); // update master, R, Q ... ip = n e w L P ( m a s t e r , Q , R ) ; // create and solve LP ... coord.getSuccessors().put(solution); // send s o l u t i o n ... coord.getPredecessor () .put (cut) ; // send cut
Ise]
from.get(synchronous);
]
a)
// sweep direction
// linear program object
// loop body of the decomposition algorithm public void iteration(int iterationNumber) { S o l u t i o n solution; // s o l u t i o n of LP Cut cut; // cut for master from = (synchronous ?
{cardinality=#slaves]
[else] ~ (~ [termination]
// s o l u t i o n from master // cuts from slaves // coordination object // source of get o p e r a t i o n // is synchronous version
}
b)
Figure 2. a) Node algorithm for nested Benders decomposition b) DAT implementation
Minimize ~
f~ (z~)
hEN"
V(n E iV') {
TnxpredC (n)xnSn + dnxn = bn
where N" denotes the set of nodes in the tree. A node n E N" is associated with a local objective function f~ wrt. decision variables zn. An, bn, and Tn describe the constraints representing the dependency on the decision variables at the predecessor node pred(n), Sn describes constraints local to node n, for example budget constraints. In the case of the root node, Tn = 0. In the nested Benders" decomposition method, every node performs an iterative procedure, acting as a master, slave, or both of them [ 1]. The master solves a linear program, sends the solution to the slaves (the successor nodes), and receives from them additional constraints (cuts) which will improve the master's solution in the next iteration. In the synchronous version of the method, each node receives the solution from its master, builds and solves the local problem, then sends the solution to its slaves, and waits for cuts from every slave. The solution process is a sequence of forward- and backward sweeps over the whole tree. In the asynchronous version, every node waits until it has received data from at least one of slaves or from the master [9]. Figure 2 shows the asynchronous node algorithm and fragments of the algorithm layer code for both versions. Note that the Accessible variable f r o m is used to express the different cases of getting data from neighbor nodes. Figure 3 shows the results of initial experiments with a multistage financial portfolio optimization problem, and tree sizes from 127 nodes to 255 nodes. Every tree node runs as a separate thread; the nested Benders decomposition code calls within a synchroni z e d block, via JNI and C, the E04MFF NAG Fortran LP routine. For the distribution of the tree, the root node and its descendents, up to a specific depth in the tree, are mapped to the "root" processor. The subtrees emanating from the nodes at that depth as a whole are mapped to the remaining
76 10,0 9,0 8,0 7,0 6,0
5,0 ~ 4,0 3,0 2,0 1,0 0,0
8,0 A__ ........... A "-.. "'- .. "'-
(3-(3127 ~ 127 sync a - w 255 ~ 255 sync A-A511 .. -..
~5,0
-,~
o=4,0
[] . . . .
7,0
A.. ..........
6,0
""-...... tSl..
". . . . .
(3- 9 127 ~ 127 sync G-D 255 ~ 255 sync A-A511 ~ 511 sync
~3,0 O- . . . . . . . . . . . . . . . . . . .
2,0
~ ....................
"-... (3- . . . .
~ ....................
1,0 1
3 compute nodes
a)
5
0,0
1
3 compute nodes
5
b)
Figure 3. Execution times on a) network of Sun workstations b) Beowulf cluster compute nodes; they require communication with the root processor only. The initial size of the local constraint matrices were 7 x 6, the values for the depth parameter d of the distribution of nt tree nodes onto n~ compute nodes were chosen (nt]nc, d) = (127]5, 4), ({12713 , 255]5}, 5), and ({255]3,511]5}, 6). The results expose some properties of the algorithm. In the synchronous version, the tree nodes perform on average less iterations than in the asynchronous one (1); this is reflected in shorter execution times on a single compute node. In the asynchronous version, the tree nodes spend less time in waiting for new data (2). When running in parallel, a compute node as a whole is idle, when all of its tree nodes are waiting. Shorter processor idle times result in larger speedups for the asynchronous version. Still, these are rather small due to small node problem sizes, resulting in weak computation/communication ratios; we expect better numbers for larger problems (which currently suffer from numerical instabilities). The number of iterations per tree node increases also with the tree size (3). The additional increase in the number of communication operations is seen as one reason for longer execution times of the asynchronous version with larger trees in parallel. In addition, low level effects such as the thread scheduling overhead of the particular runtime system (JVM and operating system) have to be taken into account. 5. CONCLUSIONS In this paper we have presented the Distributed Active Tree programming model, which allows the application programmer to express tree structured iterative algorithms at a high level in a natural way. The model can be implemented with a variety of protocols, communication mechanisms, including web services and grid technology. We described an implementation on top of Java/RMI, with the sole use of Java's multi threading, communication and synchronization mechanisms. As a case study, a parallel decomposition technique for solving large scale stochastic optimization problems has been implemented, in a synchronous and in an asynchronous version. Within the Aurora project, the nested Benders decomposition has been parallelized using OpusJava [6], and the DAT with the coordination implemented on top of JavaSymphony [5]. An alternative optimization algorithm is described in [2], parallel decomposition techniques
77 and their implementation in [4, 10, 13, 14]. For writing high performance applications in Java, language extensions have been defined. Spar [12] provides extensive support for arrays such as multidimensional arrays, specialized array representations, and tuples. It supports data parallel programming and allows via annotations for an efficient parallelization. Titanium [ 15] is a Java dialect with support for multidimensional arrays. It provides an explicitly parallel SPMD model with a global address space and global synchronization primitives. HPJava [3] adds SPMD programming and collective communication to Java. DAT does not change the syntax or semantics of Java and is specifically targeted at a high level formulation of tree structured iterative algorithms and a highly modular architecture. A classification of language extensions, libraries, and JVM modifications for high performance computing in Java is given in [8]. The implementation of the nested Benders decomposition algorithm is subject to optimization along various dimensions, such as the tree distribution (including dynamic pruning and rebalancing of the tree), the loop scheduling strategy, the underlying communication mechanism, LP solution techniques (warm start of the LP solver), and the mapping of scenario tree nodes to local problems. The Distributed Active Tree is a tool to implement and combine variants of all contributing parts, to study the interplay of effects and to achieve a high performance application. REFERENCES
[1] [2]
[3] [4] [5]
[6]
[7] [8]
[9]
[10]
J.F.Benders. Partitioning procedures for solving mixed-variable programming problems. Numer. Math., 4:238-252, 1962. S.Benkner, L.Halada, M.Lucka. Parallelization Strategies of Three-Stage Stochastic Program Based on the BQ Method. Parallel Numerics'02, Theory and Applications. Ed. R.Trobec, P.Zinterhof, M.Vajtersic, A.Uhl, pp.77-86, October,23-25, 2002. B.Carpenter, G.Zhang, G.Fox, X.Li, Y.Wen. HPJava: data parallel extensions to Java. Concurrency: Practice and Experience 10,11-13: 873-877, 1998. M.A.H.Dempster, R.T.Thompson. Parallelization and aggregation of nested benders decomposition. Annals of Operations Research, 81:163-187, 1998. T.Fahringer, A.Jugravu, B.Di Martino, S.Venticinque, H.Moritsch. On the Evaluation of JavaSymphony for Cluster Applications. In Proceedings of the IEEE International Conference on Cluster Computing (Cluster2002). September 2002, Chicago, Illinois. E.Laure, H.Moritsch. Portable Parallel Portfolio Optimization in the Aurora Financial Management System. In Proceedings of SPIE ITCom 2001 Conference: Commercial Applications for High-Performance Computing. August 2001, Denver, Colorado. D.Lea. Concurrent Programming in Java. Addison-Wesley, Reading, Mass., 1997. M.Lobosco, C.Amorim, O.Loques. Java for high-performance network based computing: a survey. Concurrency and Computation: Practice and Experience (Ed. Geoffrey Fox), John Wiley & Sons, 14:1-31, 2002. H.Moritsch, G.Ch.Pflug, M.Siomak. Asynchronous nested optimization algorithms and their parallel implementation. In Proceedings of the International Software Engineering Symposium. Wuhan, China, March 2001. Soren S.Nielsen and S.A.Zenios. Scalable parallel Benders decomposition for stochastic linear programming. Parallel Computing, 23:1069-1088, 1997.
78 [ 11] G.Ch.Pflug, A.~;wi~tanowski, E.Dockner, H.Moritsch. The AURORA Financial Management System: Model and Parallel Implementation Design. Annals of Operations Research, 99:189-206, 2000. [12] C.van Reeuwijk, F.Kuijlman, H.J.Sips. Spar: a set of extensions to Java for scientific computation. Concurrency and Computation: Practice and Experience 15,3-5: 277-297, 2003. [ 13] A.Ruszczynski. Parallel decomposition of multistage stochastic programming problems. Math. Programming, 58: 201-228, 1993. [ 14] H.Vladimirou, S.A.Zenios. Scalable parallel computations for large-scale stochastic programming. Annals of Operations Research, 90:87-129, 2000. [15] K.Yelick, L.Semenzato, G.Pike, C.Miyamoto, B.Liblit, A.Krishnamurthy, P.Hilfinger, S.Graham, D.Gay, P.Colella, A.Aiken. Titanium: a high-performance Java dialect. Concurrency and Computation: Practice and Experience 10,11-13: 825-836, 1998.
Parallel Computing: Software Technology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
79
A Rewriting Semantics for an Event-Oriented Functional Parallel Language F. Loulergue a aLaboratory of Algorithms, Complexity and Logic, 61, avenue du G6n6ral de Gaulle, 94010 Cr6teil cedex, France This paper presents the design of the core of a parallel programming language called CDS*. It is based on explicitly-distributed concrete data structures and features compositional semantics, higher-order functions and explicitly distributed objects. The denotational semantics is outlined, the (equivalent) operational semantics is presented and a new realization of the latter is given as a rewriting system. 1. I N T R O D U C T I O N Resource-aware programming tools and programs with measurable utilization of network capacity, where control parallelism can be mixed with data parallelism are advocated. In [8, 5] we proposed semantic models for languages whose semantics is functional and whose programs are explicitly parallel. Such languages address the above-stated requirements by expressing data placement, and hence communications explicitly, allowing higher-order functions to take placement strategies and generic computations as arguments, allowing higher-order functions to monitor communications within other functions, and yet avoid the complexity of concurrent programming. We also have introduced the elements of an explicitly-parallel functional language CDS*: denotational semantics, operational semantics and full abstraction result. It is inspired by Berry and Curien's sequential language CDS [1] but uses Brookes and Geva's generalized concrete data structures (gcds) [2], for compatibility with parallel execution. Here we present a detailed rewriting semantics. Unlike Multilisp [6] or CD-Scheme [10], CDS* causes no dynamic process creation in order to facilitate performance prediction. Moreover all the possible events of a CDS* program are declared statically together with their physical processor (or static process) location. We call this feature explicit processes and share it with Caml-Flight [4] and the proposed BSP library standard [7]. CDS* improves on Caml-Flight and BSP by its compositional semantics. User-defined functions define dependencies between (explicitly-located) events and thus prescribe the communications generated by their application. As observed by Berry and Curien, a program of functional type on concrete structures can observe event dependencies inside a functional argument and thus compare different algorithms before applying them. In our context, this means that a second-order function can compare first-order function's load-balancing and communication properties before applying them. This goes one step beyond the proposal of Mirani and Hudak [9] which was to use explicit and programmable schedules but to obtain cost information from system calls. Compared with data-flow programming CDS* is a gener-
80
alization by its inclusion of higher-order functions but it is beyond the scope of this paper to make an exact comparison. Let us simply observe that connecting deterministic programs with streams is a special case of programming with section. The following sections present the language's denotational and operational semantics and then the rewriting semantics. 2. CDS* AND G E N E R A L I Z E D C O N C R E T E DATA STRUCTURES W I T H INDICES. A CDS* program is made of type definitions, followed by term definitions and a request to evaluate a given term. The definition of a type inference system, ofparameterized and polymorphic types is important open problem. As a result our (strong) typing is explicit and monomorphic. A CDS* type is a gods with explicit process indices. A gods is a set of cells with allowed values for each one, seen as a game to be played. Certain cells can be filled at any time and others can only be filled after being enabled by specific finite sets of filling events. A valid gods configuration is called a state and the object of the game is to compute states monotonically: never erasing or changing the value given to a cell. As such, states are generalized traces/streams. A program of functional type r --+ r ' describes a continuous function from states of gods r to states of gods r'. Program syntax is almost standard for a functional language except for one crucial difference: there is no A-binding. Elementary terms are simply an enumeration of finite states, as sets of events. Now unlike standard functional programs which denote abstract functions, a CDS* program denotes a function between concrete domains which can be (concretely) encoded by the state of a special-purpose exponential gods. As a result, elementary terms of type r -+ r' enumerate so-called exponential events which are in fact functional dependencies between (sufficient) input events and (necessary) output events. A generalized concrete data structure is a tuple ((C, _<), V, E, ~-) where C is a countable and partially ordered set of cells, V is a countable set of values, E c_ C • V is a set of events. An event (c, v) is often written cv. The enabling relation ~ is between finite sets of events and cells. It induces aprecedence relation on cells: c << c' iff By, v. y U {cv} ~ c'. The enabling relation must be well-founded. The set of events and the enabling relation are upwards-closed with respect to cell ordering. Namely cv E E, c <_ c' ~ c'v C E and y F- c, c _< c' ~ y F- c'. We assume a finite and statically-available set I of indices. A gcds with indices is a gcds and a total function A from the set of cells to the set of indices, called the location function. A cell c is called initial if it is enabled by the empty set of events (written ~- c). A cell c is filled in a set of events y if 3v. cv E y. Write F(y) for the set of cells filled in y. If y k c, then y is an enabling of c. Let E(y) be the set of cells enabled in y, and call A ( y ) - E ( y ) - F ( y ) the set of cells accessible from y. If y ~ c and Vd E F(y), a(d) - A(c), then the enabling y F- c is local, else it is synchronizing. Let M, M', N denote gcds from now on. A state of M is a set of events x c_ EM which is functional and safe, namely where CVl, cv2 E x =~ Vl = v9 and c c F ( x ) =~ By c_ x. y F-~ c. States must be upwards-closed with respect to cell ordering: cv E x, c <_ c' =~ c'v c x. The generalized concrete domain or simply domain associated with M is the poset (7)(M), C_ ) where 7)(M) is the set of states of gcds M. In the mathematical description we sometimes leave A implicit, although it is understood that the concrete location of every event is that of its cell. Because all types-domains are concrete, the usual type constructors product and arrow have
81 concrete realizations. The product construction is straightforward. Given two gcds with indices M1, M2, we define the gcds with indices M1 x M2 by renaming the cells c of M~ into c.i (i = 1, 2) and by taking the union (using the cell renaming this union will be a disjoint union) of cell ordering, values, access relation, function of location of the two gcds with indices. As expected, and easily verified, the domain of M1 x M2 is isomorphic to the cartesian product of the domains of M1 and M2. The definition of a correct exponential is non-trivial and was in fact a central issue in the works of Berry and Curien and later Brookes and Geva. Let two gcdss with indices M, M' be given and let us call them the source structure and the target structure. The exponential M ---+ M' is (C, <, V, E, F) where C = D~in(M) x Ca4,, and 7)~in(M) is the set of finite states of M ordered by inclusion. (z, c') will be abbreviated to zc'. The finite sets of cells are ordered by inclusion and the cells of the target retain their order: < - C x _
(1).
Clearly (l) defines, for given functional state a and argument state y, a unique set of events. Moreover this set of events can be verified to be functional and safe [2]. This is therefore used as the definition of function application in CDS*. Finally, there is a type constructor specific to gcds the graft, which adds great flexibility in practical applications. Its use is illustrated in [3]. Due to lack of space we do not give a concrete syntax for CDS* and we refer to [5]. This syntax allows to define (for the moment only finite) types (gcds with indices) by enumeration of the cells and enablings and by combining types using product, sum or arrow. The values and functions are named states given by enumeration of their events. Terms are function names, application of one term to another, composition of terms, pairs or couples of terms, fix-point, currying or uncurrying of a term (of correct types of course).
3. OPERATIONAL SEMANTICS We now define the formal operational semantics of CDS* which is fully abstract with respect to the denotational semantics outlined above [5]. The operational semantics manipulates memo terms, terms which memorize parts of their evaluation. A memo term is a CDS* program term some of whose syntactic nodes are tagged with tables. Tables, initially empty, store current approximations of auxiliary states. For example, a function-application node is tagged with a table approximating the (state denoted by) the right sub-term i.e. the argument of the application. Let us recursively define the memo terms. States are memo terms. Couples, pairs, currying and uncurrying of memo terms are memo terms. If T and U are two memo terms, x is a state of the same type as U, then [T.U, x] is a memo term, where x represents the current approximation of U and is called a table. I f T is a memo term and x is a state ofthe same type as T's input, then
82 [ f i x ( T ) , x] is a memo term. If T and T' are memo terms and F is a set of pairs of states (x, x') where x (resp. x') has the same type as T's (resp. T"s) input, then [T' o T, F] is a memo term. x is an approximation of the input of T and x' is an approximation of T ( x ) . CDS* expressions are memo terms without tables x and F. We then define an inference system on a set of requests T?c and answers T!v where T is a memo term, c a cell of the same type and v a value. The rules are presented in figure 1. In the rules AP1, AP2, the table x represents the current approximation of the input U. The union II is defined over memo terms whose pure-term parts are equal. Its result IIcUc has the same pure-term part as its arguments Uc and has as approximation-part, the union of approximation parts of Ur and recursively. In the rules for composition o f T ' o T, we have an input state x and a table F of pairs (z, z') where z is a set of input events and z' a set of output events for T.z (i.e. input events for T'). From x and F, two states y, y' are defined as follows, y is the set of input events found in F and relevant w.r.t x (i.e. element of x). y' is the set of output events found in F and relevant w.r.t. x. More precisely: y = U{zl 3(z, z') E F. z c_ x} and y ' = U{z'l 3(z, z') c F. z C_ x}. By construction, for each pair (z, z') of F, z' represents an approximation of the input to T'. Rules AP2, F2, C2 and C3 are the only rules who introduce sub-requests which will be executed in parallel. These sub-requests are determinate by a set A ( x ) , which is a set of cells which are accessible from x (to follow access conditions). Rules CL1 and P1 are for couples and pairing respectively and there exists also symmetrical rules CL2 and P2 (not shown here).
Figure 1. Operational semantics
T?xc' > Tl!v' T?c E> Tl !v (T, U)?c.1 > (T1, U)!v [CL1] (T, U} ?x(c'.l) [> (T1, U)!v' [P1] T?x(x' c") > Tl !V" T?(x, x')c" > T~!v" [CUR] uncurry(T) ? (x, x')c" [> uncurry(T1) !v" [UNC] curry(T)?x(x'c") > curry(T1)!v" T?xd > T~!v' U ' = u~U~ y = u~{r I r e A(z) & U?c > Ur lAP2] [T.U, x]?c' E> [TI.U, x]!v' [AP1] IT.U, x]?c' E> [T.U', x U y}]?c' T?xc > Tl !V [FU [fix(T),x]?c > [fix(T~),x]!v T1-- UClTca, y-U Cl(Yc~U{ c1Vl })[c1 cA(x)&[fix(T),x]?cl[>[fix(T~ ),Yr ]!Vl [fix(T), x]?c [> [fix(T1), y]?c IF2] T'?y'r ~ T~!v" T1 = U~,T~, y~ = U~,{dv'} I d e AC(y') & T?yc't> Tr [c2] [T'oT, F]?xc"[>[T~oT, F]!v ''[C1] [T'oT, F]?xc"[>[T'oT1,FU(y,y'Uy~l)]?xc '' Yl Uc{CV} I c e AC(y) & x?c c>x!v cv 9 up~(~) x - T a ~>/3 /3 > T fTRN] IT' o T, F]txc" > [T' o T, F (2 { (y U Yl, Y') }]?xc" [C3] x?c > x!v [E] a > ,7 -
"
-
4. T O W A R D S AN A B S T R A C T M A C H I N E : R E W R I T I N G S E M A N T I C S Like natural semantics of the usual functional languages, the operational semantics cannot be used in practice. Thus we investigate a new semantics which is to an implementation. The
83 rewriting semantics is defined by the figures 2, 3 and 4 below representing possible transitions from left columns to right columns. The global data structure is a multi-set of tasks together with a function (set of pairs) mapping syntactic nodes of the term being evaluated to tables. In the figures the columns contain only the interesting tasks for the rules and it must be understood that the remaining tasks of the multi-set are left unchanged. There are two main forms of tasks. The first component of tasks is the name of a cell together with the term whose type contains it. The last component of tasks is always a m o d e : either a value computed for the given cell, a failure marker 2_ to indicate that the cell could not be filled, or a request mode. A simple request mode "?" simply means a general request to fill the cell. An indexed request mode ?~ means a request with the added information that the value sought is also that of cell c. As an example of this last kind of mode, see rule CL2 in the first figure. A last kind of mode ?tr indicates that a s l i c e of state ( A ( x ) in the operational semantics) is currently being filled. A task with this mode has a third, middle, component. This middle component stores a set of cells remaining to fill and the current result of the attempt at filling the slice. This result is either failure on all cells • success at least with one cell !. A special kind of task is also used (in figure 4) for filling the so-called F-tables when evaluating a composition term. The following definition is needed. V(x c) - / v iff cv E x '
.
_J_iffcv ~ x
In general, transition rules come in pairs: a call for evaluation and a return with the result. For example CL 1 with CL2. Rules in figure 2 merely follow the syntactic structure of terms. For example rule ST must be read as: if the multi-set of tasks contains the task (c~, ?) then it will be replaced by the task (c, v) if cv e x and by the task (c, 2-) otherwise.
Figure 2. Rules without table ]Tasks ST (cx,?) CL1 (C.I(T1,T2), 7) CL2 (C.I(T1,T2), ?.CT1 ) U (CT1,//) CR1 (C.2(T1,T2), 9.) CR2 (C.2(T1,T2), ?.CT2 ) ~ (CT2, v ) PL1 (c.I
Tasks' (~x,v(~,~)) (C.I(T1,T.2) , 7CT1 ) U (CT1 , ?) (C.I(TI,T2), Y)
(C.2(T1,T2), ?.CT2 ) W (CT2,9) (C.2(T1,T2), v)
(c.I,?~T~) w(cT~,?) (C.I
(X(X'C")cur(T),U) ((x, x')%nc(T),7~(~'c")T)U (~(~'~")T,7) ((x, x')~,c(T),")
Rules in figure 3 implement the definition of concrete function application. The evaluation
84 proceeds in two phases. First AP1, then AP2, then AP3 or AP4. AP2 asks the global system whether the current approximation of U (namely x) is sufficient