PARALLEL
COMPUTING: Software Technology, Algorithms, Architectures and Applications
ADVANCESIN PARALLELCOMPUTING VOLUME 13
Series Editor:
Gerhard R. Joubert ManagingEditor
(Technical University of Clausthal) Aquariuslaan 60 5632 BD Eindhoven,The Nmherlands
2004
ELSEVIER Amsterdam - Boston - Heidelberg - London - New Y o r k - Oxford Paris- San D i e g o - San Francisco- Singapore- S y d n e y - Tokyo
PARALLEL COMPUTING: Software Technology, Algorithms, Architectures and Applications
Edited by
G.R. Joubert
W.E. Nagel
Clausthal
Dresden
Germany
Germany
EJ. Peters
W.V. Walter
Eindhoven
Dresden
The Netherlands
Germany
20O4
ELSEVIER Amsterdam - Boston - Heidelberg - London - New York - Oxford Paris- San D i e g o - San Francisco- S i n g a p o r e - S y d n e y - Tokyo
ELSEVIER B.V. Sara Burgerhartstraat 25 P.O. Box 211, 1000 AE Amsterdam, The Netherlands
ELSEVIER Inc. 525 B Street, Suite 1900 San Diego, CA 92101-4495 USA
ELSEVIER Ltd The Boulevard, Langford Lane Kidlington,Oxford OX5 1GB UK
ELSEVIER Ltd 84 Theobalds Road LondonWC1X 8RR UK
9 2004 Elsevier B.V. All rights reserved. This work is protected under copyright by Elsevier B.V., and the following terms and conditions apply to its use: Photocopying Single photocopies of single chapters may be made for personal use as allowed by national copyright laws. Permission of the Publisher and payment of a fee is required for all other photocopying, including multiple or systematic copying, copying for advertising or promotional purposes, resale, and all forms of document delivery. Special rates are available for educational institutions that wish to make photocopies for non-profit educational classroom use. Permissions may be sought directly from Elsevier's Rights Department in Oxford, UK: phone (+44) 1865 843830, fax (+44) 1865 853333, e-mail:
[email protected]. Requests may also be completed on-line via the Elsevier homepage (http://www.elsevier.com/locate/permissions). In the USA, users may clear permissions and make payments through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; phone: (+1) (978) 7508400, fax: (+1) (978) 7504744, and in the UK through the Copyright Licensing Agency Rapid Clearance Service (CLARCS), 90 Tottenham Court Road, London W1P 0LP, UK; phone: (+44) 20 7631 5555; fax: (+44) 20 7631 5500. Other countries may have a local reprographic rights agency for payments. Derivative Works Tables of contents may be reproduced for internal circulation, but permission of the Publisher is required for external resale or distribution of such material. Permission of the Publisher is required for all other derivative works, including compilations and translations. Electronic Storage or Usage Permission of the Publisher is required to store or use electronically any material contained in this work, including any chapter or part of a chapter. Except as outlined above, no part of this work may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without prior written permission of the Publisher. Address permissions requests to: Elsevier's Rights Department, at the fax and e-mail addresses noted above. Notice No responsibility is assumed by the Publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made.
First edition 2004 Library of Congress Cataloging in Publication Data A catalog record is available from the Library of Congress. British Library Cataloguing in Publication Data A catalogue record is available from the British Library.
ISBN: 0-444-51689-1 O The paper used in this publication meets the requirements of ANSI/NISO Z39.48-1992 (Permanence of Paper). Printed in The Netherlands.
PREFACE Dresden, a city of science and technology, of fine arts and baroque architecture, of education and invention, location of important research institutes and high tech firms in IT and biotechnology, and gateway between Western and Eastern Europe, attracted 175 scientists for the international conference on parallel computing ParCo2003 from 2 to 5 September 2003. It was the tenth in the biannual ParCo series, the longest running European conference series covering all aspects of parallel and high performance computing. ParCo2003 was once again a milestone in gauging the status quo of research and the state of the art in the development and application of parallel and high performance computing techniques, highlighting both current and future trends. The conference was hosted by the Center of High Performance Computing (ZHR) of the Technical University of Dresden. Since its foundation in 1828, the TU Dresden has undergone tremendous transformations from engineering school to technical school of higher education to full university. Today, the TU Dresden offers a broad range of subjects and specialisations in a wide variety of fields to about 33000 students. In the tradition of many inventions in the development of mechanical calculators and early computers, the Center for High Performance Computing was founded in 1997 and has played an important role in the development of modem methods and tools to support high performance computing at the university and beyond. Nowadays, many aspects of parallel computing have become part of mainstream computing. It is now commonplace to buy commodity off-the-shelf computers for home and office use that incorporate parallel techniques such as superscaIarity, hyper threading, VLIW (Very Long Instruction Word) and even cluster technologies that were considered advanced a mere decade ago. Quite apart from the speed with which new parallel technologies find their way into new products, these developments underline the importance of parallel computing research and development for the advancement of computer science and IT in general. In view of the rapid technology transfer taking place, one could be led to conclude that parallel computing research and development has passed its zenith since it has become standard computing practice. ParCo2003 showed that such a conclusion is invalid and that many complex research issues remain to be investigated. Thus it is c l e a r - and this has been the case for a number of years - that future research in parallel computing will have to concentrate increasingly on all aspects of software engineering. In addition the development of new architectures, especially those based on new technologies such as nanotechnologies, biocomputing, improved methods for performance evaluation, advanced algorithms, etc. must continue to receive appropriate attention.
Historical aspects and current trends were highlighted by three invited talks: 9 Friedel HoBfeld (Germany): Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday 9 Manfred Zorn (USA): Computational Challenges in the Genomics Era 9 Charles D. Hansen (USA): High-Performance Visualisation: So Much Data, So Little Time
vi Furthermore, there were more than 80 contributed papers broadly grouped under three main topics: Applications, Algorithms, and Software & Technology. Most of the papers received three and sometimes even four reviews, and we want to thank all members of the Programme Committee for their diligent work. In contrast to previous ParCo conferences, this conference put a strong emphasis on minisymposia, which constituted two entire parallel tracks of the conference. The topics of the seven minisymposia were: Title Organised by Franz-Josef Pfreundt Grid Computing Manfred Zom and Craig Stewart Bioinformatics, Allen Malony Performance Analysis Barbara Chapman OpenMP Tor Sorevik Parallel Applications Anne C. Elster Cluster Computing Alessandro Genco Mobile Agents We wish to thank the organisers of these minisymposia for their tremendous work and support in attracting excellent speakers and in reviewing the papers. The invited speakers, authors of contributed papers and participants in the industrial session as well as in the various minisymposia highlighted many challenging application areas of parallel computing, such as bioinformatics and genomics, visualisation, image and video processing, modelling and simulation, mobile agents and data mining. Due to the complexity of the problems encountered, parallel computing paradigms often provide the only feasible approach. The overall picture conveyed by the conference was thus one of consolidation of parallel computing technologies and transfer of these into off-the-shelf products on the one hand, and of emerging new areas of research and development on the other. The editors are greatly indebted to the members of the International Programme Committee as well as of the Steering, Organising, Finance and Exhibition Committees for the time they spent in making this conference such a successful event. Many thanks are due to the staff of the Center for High Performance Computing for their enthusiastic support. This was a major factor in making this conference a great success. Our special thanks go to Claudia Schmidt (general organisation), Heike Jagode (overall design), Thomas Blfimel (web administration), Stefan Pflfiger (exhibition), and Guido Juckeland (proceedings).
Gerhard R. Joubert Germany
February 2004
Wolfgang E. Nagel Germany
Frans J. Peters Netherlands
Wolfgang V. Walter Germany
vii
SPONSORS AMD GmbH Cray Computer Deutschland GmbH Hewlett-Packard GmbH, Gesch~iftsstelle Berlin IBM Deutschland GmbH, Fachb. Lehre u. Forschung Megware Computer GmbH NEC High Performance Computing Europe GmbH Pallas GmbH Silicon Graphics GmbH SUN Microsystems GmbH
E X H I B I T O R S / PARTICIPANTS IN THE I N D U S T R I A L T R A C K Cray Computer Deutschland GmbH Hewlett-Packard GmbH, Gesch~iftsstelle Berlin IBM Deutschland GmbH, Fachb. Lehre u. Forschung Megware Computer GmbH NEC High Performance Computing Europe GmbH Silicon Graphics GmbH SUN Microsystems GmbH
viii CONFERENCE
COMMITTEE
Gerhard R. Joubert (Conference Chair, Germany/Netherlands) Wolfgang E. Nagel (Germany) Frans J. Peters (Netherlands) Wolfgang V. Walter (Germany)
STEERING COMMITTEE Frans J. Peters (Chair, Netherlands) Friedel Hossfeld (Germany) Paul Messina (USA) Masaaki Shimasaki (Japan) Denis Trystram (France) Marco Vanneschi (Italy)
ORGANISING COMMITTEE Wolfgang V. Walter (Chair, Germany) Thomas Blfimel (Germany) Uwe Fladrich (Germany) Heike Jagode (Germany) Guido Juckeland (Germany) Claudia Schmidt (Germany) Bernd Trenkler (Germany) Andrea Walther (Germany)
EXHIBITION SUB-COMMITTEE Stefan Pflfiger (Chair, Germany) Norbert Attig (Germany) Hubert Busch (Germany) Wolf-Dietrich Harz (Germany) Matthias Mfiller (Germany)
FINANCE COMMITTEE Frans J. Peters (Chair, Netherlands)
ix
PROGRAMME COMMITTEE Wolfgang E. Nagel (Chair, Germany) Nikolaus Adams (Germany) Hamid Arabnia (USA) Norbert Attig (Germany) Eduard Ayguad6 (Spain) Achim Basermann (Germany) Christian Bischof (Germany) Petter E. Bjorstad (Norway) Arndt Bode (Germany) Thomas Brandes (Germany) Mats Brorsson (Sweden) Helmar Burkhart (Switzerland) Barbara Chapman (USA) Michel Cosnard (France) Pasqua D'Ambra (Italy) Luisa D'Amore (Italy) Erik D'Hollander (Belgium) Koen De Bosschere (Belgium) Luiz DeRose (USA) Andreas Deutsch (Germany) Beniamino Di Martino (Italy) Michael Eiermann (Germany) Rtidiger Esser (Germany) Thomas Fahringer (Austria) Afonso Ferreira (France) Salvatore Filippone (Italy) Michael Gerndt (Germany) Lucio Grandinetti (Italy) Andreas Griewank (Germany) John Gurd (UK) Volker Gtilzow (Germany) Bianca Habermann (Germany) Rolf Hempel (Germany) Hans-Christian Hoppe (Germany) Lennart Johnsson (USA) Dieter Kranzlmiiller (Austria) Norbert Kroll (Germany) Herbert Kuchen (Germany) Keqin Li (USA) Thomas Lippert (Germany) Thomas Ludwig (Germany) Allen Malony (USA)
Tomas Margalef (Spain) Djordje Maric (Switzerland) Federico Massaioli (Italy) Arndt Meyer (Germany) Bemd Mohr (Germany) Almerico Murli (Italy) Per Oster (Sweden) Jean-Louis Pazat (France) Franz-Josef Pfreundt (Germany) Wilfried Philips (Belgium) Michael J. Quinn (USA) Roll Rabenseifner (Germany) Thomas Rauber (Germany) Alexander Reinefeld (Germany) Michael Resch (Germany) Richard Reuter (Germany) Jean Roman (France) Mathilde Romberg (Germany) Dirk Roose (Belgium) Hanns Ruder (Germany) Gudula Rtinger (Germany) Tor Sorevik (Norway) Jens Simon (Germany) Horst Simon (Germany) Henk J. Sips (Netherlands) Erich Strohmaier (USA) Vaidy Sunderam (USA) Mateo Valero (Spain) Marco Vanneschi (Italy) Jeffrey Vetter (USA) Heinrich Voss (Germany) Martin Walker (Switzerland) Wolfgang Walter (Germany) Helmut Weberpals (Germany) Roland Wismtiller (Germany) Gabriel Wittum (Germany) Rtidiger Wolff (Germany) Emilio Zapata (Spain) Hans Zima (Austria) Manfred Zorn (USA)
This Page Intentionally Left Blank
xi CONTENTS Preface Sponsors, Exhibitors / Participants in the industrial track Committees
V
vii viii
Invited Papers Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday E Hossfeld So Much Data, So Little Time... C. Hansen, S. Parker, C. Gribble
Software Technology
13
21
On Compiler Support for Mixed Task and Data Parallelism T. Rauber, R. Reilein, G. Riinger
23
Distributed Process Networks - Using Half FIFO Queues in CORBA A. Amar, P. Boulet, J.-L. Dekeyser, E Theeuwen
31
An efficient data race detector backend for DIOTA M. Ronsse, B. Stougie, J. Maebe, E Cornelis, K. De Bosschere
39
Pipelined parallelism for multi-join queries on shared nothing machines M. Bamha, M. Exbrayat
47
Towards the Hierarchical Group Consistency for DSM systems : an efficient way to share data objects L. LefOvre, A. Bonhomme An operational semantics for skeletons M. Aldinucci, M. Danelutto A Programming Model for Tree Structured Parallel and Distributed Algorithms and its Implementation in a Java Environment H. Moritsch
55
63
71
A Rewriting Semantics for an Event-Oriented Functional Parallel Language E Loulergue
79
RMI-like communication for migratable software components in HARNESS M. Migliardi, R. Podesta
87
Semantics of a Functional BSP Language with Imperative Features E Gava, E Loulergue
95
xii The Use of Parallel Genetic Algorithms for Optimization in the Early Design Phases E. Slaby, W. Funk An Integrated Annotation and Compilation Framework for Task and Data Parallel Programming in Java H.J. Sips, K. van Reeuwijk
103
111
On The Use of Java Arrays for Sparse Matrix Computations G. Gundersen, T. Steihaug
119
A Calculus of Functional BSP Programs with Explicit Substitution
127
E Loulergue JToe: a Java API for Object Exchange
135
S. Chaumette, P. Grange, B. MOtrot, P. Vign&as A Modular Debugging Infrastructure for Parallel Programs D. Kranzlmiiller, Ch. Schaubschldger, M. Scarpa, J. Volkert
143
Toward a Distributed Computational Steering Environment based on CORBA O. Coulaud, M. Dussere, A. Esnard
151
Parallel Decimation of 3D Meshes for Efficient Web-Based Isosurface Extraction A. Clematis, D. D'Agostino, M. Mancini, V. Gianuzzi
159
Parallel P r o g r a m m i n g
167
MPI on a Virtual Shared Memory F. Baiardi, D. GuerrL P. Mori, L. RiccL L. Vaglini
169
OpenMP vs. MPI on a Shared Memory Multiprocessor J. Behrens, O. Haan, L. Kornblueh
177
MPI and OpenMP implementations of Branch-and-Bound Skeletons I. Dorta, C. Le6n, C. Rodriguez, A. Rojas
185
Parallel Overlapped Block-Matching Motion Compensation Using MPI and OpenMP E. Pschernig, A. Uhl
193
A comparison of OpenMP and MPI for neural network simulations on a SunFire 6800 A. Strey
201
Comparison of Parallel Implementations of Runge-Kutta Solvers: Message Passing vs. Threads M. Korch, T. Rauber
209
xiii
Scheduling
217
Extending the Divisible Task Model for Workload Balancing in Clusters U. Rerrer, O. Kao, F. Drews
219
The generalized diffusion method for the load balancing problem G. Karagiorgos, N. Missirlis, E Tzaferis
225
Delivering High Performance to Parallel Applications Using Advanced Scheduling N. Drosinos, G. Goumas, M. Athanasaki, N. Koziris
233
Algorithms
241
Multilevel Extended Algorithms in Structural Dynamics on Parallel Computers K. Elssel, H. Voss
243
Parallel Model Reduction of Large-Scale Unstable Systems P. Benner, M. Castillo, E.S. Quintana-Orti, G. Quintana-Orti
251
Parallel Decomposition Approaches for Training Support Vector Machines T. Serafini, G. Zanghirati, L. Zanni
259
Fast parallel solvers for fourth-order boundary value problems M. Jung
267
Parallel Solution of Sparse Eigenproblems by Simultaneous Rayleigh Quotient Optimization with FSAI preconditioning L. Bergamaschi, ,4. Martinez, G. Pini
275
An Accurate and Efficient Selfverifying Solver for Systems with Banded Coefficient Matrix C. H61big, W. Krdmer, T.A. Diverio
283
3D parallel calculations of dendritic growth with the lattice Boltzmann method W. Miller, E Pimentel, I. Rasin, U. Rehse
291
Distributed Negative Cycle Detection Algorithms L. Brim, 1. Cernd, L. Hejtmdnek
297
A Framework for Seamlesly Making Object Oriented Applications Distributed S. Chaumette, P. VignOras
305
Performance Evaluation of Parallel Genetic Algorithms for Optimization Problems of Different Complexity P. K6chel, M. Riedel
313
xiv Extensible and Customizable Just-In-Time Security (JITS) Management of ClientServer Communication in Java S. Chaumette, P. VignOras
Applications & Simulation An Object-Oriented Parallel Multidisciplinary Simulation System U. Tremel, F. Deister, K.A. Sorensen, H. Rieger, N.P. Weatherill
321
329 The SimServer
331
Computer Simulation of Action Potential Propagation on Cardiac Tissues: An Efficient and Scalable Parallel Approach J.M. Alonso, J.M. Ferrero (Jr.), V. Herndndez, G. Molt6, M. Monserrat, J. Saiz
339
M o D y S i m - A parallel dynamical UMTS simulator M.J. Fleuren, H. Stiiben, G.F. Zegwaard
347
apeNEXT: a Multi-TFlops Computer for Elementary Particle Physics F. Bodin, Ph. Boucaud, N. Cabibbo, F. Di Carlo, R. De Pietri, E Di Renzo, H. Kaldass, A. Lonardo, M. Lukyanov, S. de Luca, J. Micheli, V. Morenas, N. Paschedag, O. Pene, D. Pleiter, E Rapuano, L. Sartori, F. Schifano, H. Simma, R. Tripiccione, P. Vicini
355
The Parallel Model System LM-MUSCAT for Chemistry-Transport Simulations: Coupling Scheme, Parallelization and Applications R. Wolke, O. Knoth, O. Hellmuth, W. Schrdder, E. Renner
363
Real-time Visualization of Smoke through Parallelizations T. Vik, A.C. Elster, T. Hallgren
371
Parallel Simulation of Cavitated Flows in High Pressure Systems P.A. Adamidis, F. Wrona, U. Iben, R. Rabenseifner, C.-D. Munz
379
Improvements in black hole detection using parallelism F. Almeida, E. Mediavilla, A. Oscoz, E de Sande
387
High Throughput Computing for Neural Network Simulation J. Culloty, P. Walsh
395
Parallel algorithms and data assimilation for hydraulic models C. Mazauric, V.D. Tran, W. Castaings, D. Froehlich, EX. Le Dimet
403
Multimedia Applications Parallelization of VQ Codebook Generation using Lazy PNN Algorithm A. Wakatani
413 415
XV
A Scalable Parallel Video Server Based on Autonomous Network-attached Storage G. Tan, S. Wu, H. Jin, E Xian
423
Efficient Parallel Search in Video Databases with Dynamic Feature Extraction S. Geisler
431
Architectures
439
Introspection in a Massively Parallel PIM-Based Architecture H.P. Zima Time-Transparent Inter-Processor Connection Reconfiguration in Parallel Systems Based on Multiple Crossbar Switches E. Laskowski, M. Tudruj SIMD design to solve partial differential equations R. W. Schulze
441
449
457
Caches
465
Trade-offs for Skewed-Associative Caches H. Vandierendonck, K. De Bosschere
467
Cache Memory Behavior of Advanced PDE Solvers D. Wallin, H. Johansson, S. Holmgren
475
Performance
483
A Comparative Study of MPI Implementations on a Cluster of SMP Workstations G. Riinger, S. Trautmann
485
MARMOT: An MPI Analysis and Checking Tool B. Krammer, K. Bidmon, M.S. Miiller, M.M. Resch
493
BenchIT- Performance Measurement and Comparison for Scientific Applications G. Juckeland, S. Bdrner, M. Kluge, S. K6lling, WE. Nagel, S. Pfliiger, H. R6ding, S. Seidl, T. William, R. Wloch
501
Performance Issues in the Implementation of the M-VIA Communication Software Ch. Fearing, D. Hickey, P.A. Wilsey, K. Tomko
509
Performance and performance counters on the Itanium 2 study U. Andersson, P. Ekman, P. Oster
A benchmarking case
On the parallel prediction of the RNA secondary structure F. Almeida, R. Andonov, L.M. Moreno, V. Poirriez, M. Pdrez, C. Rodriguez
517
525
xvi Clusters
533
M D I C E - a MATLAB Toolbox for Efficient Cluster Computing R. Pfarrhofer, P Bachhiesl, M. Kelz, H. St6gner, A. Uhl
535
Parallelization of Krylov Subspace Methods in Multiprocessor PC Clusters D. Picinin Jr., A.L. Martinotto, R. V. Dorneles, R.L. Rizzi, C. H61big, T.A. Diverio, P. O.A. Navaux
543
First Impressions of Different Parallel Cluster File Systems T.P Boenisch, P W. Haas, M. Hess, B. Krischok
551
Fast Parallel I/O on ParaStation Clusters N. Eicker, F. Isaila, T. Lippert, T. Moschny, W.F. Tichy
559
PRFX : a runtime library for high performance programming on clusters of SMP nodes B. Cirou, M.C. Counilh, J. Roman Grids
569
577
Experiences about Job Migration on a Dynamic Grid Environment R.S. Montero, E. Huedo, I.M. Llorente
579
Security in a Peer-to-Peer Distributed Virtual Environment J. K6hnlein
587
A Grid Environment for Diesel Engine Chamber Optimization G. Aloisio, E. Blasi, M. Cafaro, I. Epicoco, S. Fiore, S. Mocavero
599
A Broker Architecture for Object-Oriented Master/Slave Computing in a Hierarchical Grid System M. Di Santo, N. Ranaldo, E. Zimeo A framework for experimenting with structured parallel programming environment design M. Aldinucci, S. Campa, P. Ciullo, M. Coppola, M. Danelutto, P. Pesciullesi, R. Ravazzolo, M. Torquati, M. Vanneschi, C. Zoccolo M i n i s y m p o s i u m - Grid C o m p u t i n g
Considerations for Resource Brokerage and Scheduling in Grids R. Yahyapour Job Description Language and User Interface in a Grid context: The EU DataGrid experience G. Avellino, S. Beco, E Pacini, A. Maraschini, A. Terracina
609
617
625
627
635
xvii On Pattern Oriented Software Architecture for the Grid H. Prem, N.R. Srinivasa Raghavan
Minisymposium- Bioinformatics Green Destiny + mpiBLAST = Bioinfomagic W. Feng
643
651 653
Parallel Processing on Large Redundant Biological Data Sets: Protein Structures Classification with CEPAR D. Pekurovsky, I. Shindyalov, P. Bourneb
661
MDGRAPE-3: A Petaflops Special-Purpose Computer System for Molecular Dynamics Simulations M. Taiji, T. Narumi, Y. Ohno, A. Konagaya
669
Structural Protein Interactions: From Months to Minutes P. Dafas, J. Gomoluch, A. Kozlenkov, M. Schroeder
677
Spatially Realistic Computational Physiology: Past, Present and Future J.R. Stiles, W.C. Ford, J.M. Pattillo, T.E. Deerinck, M.H. Ellisman, T.M. Bartol, T.J. Sejnowski
685
Cellular automaton modeling of pattern formation in interacting cell systems A. Deutsch, U. BOrner, M. B~ir
695
Numerical Simulation for eHealth: Grid-enabled Medical Simulation Services S. Benkner, W. Backfrieder, G. Berti, J. Fingberg, G. Kohring, J.G. Schmidt, S.E. Middleton, D. Jones, J. Fenner
705
Parallel computing in biomedical research and the search for peta-scale biomedical applications C.A. Stewart, D. Hart, R. W. Sheppard, H. Li, R. Cruise, V. Moskvin, L. Papiez
Minisymposium- Performance Analysis
719
727
Big Systems and Big Reliability Challenges D. A. Reed, C. Lu, C.L. Mendes
729
Scalable Performance Analysis of Parallel Systems: Concepts and Experiences H. Brunst, W.E. Nagel
737
CrossWalk: A Tool for Performance Profiling Across the User-Kernel Boundary A. V. Mirgorodskiy, B.P. Miller
745
Hardware-Counter Based Automatic Performance Analysis of Parallel Programs F. Wolf, B. Mohr
753
xviii Online Performance Observation of Large-Scale Parallel Applications A.D. Malony, S. Shende, R. Bell
761
Deriving analytical models from a limited number of runs R.M. Badia, G. Rodriguez, J. Labarta
769
Performance Modeling of HPC Applications A. Snavely, X. Gao, C. Lee, L. Carrington, N. Wolter, J. Labarta, J. Gimenez, P. Jones
777
Minisymposium - OpenMP
785
Thread based OpenMP for nested parallelization R. Blikberg, T. Sorevik
787
OpenMP on Distributed Memory via Global Arrays L. Huang, B. Chapman, R.A. Kendall
795
Performance Simulation of a Hybrid OpenMP/MPI Application with HESSE R. Aversa, B. Di Martino, M. Rak, S. Venticinque, U. Villano
803
An environment for OpenMP code parallelization C.S. Ierotheou, 11. Jin, G. Matthews, S.P. Johnson, R. Hood
811
Hindrances in OpenMP programming E Massaioli
819
Wavelet-Based Still Image Coding Standards on SMPs using OpenMP R. Norcen, A. Uhl
827
M i n i s y m p o s i u m - Parallel A p p l i c a t i o n s
835
Parallel Solution of the Bidomain Equations with High Resolutions X. CaL G.T. Lines, A. Tveito
837
Balancing Domain Decomposition Applied to Structural Analysis Problems P. E. Bjorstad, J. Koster
845
Multiperiod Portfolio Management Using Parallel Interior Point Method L. Halada, M. Lucka, I. Melichercik
853
Performance of a parallel split operator method for the time dependent Schr6dinger equation T. Matthey, T. Sorevik
861
xix Minisymposium - Cluster Computing
869
Design and implementation of a 512 CPU cluster for general purpose supercomputing B. Vinter
871
Experiences Parallelizing, Configuring, Monitoring, and Visualizing Applications for Clusters and Multi-Clusters O.J. Anshus, J.M. Bjorndalen, L.A. Bongo
879
Cluster Computing as a Teaching Tool O.J. Anshus, A.C. Elster, B. Vinter Minisymposium- Mobile Agents
887
895
Mobile Agents Principles of Operation A. Genco
897
Mobile Agent Application Fields E Agostaro, A. Genco, S. Sorce
905
Mobile Agents and Grid Computing E Agostaro, A. Chiello, A. Genco, S. Sorce
913
Mobile Agents, Globus and Resource Discovery E Agostaro, A. Genco, S. Sorce
919
A Mobile Agent Tool for Resource Discovery E Agostaro, A. Genco, S. Sorce
927
Mobile Agents and Knowledge Discovery in Ubiquitous Computing A. Genco
935
Author & Subject Index
Author Index Subject Index
943 945 951
This Page Intentionally Left Blank
Invited Papers
This Page Intentionally Left Blank
Parallel Computing:SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
Parallel Machines and the "Digital Brain"- An Intricate Extrapolation on Occasion of JvN's 100-th Birthday F. Hossfeld a aChair for Technical Informatics and Computer Sciences, University of Technology (RWTH) Aachen, Central Institute for Applied Mathematics, Research Centre Juelich, Germany On 28 December 2003, the scientific community will celebrate the 100-th anniversary of John von Neumann's birthday. On this occasion, we are reminded of his achievements as outstanding mathematician and creator of Game Theory, but even more that he laid the very conceptional foundations of the digital computer. His concept had the fortune - contrary to Konrad Zuse's in Germany- that the transistor was invented in 1947, just when JvN wrote his famous reports on the digital computer, at Bell Labs thus giving rise to the extraordinary technological development of microelectronics pushed further by other inventions like photolithography and integrated circuits in the years to come. Since decades, the exponential growth of the power of microchips- every 18 months, the integration density of transistors on the chips is doublingis steering the also exponential growth of computer power. The top computers have surpassed the teraflops level by far today targeting towards 100 teraflops or even petaflops. The computer has become ubiquitous, and protagonists of robotics and artificial intelligence are tempted to attribute to it omnipotent capabilities which will lead to autonomous "humanoids" (Moravec, Kurzweil) on the one hand and threatening horror scenarios on the other (Joy). The predictions of the semiconductor industry tell us that "Moore's Law" describing the exponential evolution in microelectronics might remain valid for other 10 to 15 years. Beyond Moore's Law, quantum effects will definitely end the orderly functioning of"classical" circuits. In 1982, Richard Feynman pointed out that certain quantum mechanical systems could not be simulated efficiently on a "classical" digital computer how powerful it may evolve. This led to speculations that computation in general could be done, in principle and even more efficiently, if a novel computer could make thorough use of quantum effects, thus providing a challenging option for parallel computing by exploiting the exponential speedup through quantum parallelism. Peter Shor's factorization algorithm of 1994 showed that the quantum computer is capable to solve NP-hard problems efficiently. Experimentalists work on different physical concepts to realize quantum computation; for instance, quantum dots, trapped ions, superconducting devices, and NMR technology have been shown to provide the principles and the technology to build quantum computers. 1. LEGACIES In the early 1950-s, John von Neumann once said that three scientific achievements had changed the view of the world in the 20-th century: (1) Einstein's Theory of Relativity, (2)
Heisenberg's Quantum Mechanics, and (3) G6del's Incompleteness Theorem. In the second half of the 20-th century, the history of science and technology added a then unforeseen 4-th one: von Neumann's Digital Computer. John von Neumann was born in December 1903 in Budapest as son of a Hungarian Jewish banker. JvN was a mathematical infant prodigy with kind of photographic memory. Of course, he later was disposed to study mathematics; however, his father seeking advice from the famous von Kfirmfin then at the University of Technology in Aachen, JvN was to study chemical engineering which he humbly d i d - obviously in Budapest, Berlin, and Zurich. He finished these studies with the diploma in 1926. In the same year, however, in parallel he got his PhD in mathematics with a thesis on set theory certainly influenced at his visits to G6ttingen by Hilbert and his program. After lecturing in Berlin and Hamburg, he received an invitation as visiting professor from Princeton University which he accepted in 1930. In 1933, after the foundation of the Institute of Advanced Study in Princeton in 1930, he became together with Alexander and Veblen the first professors of mathematics at the institute which attracted afterwards Einstein, G6del, Church, Dyson, and Oppenheimer and many other famous scientists as well as, in 1937, as a guest scientist also Turing. Hence, JvN was well informed not only about the progress of physics but also on the Hilbert-program deconstructing work of G6del and on Turing's and Church's ideas of computation and computability. The concept of the Turing machine and algorithmics gave definitely major impact on his later plans and projects of designing and building the digital computer. Highly awarded, JvN died of bone cancer on February 8, 1957. Already in 1921, he received the Award of the Best Mathematician in Hungary. In 1927, he published five papers: three on quantum mechanics which had just been created by Heisenberg and Schr6dinger, one on a mathematical problem, and one which is considered as the foundation of game theory. In 1932, he published the important monography - in German - on the "Mathematical Foundations of Quantum Mechanics" where he set this theory on solid mathematical ground especially with his ideas on algebras which anew gain much attention in modem theoretical physics. From 1940 to 1945, he was deeply involved in the US Government's Manhattan Project continuing his involvement as an esteemed advisor until his death in Washington. However, simultaneously, he focussed his mathematical interests on grand challenges arising from the stalemate of the analysis of partial differential equations which again guided him to the definition, design, and construction of the digital computer to overcome the stagnation of the analytic mathematical treatment of complex problems like climate and weather forecast, to indicate only a few out of his broad spectrum of interdisciplinary challenges which led him to establish computer simulation as the third category of scientific methodology in addition to theory and experiment, a discipline which has later been named Computational Science (& Engineering) by the Nobel ~rize Winner Kenneth Wilson (1982). Honoured with the Medal of Freedom, the Albert Einstein Memorative Award and the Enrico Fermi Award and the membership in the American Academy of Sciences, the American Academy of Arts and Science, the American Philosophical Society and many other academies abroad, he dies too early to complete, besides his diverse projects in mathematics and computing, his thorough analysis of the potential of the digital computer and his "organs"- as he called its various components- and the similarities of the "digital brain" with, or rather its fundamental distinction from the human brain as he elaborated in his last publication, the fragmentary analysis of "The Computer and the Brain" which he worked out for the invited Silliman Lectures in 1956 which he never could give at Yale University [1, 2, 3, 4, 5].
2. COMPUTER TECHNOLOGY AND SUPERCOMPUTER ARCHITECTURE
While Konrad Zuse in Germany [6] as the definitely, and at last also in the US accepted, first digital computer pioneer struggled since the 1930s with the limited technical and technological resources available, JvN was lucky enough to design his digital computer concept at a time when the transistor was invented by John Bardeen, Walter Brattain and William Shockley in 1947 at Bell Labs - earning them the Nobel Prize in 1956 - which gave a tremendous push to the development of digital devices encouraging the STRETCH Program at IBM in 1955 in parallel with the development of the ferrite core memory by Jay Forrester and stimulating John Backus to create the first compiler for the high-level language Fortran in 1957 to harness the IBM 704 computer with 32,768 32-bit word ferrite core memory and a magnetic drum as secondary storage. In 1959, Jack Kilby at Texas Instruments developed the first integrated circuit (then on germanium substrate) and Robert Noyce at Fairchild Camera invented photolithography. These giant technological steps created the incredible explosion of microelectronics which resulted in 1970 in the road-paving achievement of the first microprocessor Inte14004. From then on, we can follow an exponential growth of digital computer power in parallel with the exponential growth of the densitiy of transistors on the chips which is acknowledged today as "Moore's Law" allowing for the doubling of microprocessor power and memory size within about 18 months. Personal computers, workstations, servers, and supercomputers follow this "Law" since decades enhanced even more by innovations in computer architecture and operating system concepts [7] leading to vector-processing functions and parallelism in modern supercomputers which nowadays reach far beyond the teraflops performance even with large microprocessor clusters while since early 2002 the Japanese "Earth Simulator"- a powerful combination of vector-processing and parallel structures built on NEC technology- is heading the TOP-500 list with more than 40 teraflops peak and about 35 teraflops sustained Linpack performance [8], thus causing similar shock waves -"Computnik" - within the political and technocratic circles in USA as by the Sputnik launched then by the late Soviet Union. This tremendous explosion of the digital computer is in almost all aspects guiding back to roots of computer architecture and programming as layed by JvN in the 1940-s. Scanning the history since the very birthday of Computational Science and Engineering, which as mentioned may be dated back to 1946 when JvN formulated the strategic program in his famous report on the necessity and future of digital computing together with H. H. Goldstine, at that time complex systems were primarily involved with fluid dynamics. JvN expected that really efficient high-speed digital computers would "break the stalemate created by the failure of the purely analytical approach to nonlinear problems" and suggested fluid dynamics as a source of problems through which a mathematical penetration into the area of nonlinear partial differential equations could be initiated. JvN envisioned computer output as providing scientists with those heuristic hints needed in all parts of mathematics for genuine progress and to break the deadlock- the "present stalemate"- in fluid dynamics by giving clues to decisive mathematical ideas. In a sense, his arguments sound very young and familiar. As far as fluid dynamics is concerned, in his John von Neumann Lecture at the SIAM National Meeting in 1981 yet Garett Birkhoff came to the conclusion on the development of fluid dynamics that it be unlikely that computational fluid dynamics (CFD) would become a truly mathematical science in the near future, although computers might soon rival windtunnels in their capabilities; both, however, would be ever essential for research [9, 10].
The various strategic position papers in the 1980s [11, 12, 13, 14, 15] and the government technology programs in the USA, in Europe, and in Japan in the early 1990s claimed that the timely provision of supercomputers to science and engineering and the ambitious development of innovative supercomputing hardware and software architectures as well as new algorithms and effective programming tools are an urgent research-strategic response to the grand challenges arising from these huge scientific and technological barriers. The solutions of complex problems are critical to the future of science, technology, and society. Supercomputing will be a crucial factor for the industry as well in order to meet the requirements of international economic competition especially in the area of high-tech products. Despite the remarkable investments in research centers and universities in building up supercomputing power and skills and also some sporadic efforts in the industry concerning supercomputing in Europe, it took until the 1990s that the U.S. Government and the European Union as well as several national European governments started non-military strategic support programs [ 16, 17, 18]. Their goals were also to enhance supercomputing by stimulating the technology transfer from universities and research institutions into industry and by increasing the fraction of the technical community which gets the opportunity to develop the skills required to efficiently access the highperformance computing resources. In recent years, computer simulation has reached even the highest political level, since, in 1996, the United Nations voted to adopt the Comprehensive Test-Ban Treaty banning all nuclear testing for military and peaceful purposes. Banning physical nuclear testing created a need for full-physical modeling and high-confidence computer simulation and, hence, unprecedented steps in supercomputer power. DoE's Accelerated Strategic Computing Initiative (ASCI) [ 19, 20] aiming to replace physical nuclear-weapons testing by computer simulation, and NSF's Partnerships for Advanced Computational Infrastructures [21 ] in the US targeting at the advancement of new computing and communication infrastructures for grid computing [22, 23] intended to revolutionize science and engineering- as well as business- is definitely establishing computer simulation as a fundamental methodology in science and engineering. The dedication of the Nobel Prize for Chemistry in 1998 to Computational Chemistry, in addition, confirmed its significance in the scientific community as well as in industry and politics. In the early 1990s, Cray vector-supercomputers with shared memory architecture and proprietary bipolar CPU technology were foreseeable to run into difficulties to provide the giant steps in compute power needed by greedy users, then particularly in physics. Supercomputer architectures with massive parallelism and distributed memory emerged as the future, also more cost-effective, line for the very top end. During these years, nearly thirty companies were offering massively parallel systems and others were planning to enter the market with new products, although many experts predicted that the market would not be able to sustain thus many vendors [24]. In [25], a chronological compilation of high performance computer history illuminates the "Darwinistic" forces affecting the supercomputer evolution lines. It demonstrates that the expected shake-out in the computer industry took place questioning the health and the future potential of this industry in total. Some went out of the parallel computer business for quite different reasons -, others became just mergers. The dramatic survival battle in the supercomputer industry was also giving severe damage to the users and the customers in the supercomputing arena. Quite often the critical situation of parallel computing has rigorously been analyzed with respect to the possible negative impacts on the future perspectives and the progress of this scientific discipline. Already Goldstine said that the history of computing is
littered with "australopithecanes", short computer lines which do not lead anywhere [3]. Following in the wake of DoE's ASCI program, in recent years powerful new supercomputers were brought into the market by the manufacturers participating in the high-performance computing race again. It seemed that only those supercomputer manufacturing companies would have a realistic chance to survive in the years to come who had the potential, capabilities, and favour to get involved in the ASCI or other significant US Government supported programs. However, in early 2002, the Japanese took the lead with the "Earth Simulator" which was argued by the climate problems arising around the world. The parallel architectures became quite successful on the TOP 500 list recently extending the parallel computer concept towards clustered symmetric multiprocessor (SMP) nodes mainly based on commodity components tying together possibly tens of thousands of processing elements [26]. Hence, the Central Institute for Applied Mathematics at the Research Centre Juelich running the National Supercomputer Centre "John von Neumann Institute for Computing" just inaugurated its 8.9 teraflops SMP-based new supercomputer "JUMP", now No. 1 in Europe, with 41 IBM p690 nodes of 32 processors each thus adding up to 1312 processors and surpassing its MPP system CRAY T3E/1200 by more than a factor of 10 in performance [27, 28]. The simulation of extremely complex systems may also determine future large-scale computing by interconnecting supercomputers of diverse architectures as giant supercomputer complexes via the new paradigm of grid computing. These developments will challenge not only system reliability, availability and serviceability to novel levels, but also interactivity of concurrent algorithms and, in particular, adaptivity, accuracy, and stability of parallel numerical methods [29]. In any case, however, a new technology will be necessary to target at and beyond the petaflops barrier in supercomputing, since the further expansion of the floor space of several thousands of square-meters needed by today's SMP-clustered systems with only tenths of teraflops peak performance will be neither reasonable nor technically sensibly feasible. IBM's Blue Gene project may give some guidance and, more aggressively, nanotechnology may provide innovative transistor concepts which will shift the limit of Moore's Law a bit more into the next decades [30]. 3. FROM BITS TO QUBITS
Nevertheless, the end of Moore's Law has been predicted for the time around 2020, at last, since extrapolation leads into nanoscales where the quantum regime will totally govern and, thus, by noise disturb the transistor functions. While the applications of"ubiquitous" computing and its specific requirements may be satisfied by the currently available and further improved technology, high-performance computing targeting towards multi-petaflops performance may severely suffer from the ending exponential growth. Therefore, alternative computing concepts breaking those performance barriers should be intensively explored in order to provide the means to deal with complex problems in future science, industry, and society. In 1982, Richard Feynman explained that the simulation of certain quantum physical systems cannot be simulated by "classical" computers; he, however, pointed out that quantum mechanics could possibly provide the principles to develop a fundamentally new computing paradigm: quantum computing, based on technologically and logically different computer architectures [31]. In 1984 already, David Deutsch designed a quantum computer model [32] which has been elaborated since then theoretically in remarkable depth [33] yielding, as an expansion
of the Church-Turing computational concept to quantum Turing machines, deep complexitytheoretical results and quantum algorithms which meet the expectation of exponential computational speedup due to the inherent exponential superposition of quantum states which, in analogy to classical Boolean bit-based logics, are represented by "qubits", thus exploiting the so-called quantum parallelism. Like in Boolean switching algebra with the NAND and NOR gates, in quantum computing also universal gates (Qubit and Conditional NOT: CNOT) are available besides versatile gates to build up logical circuits, and remarkably- like in classical computation- the Fourier Transform turned out to be a fundamental algorithmic element to exploit the exponential speedup in quantum algorithms. On this basis, Peter Shor succeeded in developing a factorization quantum algorithm [34] which not only provides exponential speedup compared with classical sequential algorithms for this well-known computationally hard problem, but also demonstrated that quantum computing is fundamentally capable to dissolve at least for a significant subset of NP-hard problems the NP - P question by providing efficient quantum algorithms [35]. Although there are diverse physical methods which provide the principal means to establish quantum computer technology and architecture (like NMR technology, quantum dots, trapped ions, superconducting devices and others) which in some cases already have been successfully demonstrated with small numbers of qubits, it will definitely take many years to establish, if at all, a sound quantum computer technology. There are many sources for limited stability and fundamental perturbations of entangled quantum states (decoherence). However, as science history has shown elsewhere, once the idea is born, it will be definitely pursued by scientific curiosity in innovative computational concepts and by the need of greedy applications as well. Thus, quantum computing represents a grand challenge for the years to come; it opens a new space for future intellectual adventures in research and development of physics, mathematics, and computer science. 4. TOWARDS THE "DIGITAL BRAIN"? Along with considerations on the future of computation and the understanding of the human brain, in his popular books Roger Penrose also addressed quantum mechanics and possibly emerging new theories in physics like String Theory and its future extensions as potential scientific vehicles to explain human brain functions and understanding mind and consciousness [36, 37, 38]. As indicated above, JvN discussed in his preliminarily sketched Silliman Lectures, published as "The Computer and the Brain" [5], the similarities and differences of the computer, as he defined it, and the brain, as he understood it at his time. Interestingly enough, in his famous reports defining the digital computer in the 1940-s he talked about the major computer components as "organs" thus showing that he in some sense entertained the long European tradition of ideas to design and construct humanoids which boomed in the 17th and 18-th centuries giving space to many, as seen today, ridiculous appearances of fluteplaying monkeys, chess-playing fakes and other curiosities. However, as a great mathematician he certainly also stood in the tradition of occidental philosophy which for centuries up to today is challenged by the mind-body problem reaching back to the ancient philosophers Thales of Milet ("the four element S " ), Anaxagoras of Clazomenae (introducing "nous", the mind), Leukippos, Demokrit and, last not least, Aristoteles and Platon via the Renaissance with Locke , ,, ), ("Essay concerning understanding"), Leibniz ("Nouveaux essais sur l'entendement h umam de la Mettrie ("L'homme plus que machine"), Descartes ("Discours de la m&hode" and "Med-
itationes de prima philosophia") towards Eccles ("The self and its brain", together with Popper) and the brain researchers of today like Edelman, Chalmers, Singer, Roth and many others [39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51 ] pointing out that the human brain be definitely the most complex system in the - k n o w n - universe. The more one learns about the brain, the greater the distance seems to grow to the digital computer, although this statement does not match with the expectations of Artificial Intelligence (AI). The "strong" Artificial Intelligence still maintains its postulation that the exponentially growing complexity of the digital computer will automatically create mind and (self-)consciousness in the digital machine, and Ray Kurzweil [52] and Hans Moravec [53] draw dramatic pictures of a future of autonomous intelligent robots and humanoids which might take over control already during this century keeping us humans as pets if we behave. While Moravec is focussed mainly on robotics, he predicts that the digital computer has reached or is not far from the capacity and capabilitiy to simulate living beings and, pointing at recent computer chess matches with Kasparov, even compete with human intelligence. Based on simplistic calculations comparing vertebrate retina performance and robot vision requirements, he states that 100 tera-instructions per second of computer power will be sufficient to match human behaviour by simulation. This computer performance indeed will be soon available, if it is not already today. On the other hand, IT experts as Bill Joy, philosophers and social scientists as well as brain researchers consider Kurzweil's predictions as either horror scenarios or fundamental impossibilities [54]. However, as long as we do not know what it is about human intelligence and its essence and the possibly - if not, as a scientist, to say: certainly - materialistic ground of human mind and consciousness, what they are and how they emerge, there will be no final answer to the mind-body problem a n d - vice versa - no understanding of the human brain, and, thus, it is not possible to make true statements about the "intelligent" evolution of the digital computer as well. Penrose discussed- in "Shadows of the Mind" [38] - four viewpoints on the question whether computing equals thinking and thinking equals computation: 9 Position A: All thinking is computation ("computation" in the sense of Church and Turing, i.e. based on algorithms and representable by Turing machines); in particular, feelings of conscious awareness are evoked merely by executing appropriate computations. ("Strong" AI) 9 Position B: Awareness is a feature of the brain's physical action; and whereas any physical action can be simulated computationally, computational simulation cannot by itself evoke awareness. ("Weak" AI) 9 Position C: Appropriate physical action of the brain evokes awareness, but this physical action cannot be even properly simulated computationally. 9 Position D: Awareness cannot be explained by physical, computational, or any other scientific terms. It seems to be quite natural and consequent that Penrose, as a mathematician and scientist rather than a strong-AI promoter or a theologically locked esoteric, discarded Positions A, B, and D as inadequate answers to the question, and in strengthening Position C he referred to the evolution of physical theories extending quantum mechanics which will favour and finally
10 confirm Position C. In this respect, it is interesting that also Nobel Prize winner John Eccles seems to have pursued during his last scientific activities (still vague) ideas of involving quantum mechanical processes in the description of the functions of the human brain concerning the interaction of mind and matter. Looking back to JvN's first monograph defining and clarifying rigorously the mathematical foundations of quantum mechanics and to his last work comparing the computer and the brain, we must say that his early death caused a great loss also for this challenging and futuredetermining field of research. If we listen to the predictions of the Kurzweils and Moravecs, we may be justified to state that JvN as one last genius of the 20-th century would have activated his exceptional intellectual capabilities and sharp theoretical instruments to shape this field in his strong scientific manner as he did in his relatively short life in other areas, rather than by blue speculations. The science community has indeed good reasons to celebrate his 100-th birthday! REFERENCES
[ 1] [2] [3] [4] [5]
[6]
[7] [8] [9] [10] [11] [12]
[13]
[ 14] [15]
von Neumann, J., Collected Works, Vol. I-VI, Pergamon Press, 1961-1963. Aspray, W., John yon Neumann and the Origins of Modern Computing, MIT Press, 1990. Macrae, N., John yon Neumann, Pantheon Books, 1992. yon Neumann, J., Mathematical Foundations of Quantum Mechanics, Princeton University Press, 1955. von Neumann, J., The Computer and the Brain, 2nd Edition (with a Foreword by Paul. M. Churchland and Patricia S. Churchland), Yale University Press, Yale Nota Bene Book, 2000. Ceruzzi, P. E., The Early Computers of Konrad Zuse, 1935 to 1945, Ann. Hist. Comp. Vol. 3 (1981), No. 3,241-262; Ritchie, D., The Computer Pioneers, Chapter 3, Simon & Schuster, 1986. Hwang, K., Advanced Computer Architecture- Parallelism, Scalability, Programmability, McGraw-Hill, 1993. www.TOP500.org/lists/2003/11. Birkhoff, G., Numerical Fluid Dynamics, SIAM Review Vol. 25 (1983), 1-34. Roache, P. J., Fundamentals of Computational Fluid Dynamics, Hermosa Publishers, 1998. Special Double Issue: Grand Challenges to Computational Science, Future Generation Computer Systems 5 (1989), No. 2/3. Committee on Physical, Mathematical, and Engineering Sciences, Federal Coordinating Council for Science, Engineering, and Technology, Grand Challenges 1993: High Performance Computing and Communications, The FY 1993 U.S. Research and Development Program, Office of Science and Technology Policy, Washington, 1992. Board on Mathematical Sciences of the National Research Council (USA), The David II Report: Renewing U.S. Mathematics - A Plan for the 1990s, in: Notices of the American Mathematical Society, May/June 1990, 542-546; September 1990, 813-837; October 1990, 984-1004 Commission of the European Communities, Report of the EEC Working Group on HighPerformance Computing (Chairman: C. Rubbia), February 1991. Trottenberg, U., et al., Situation und Erfordernisse des wissenschaftlichen H6chstleistungsrechnens in Deutschland- Memorandum zur Initiative High Performance Scientific
11
[16] [17]
[ 18] [ 19] [20] [21 ]
[22] [23]
[24]
[25]
[26]
[27] [28] [29]
[30] [31] [32] [33] [34]
Computing (HPSC), Februar 1992; published in: Informatik-Spektrum Band 15 (1992), H. 4, 218. The Congress of the United States, Congressional Budget Office, Promoting HighPerformance Computing and Communications, Washington, June 1993. High-Performance Computing Applications Requirements Group, High-Performance Computing and Networking, Report, European Union, April 1994; - High-Performance Networking Requirements Group, Report, European Union, April 1994. Bundesministerium f'tir Forschung und Technologie, Initiative zur F6rderung des parallelen H6chstleistungsrechnens in Wissenschafl und Wirtschafl, BMFT, Bonn, Juni 1993. www.llnl.gov/asci/. www.llnl.gov/asci-pathforward/. Smarr, L., Toward the 21-st Century, Comm. ACM Vol. 40 (1997), No. 11, 28-32. - Smith, Ph. L., The NSF Partnerships and the Tradition of U.S. Science and Engineering, Comm. ACM Vol. 40 (1997), No. 11, 35-37. Foster, I., and C. Kesselman (eds.), The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, 1999. Atkins, D. E., et al., Revolutionizing Science and Engineering Through Cyberinfrastructure, National Science Foundation Report, Blue Ribbon Advisory Panel on Cyberinfrastructure, USA, January 2003. The Superperformance Computing Service, Palo Alto Management Group, Inc., Massively Parallel Processing: Computing for the 90s, SPCS Report 25, Second Edition, Mountain View, California, June 1994. Strohmaier, E., et al., The marketplace of high-performance computing, Special Anniversary Issue (G. Joubert et al., eds.), Parallel Computing Vol. 25 (1999), No. 13/14, 15171544. Hossfeld, F., et al., Gekoppelte SMP-Systeme im wissenschafllich-technischen Hochleistungsrechnen - Status und Entwicklungsbedarf- (GoSMP), Analyse im Auflrag des BMBF, F6rderkennzeichen 01 IR 903, Dezember 1999. www. fz-j uelich.de/zam. http ://jumpdoc, fz-j uelich.de/. Hossfeld, F., Teraflops Computing: A Challenge to Parallel Numerics?, in: Zinterhof, P., et al. (eds.), Parallel Computation, Proceedings of the 4-th International ACPC Conference 1999, Salzburg, Austria, pp. 1-12; Feeding Greedies on Meager Roadmaps, in: Bubak, M., et al. (eds.), Proceedings of SGI'2000, Krakow, Poland, pp. 11-22. Semiconductor Industry Association (SIA), International Technology Roadmap for Semiconductors, 2003 Edition (ITRS 2003). Feynman, R. P., Simulating physics with computers, Int. J. Theor. Phys. Vol. 21 (1982), 467. Deutsch, D., Quantum Theory, the Church-Tufing Principle and the universal quantum computer, Proc. Roy. Soc. Lond. A Vol. 400 (1985), 97. Nielsen, M. A., and Chuang, I. L., Quantum Computation and Quantum Information, Cambridge University Press, 2000. Shot, P. W., Algorithms for quantum computation: discrete logarithms and factoring, in: Proceedings 35-th Annual Symposium on Foundation of Computer Science, IEEE Press, 1994; Polynomial-time algorithms for prime factorization and discrete logarithms on a
12 quantum computer, SIAM J. Comp. Vol. 26 (1997), No. 5, 1484-1509. [35] Moret, B. M. E., and Shapiro, H. D., Algorithms from P to NP - Volume 1: Design & Efficiency, The Benjamin/Cummings Publishing Company, 1991. [36] Penrose, R., The Emperor's New Mind, Oxford University Press, 1989. [37] Penrose, R., Shadows of the M i n d - A Search for the Missing Science of Consciousness, Oxford University Press, 1994. [38] Penrose, R., The Large, the Small, and the Human Mind, Cambridge University Press, 1997. [39] Locke, J., An Essay Concerning Human Understanding, Great Books in Philosophy, Prometheus Books, 1995. [40] de La Mettrie, J. O., Machine Man (L'homme plus que machine), in: Cambridge Texts in the History of Philosophy (ed. by Ann Thomson),: La Mettrie, Machine Man and Other Writings, Cambridge University Press, 1996. [41 ] Descartes, R., Discours de la m6thode pour bien conduire sa raison, et chercher la verit6 dans les sciences; Meditationes de prima philosophia, in: Philosophische Schriften in einem Band, Felix Meiner Verlag, 1996. [42] Leibniz, G. W., Nouveaux Essais sur L'Entendement Humain, Livre I-IV, Philosophische Schriften Band III, Wissenschaftliche Buchgesellschaft, 1985; Schriften zur Logik und zur philosophischen Grundlegung von Mathematik und Naturwissenschaft (orig. latin), Philosophische Schriften Band IV, Wissenschaftliche Buchgesellschaft, 1992. [43] Popper, K., and Eccles, J., The Self and Its Brain - An Argument for Interactionism, Springer Verlag, 1977. [44] Chalmers, D. J., The Conscious Mind- In Search of a Fundamental Theory, Oxford University Press, 1996. [45] Edelman, G. M., and Tononi, G., A Universe of Consciousness - How Matter Becomes Imagination, Basic Books, 2000. [46] Pinker, St., How the Mind Works, Penguin Books, 1998. [47] Singer, R., Der Beobachter im Gehirn, Essays zur Hirnforschung, Suhrkamp Taschenbuch Wissenschaft Band 1571, Suhrkamp Verlag, 2002. [48] Roth, G., Das Gehirn und seine Wirklichkeit- Kognitive Neurobiologie und ihre philosophischen Konsequenzen, Suhrkamp Taschenbuch Wissenschaft Band 1275, Suhrkamp Verlag, 1997. [49] Pauen, M., und Roth, G. (Hrsg.), Neurowissenschaften und Philosophie - Eine Einfiihrung, Wilhelm Fink Verlag, 2001. [50] Pauen, M., Grundprobleme der Philosophic des Geistes - Eine Einf'fihrung, Fischer Taschenbuch Verlag, 2001. [51 ] Zoglauer, Th., Geist und Gehirn- Das Leib-Seele-Problem in der aktuellen Diskussion, Vandenhoek & Ruprecht, 1998. [52] Kurzweil, R., The Age of Spiritual Machines; in German: Homo s@piens, Kiepenheuer & Witsch, 1999. [53] Moravec, H. P., Mind Children: The Future of Robots and Human Intelligence, Harvard University Press, 1990; Robot: Mere Machine to Transcendent Mind, Oxford University Press, 2000. [54] Schirrmacher, F. (Hrsg.), Die Darwin A G - Wie Nanotechnologie, Biotechnologie und Computer den neuen Menschen tr~iumen, Kiepenheuer & Witsch, 2001.
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004ElsevierB.V. All rights reserved.
13
So M u c h Data, So Little Time... C. Hansen a, S. Parker a, and C. Gribble ~ aScientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT 84112 USA Massively parallel computers have been around for the past decade. With the advent of such powerful resources, scientific computation rapidly expanded the size of computational domains. With the increased amount of data, visualization software strove to keep pace through the implementation of parallel visualization tools and parallel rendering leveraging the computational resources. Tightly coupled ccNUMA parallel processors with attached graphics adapters have shifted the research of visualization to leverage the more unified memory architecture. Our research at the Scientific Computing and Imaging (SCI) Institute at the university of Utah has focused on innovative, scalable techniques for large-scale 3D visualization. Real-time ray-tracing for isosurfacing has proven to be the most interactive method for large scale scientific data. We have also investigated cluster-based volume rendering leveraging multiple nodes of commodity components. 1. INTRODUCTION In recent years, scalable architectures and algorithms have led to unprecedented growth in computational data. The effectiveness of using such advanced hardware that produces large amounts of high-resolution data will hinge upon the ability of human experts to interact with their data and extract useful information. Needless to say, the interactive analysis of such large, and sometimes unwieldy, data sets has become a computation bottleneck in its own right. To address this challenge, the field of visualization and computer graphics has made significant advances in the area of parallel methods for both scientific visualization and image generation. 2. SCALABLE ISOSURFACING Many applications generate scalar fields p(x, y, z) which can be viewed by displaying isosur.faces where p(x, y, z) = Piso" Ideally, the value for Piso is interactively controlled by the user. When the scalar field is stored as a structured set of point samples, the most common technique for generating a given isosurface is to create an explicit polygonal representation for the surface using a technique such as Marching Cubes [ 1, 10]. This surface is subsequently rendered with attached graphics hardware accelerators such as the SGI Infinite Reality. Marching Cubes can generate an extraordinary number of polygons, which take time to construct and to render. For very large (i.e., greater than several million polygons) surfaces the isosurface extraction and rendering times limit the interactivity. In this paper, we generate images of isosurfaces directly
14 with no intermediate surface representation through the use of ray tracing. Ray tracing for isosurfaces has been used in the past (e.g. [9, 12, 21 ]), but we apply it to very large datasets in an interactive setting. It is well understood that ray tracing is accelerated through two main techniques [ 18]: accelerating or eliminating ray/voxel intersection tests and parallelization. Acceleration is usually accomplished by a combination of spatial subdivision and early ray termination [8, 5, 20]. Ray tracing has been used for volume visualization in many contexts (e.g., [8, 19, 23]). Ray tracing for volume visualization naturally lends itself towards parallel implementations [11, 14]. The computation for each pixel is independent of all other pixels, and the data structures used for casting rays are usually read-only. These properties have resulted in many parallel implementations. A variety of techniques have been used to make such systems parallel, and many successful systems have been built. These techniques are surveyed by Whitman [24]. Ray tracing naturally lends itself towards parallel implementations. The computation for each pixel is independent of all other pixels, and the data structures used for casting rays are usually read-only. Simply implementations work, but achieving scalability on large parallel resources requires careful attention to synchronization costs and resource assignment. To reduce synchronization overhead we can assign groups of rays to each processor. The larger these groups are, the less synchronization is required. However, as they become larger, more time is potentially lost due to poor load balancing because all processors must wait for the last job of the frame to finish before starting the next frame. We address this through a load balancing scheme that uses a static set of variable size jobs that are dispatched in a queue where jobs linearly decrease in size. The implementation of the work queue assignment uses the hardware fetch and op counters on the Origin architecture. This allows efficient access to the central work queue resource. This approach to dividing the work between processors seems to scale very well. Rendering a scene with a large memory footprint (rendering of isosurfaces from the visible female dataset [ 17]) uses only 2.1 to 8.4 Mb/s of main memory bandwidth. These statistics were gathered using the SGI perfex utility, benchmarked with 60 processors.
2.1. Rectilinear Isosurfacing Our algorithm has three phases: traversing a ray through cells which do not contain an isosurface, analytically computing the isosurface when intersecting a voxel containing the isosurface, and shading the resulting intersection point [ 16]. This process is repeated for each pixel on the screen. Since each ray is independent, parallelization is straightforward. An additional benefit is that adding incremental features to the rendering has only incremental cost. For example, if one is visualizing multiple isosurfaces with some of them rendered transparently, the correct compositing order is guaranteed since we traverse the volume in a front-to-back order along the rays. Additional shading techniques, such as shadows and specular reflection, can easily be incorporated for enhanced visual cues. Another benefit is the ability to exploit texture maps which are much larger than texture memory (typically 32MB to 256MB). 3. PARALLEL ISOSURFACING RESULTS We applied the ray tracing isosurface extraction to interactively visualize the Visible Woman dataset. The Visible Woman dataset is available through the National Library of Medicine as part of its Visible Human Project [ 15]. We used the computed tomography (CT) data which
15 was acquired in lmm slices with varying in-slice resolution. This data is composed of 1734 slices of 512x512 images at 16 bits. The complete dataset is 910MBytes. Rather than downsample the data with a loss of resolution, we utilize the full resolution data in our experiments. As previously described, our algorithm has three phases: traversing a ray through cells that do not contain an isosurface, analytically computing the isosurface when intersecting a voxel containing the isosurface, and shading the resulting intersection point.
# processors 2 3 4 6 8 12 16 24 32 48 64 96 124
0.427/1.00 0.84/1.97 1.26/2.94 1.67/3.91 2.45/5.73 3.20/7.50 4.81 /11.26 6.38/14.93 9.54/22.33 12.65/29.61 18.85/44.13 24.73/57.90 35.38/82.82 43.06/100.79
Frame rate/Speedup 0.084 /1.00 0.155 /1.00 0.17 /1.99 0.31 /2.00 0.25 /2.95 0.46 /2.96 0.33 /3.96 0.62 /3.97 0.50 /5.97 0.93 /5.96 0.67 /7.94 1.23 /7.93 1.00 /11.89 1.84 /11.88 1.33 /15.84 2.45 /15.80 1.98 /23.54 3.65 /23.49 2.63 /31.38 4.88 /31.47 3.92 /46.72 7.30 /47.02 5.18 /61.78 9.64 /62.14 7.67 /91.38 14.28 /92.02 9.73 /115.88 18.17 /117.08
0.304 /1.00 0.60 /1.98 0.89 /2.93 1.19 /3.92 1.76 /5.77 2.32 /7.61 3.44 /11.30 4.59 /15.08 6.84 /22.48 9.12 /29.96 13.52 /44.39 17.72 /58.19 25.04 /82.23 30.28 /99.45
0.568 /1.00 1.13 /1.98 1.68 /2.96 2.24 /3.94 3.29 /5.80 4.36 /7.67 6.51 /11.47 8.64 /15.21 12.92 /22.76 17.09 /30.10 25.27 /44.50 32.25 /56.80 45.50 /80.14 57.70 /101.63
~.20 o
7
~15
m
o
~10 o
ffl
~
JO ~i~
j',i!
'
'
'
'
'i'
'
I
'
iFrame N~mber~(tihe) I
Figure l. Variation in framerate as the viewpoint and isovalue changes.
~a
I
i
I
i
I
16 The interactivity of our system allows exploration of both the data by interactively changing the isovalue or viewpoint. For example, one could view the entire skeleton and interactively zoom in and modify the isovalue to examine the detail in the toes all at about ten FPS. The variation in framerate is shown in Figure 1. 4. INTERACTIVE VOLUME RENDERING WITH SIMIAN Simian is a scientific visualization tool that utilizes the texture processing capabilities of consumer graphics accelerators to produce direct volume rendered images of scientific datasets. The true power of Simian is its rich user interface. Simian employs direct manipulation widgets for defining multi-dimensional transfer functions; for probing, clipping, and classifying the data; and for shading and coloring the resulting visualization [6]. A complete discussion of direct volume rendering using commodity graphics hardware is given in [2]. For more on using multi-dimensional transfer functions in interactive volume visualization, see [7]. All of the direct manipulation widgets provided by Simian are described thoroughly in [6]. The size of a volumetric dataset that Simian can visualize interactively is largely dependent on the size of the texture memory provided by the local graphics hardware. For typical commodity graphics accelerators, the size of this memory ranges from 32MB to 256MB. However, even small scientific datasets can consume hundreds of megabytes, and these datasets are rapidly growing larger as time progresses. Although Simian provides mechanisms for swapping smaller portions of a large dataset between the available texture memory and the system's main memory (a process that is similar to virtual memory paging), performance drops significantly and interactivity disappears. Moreover, because the size of the texture memory on commodity graphics hardware is not growing as quickly as the size of scientific datasets, using the graphics accelerators of many nodes in a cluster-based system is necessary to interactively visualize large-scale datasets. Naturally, cluster-based visualization introduces many challenges that are of little or no concern when rendering a dataset on a single node, and there are many techniques for dealing with the problems that arise. Our goal was to create an interactive volume rendering tool that provides a full-featured interface for navigating and visualizing large-scale scientific datasets. Using Simian, we examine two approaches to cluster-based interactive volume rendering: (1) a "cluster-aware" version of the application that makes explicit use of remote nodes through a message-passing interface (MPI), and (2) the unmodified application running atop the Chromium clustered rendering framework. 5. CHROMIUM CLUSTERED RENDERING F R A M E W O R K Chromium is a system for manipulating streams of OpenGL graphics commands on commodity-based visualization clusters [3]. For Linux-based systems, Chromium is implemented as a set of shared-object libraries that export a large subset of the OpenGL application programming interface (API). Extensions for parallel synchronization are also included [4]. Chromium's crappfaker libraries operate as the client-side stub in a simple client-server model. The stub intercepts graphics calls made by an OpenGL application, filters them through a userdefined chain of stream processing units (SPUs) that may modify the command stream, and finally redirects the stream to designated rendering servers. The Chromium rendering servers,
17 or crservers, process the OpenGL commands using the locally available graphics hardware, the results of which may be delivered to a tiled display wall, returned to the client for further processing, or composited and displayed using specialized image composition hardware such as Lightning-2 [22]. Depending on the particular system configuration, it is possible to implement a wide variety of parallel rendering systems, including the common sort-first and sort-last architectures [13]. For the details of the Chromium clustered rendering framework, see [3]. In principle, Chromium provides a very simple mechanism for hiding the distributed nature of clustered rendering from OpenGL applications. By simply loading the Chromium libraries rather than the system's native OpenGL implementation, graphics commands can be processed by remote hardware without modifying the calling application. However, for the Simian volume rendering application, Chromium does not currently provide features that sufficiently mask the underlying distributed operation and still enable the application to realize the extended functionality that we seek. OpenGL applications may still require significant modifications to effectively utilize a cluster's resources, even when employing the Chromium framework. It was necessary to implement some OpenGL functionality that was not supported by Chromium. First, we implemented the subset of OpenGL calls related to 3D textures, including glTexImage3D, which is the workhorse of the Simian volume rendering application. In addition, we also added limited support for the NVIDIA GL NV t e x t u r e s h a d e r and G L _ N ' V _ t e x t u r e s h a d e r 2 extensions, implementing only those particular features of the extensions that are explicitly used by Simian. 6. CLUSTER-BASED VOLUME RENDERING RESULTS We experimented with two C-SAFE fire-spread datasets simulating a heptane pool fire, h300_0075 and h300_0130, that were produced by LES codes. Both of the original datasets were 302x302x302 byte volumes storing scalar values as u n s i g n e d c h a r s . Simian is capable of rendering volumes using gradient and Hessian components in addition to scalar values. Therefore, each of the fire-spread datasets was pre-processed to included these additional components and then padded to a power of 2, resulting in two 512x512x512x3 byte volumes that store the value, gradient, and Hessian components ("vgh") as u n s i g n e d c h a r s . 1 We first attempted to visualize both the original and vgh versions of each fire-spread dataset using the stand-alone version of Simian and a fairly powerful workstation. This workstation was equipped with an Intel Xeon 2.66GHz processor, 1GB of physical memory, and an NVIDIA GeForce4 Quadro4 900 XGL graphics accelerator with 128MB of texture memory. On this machine, Simian was able to load and render the original datasets using the swapping mechanisms inherent in OpenGL. As expected, the need to swap severely penalized the application's performance, resulting in rates of 0.55 frames per second. However, the OpenGL drivers could not properly handle the large-scale vgh datasets, crashing the application as result. Having firmly established the need for a cluster-based approach, we tested both the clusteraware version and the Simian/Chromium combination with the vgh datasets using 8, 16, and 32 of the C-SAFE cluster nodes. The results are summarized in Table 1. Note that, with 8 and 16 rendering nodes, the vgh subvolumes distributed by the clusteraware version are 256x256x256 bytes and 256x256x128 bytes, respectively. These subvol1Forthe remainder of this discussion, it is assumedthat the numberof bytes consumedby a raw volumetric dataset is the given size multiplied by three, accounting for each of the three, 1-bytecomponentsin a vgh dataset.
18 Rendering Approach
Number of Nodes
Cluster-aware Simian
8 16
32 Simian/Chromium combination
8 16
h300_0075 h300_0130
vgh dataset
vgh dataset
0.52
0.58 2.15 3.44 0.05 0.04 0.02
1.47 2.87 0.07 0.05 0.03
32 Table 1 Average frame rates (in frames per second) using various cluster configurations
umes exceed the maximum "no-swap" volume size permitted by the cluster's graphics hardware (128x128x256 bytes), so in either of these configurations, even the cluster-aware Simian must invoke its texture swapping facilities. However, with 16 rendering nodes, the swapping is much less frequent, resulting in reasonably interactive frame rates. With 32 rendering nodes, the subvolume size is reduced to 128x128x256 bytes, so no texture swapping occurs and interactive frame rates are restored. Chromium is not able to distribute subvolumes among the rendering nodes. As a result, Simian must rely on its texture swapping mechanism. When the application calls for a new block to be swapped into texture memory, Chromium must transmit the block to the appropriate rendering nodes. The resulting delays impose severe performance penalties that grow with the number of rendering nodes. This behavior is reflected in the low frame rates given in Table 1. 7. CONCLUSIONS The architecture of the parallel machine plays an important role in the success of visualization techniques. In parallel isosurfacing since any processor can randomly access the entire dataset, the dataset must be available to each processor. Nonetheless, there is fairly high locality in the dataset for any particular processor. As a result, a shared memory or distributed shared memory machine, such as the SGI Origin, is ideally suited for this application. The load balancing mechanism also requires a fine-grained low-latency communication mechanism for synchronizing work assignments and returning completed image tiles. With an attached graphics engine, we can display images at high frame rates without network bottlenecks. We have implemented a similar technique on a distributed memory machine that proved challenging. Frame-rates were, of course, lower and scalability was reduced. We have shown that ray tracing can be a practical alternative to explicit isosurface extraction for very large datasets. As data sets get larger, and as general purpose processing hardware becomes more powerful, we expect this to become a very attractive method for visualizing large scale scalar data both in terms of speed and rendering accuracy. For cluster-based volume rendering, we have demonstrated that a cluster-aware MPI version performs far better than simply replacing the underlying shared graphics library with a parallelized library. There are several advantages of developing cluster-aware visualization tools. First, these tools can exploit application-specific knowledge to reduce overhead and enhance performance and efficiency when running in a clustered environment. Second, cluster-aware applications that are built upon open standards such as MPI are readily ported to a wide variety
19 of hardware and software platforms. While frameworks that mask the clustered environment may provide certain advantages, using standard interfaces allows an application's components to be reused or combined in new, more flexible ways. Third, cluster-aware applications are not dependent upon the functionality provided by a clustered rendering package. Finally, because they do not access remote resources via a lower-level framework, cluster-aware applications can exploit the capabilities of the underlying system directly.
REFERENCES
[1]
B. Wyvill G. Wyvill, C. McPheeters. Data structures for soft objects. The Visual Com-
puter, 2:227-234, 1986. [2] [3]
[4] [5] [6]
[7]
[8] [9] [ 10]
[11 ] [ 12] [13] [14]
[15]
[16]
M. Hadwiger et al. High-quality volume graphics on consumer pc hardware. IEEE Visualization 2002 Course Notes. G. Humphreys, Mike Houston, Yi-Ren Ng, Randall Frank, Sean Ahem Peter Kirchner, and James T. Klosowski. Chromium: A stream-processing framework for interactive graphics on clusters. In Proceedings of SIGGRAPH, 2002. H. Igehy, G. Stoll, and P. Hanrahan. The design of a parallel graphics interface. In SIGGRAPH Computer Graphics, 1998. Arie Kaufman. Volume Visualization. IEEE CS Press, 1991. J. Kniss, G. Kindlmann, and C. Hansen. Interactive volume rendering using multidimensional transfer functions and direct manipulation widgets. In Proceedings oflEEE Visualization, 2001. J. Kniss, G. Kindlmann, and C. Hansen. Multi-dimensional transfer functions for interactive volume rendering. IEEE Transactions on Visualization and Computer Graphics, pages 270-285, July 2002. Mark Levoy. Display of surfaces from volume data. IEEE Computer Graphics & Applications, 8(3):29-37, 1988. Chyi-Cheng Lin and Yu-Tai Ching. An efficient volume-rendering algorithm with an analytic approach. The Visual Computer, 12(10):515-526, 1996. William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. Computer Graphics, 21 (4): 163-169, July 1987. ACM Siggraph '87 Conference Proceedings. K.L. Ma, J.S. Painter, C.D. Hansen, and M.F. Krogh. Parallel Volume Rendering using Binary-Swap Compositing. IEEE Comput. Graphics and Appl., 14(4):59-68, July 1993. Stephen Marschner and Richard Lobb. An evaluation of reconstruction filters for volume rendering. In Proceedings of Visualization '94, pages 100-107, October 1994. S. Molnar, M. Cox, D. Ellsworth, and H. Fuchs. A sorting classification of parallel rendering. 1EEE Computer Graphics and Applications, July 1994. Michael J. Muuss. Rt and remrt - shared memory parllel and network distributed raytracing programs. In USENIX: Proceedings of the Fourth Computer Graphics Workshop, October 1987. National Library of Medicine (U.S.) Board of Regents. Electronic imaging: Report of the board of regents, u.s. department of health and human services, public health service, national institutes of health. NIH Publication 90-2197, 1990. Steven Parker, Michael Parker, Yarden Livnat, Peter-Pike Sloan, Charles Hansen, and
20
[ 17]
[ 18] [ 19] [20] [21 ] [22]
[23] [24]
Peter Shirley. Interactive Ray Tracing for Volume Visualization. IEEE Transactions on Visualization and Computer Graphics, 5(3):238-250, July 1999. Steven Parker, Peter Shirley, Yarden Livnat, Charles Hansen, and Peter-Pike Sloan. Interactive ray tracing for isosurface rendering. In Proceedings of Visualization '98, October 1998. E. Reinhard, A.G. Chalmers, and F.W. Jansen. Overview of parallel photo-realistic graphics. In Eurographics "98, 1998. Paolo Sabella. A rendering algorithm for visualizing 3d scalar fields. Computer Graphics, 22(4):51-58, July 1988. ACM Siggraph '88 Conference Proceedings. Lisa Sobierajski and Arie Kaufman. Volumetric Ray Tracing. 1994 Workshop on Volume Visualization, pages 11-18, October 1994. Milos Sramek. Fast surface rendering from raster data by voxel traversal using chessboard distance. In Proceedings of Visualization '94, pages 188-195, October 1994. G. Stoll, M. Eldridge, D. Patterson, A. Webb, S. Berman, R. Levy, C. Caywood, M. Taveira, S. Hunt, and P. Hanrahan. Lighming-2: A high-performance display subsystem for PC clusters. In SIGGRAPH Computer Graphics, 2001. Craig Upson and Micheal Keeler. V-buffer: Visible volume rendering. Computer Graphics, 22(4):59-64, July 1988. ACM Siggraph '88 Conference Proceedings. Scott Whitman. A Survey of Parallel Algorithms for Graphics and Visualization. In High Performance Computing for Computer Graphics and Visualization, pages 3-22, 1995. Swansea, July 3-4.
Software Technology
This Page Intentionally Left Blank
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
23
On Compiler Support for Mixed Task and Data Parallelism T. Rauber ~, R. Reilein b, and G. Rfingerb a Department of Mathematics, Physics, and Computer Science University of Bayreuth E-mail: rauber@uni-bayreuth, de b Department of Computer Science Chemnitz University of Technology E-mail: {reilein, ruenger}@cs, tu-chemnitz, de The combination of task and data parallelism can lead to an improvement of speedup and scalability for parallel applications on distributed memory machines. To support a systematic design of mixed task and data parallel programs the TwoL model has been introduced. A key feature of this model is the development support for applications using multiprocessor tasks on top of data parallel modules. In this paper we discuss implementation issues of the TwoL model as an open framework. We focus on the design of the framework and its internal algorithms and data structures. As examples fast parallel matrix multiplication algorithms are presented to illustrate the applicability of our approach. 1. INTRODUCTION Parallel applications in the area of scientific computing are often designed in a data parallel SPMD (Single Program Multiple Data) style based on the MPI standard. The advantage of this method is a clear programming model but on large parallel platforms or cluster systems the speedup and scalability can be limited especially when collective communication operations are used frequently. The combination of task and data parallelism can improve the scalability of many applications but requires a more intricate program development. The adaption of complex program code to the characteristics of a specific parallel machine may be quite time consuming and often results in a code structure which causes a high reprogramming effort when porting the software to another parallel system. To support the systematic development of mixed task and data parallel programs the TwoL model has been introduced. The model provides a stepwise development process which is subdivided into several phases. Applications are hierarchically composed of predefined or user-supplied basic data parallel modules. The development starts with a specification of the parallelism inherent in the algorithm to be implemented. The specification is transformed stepwise into a coordination program by applying scheduling, static load balancing, and data distribution algorithms. During this transformation process the code is adapted to the characteristics of a specific parallel system. In this paper we present an implementation of the TwoL model as an open compiler framework (TwoL-OF). The open framework implements the core concepts of the TwoL model and additionally provides several intermediate program representations resulting within the trans-
24 formation process. The intermediate representations are produced in specific TwoL-OF formats to provide access interfaces for compiler users. The advantage is that the framework can be used by both, application programmers and algorithm developers. Application programmers can use the framework as transformation tool by providing a specification of the problem which is transformed into an efficient parallel program. Developers can test new scheduling and load balancing techniques by exploiting the access interfaces of the intermediate formats. The remaining paper is structured as follows. Section 2 gives a short overview of the TwoL model. In Section 3 the TwoL-OF compiler is introduced and the compilation process is illustrated. Section 4 presents first results with fast matrix multiplication algorithms and Section 5 concludes. 2. THE TwoL MODEL The TwoL (Two Level) model has been developed to exploit the combination of task and data parallelism [6, 7]. It defines a transformation process from a specification of the parallelism inherent in an algorithm into a coordination program for a specific parallel system. The parallelism is exploited on two levels, an upper task parallel level consisting of hierarchically structured multiprocessor tasks and a lower data parallel level. On the lower level the user provides module specifications and function implementations which declare and realize multiprocessor tasks. On the upper level consecutive transformation steps generate a coordination program from the specification of potential task parallelism. Figure 1 illustrates the derivation of the coordination program. non-executable specification coordination program ~...@.,,,.,~ stepwise derivation --~ double *a,*b; ~
@/Q~@
@ ; ~ ~
.Transformation
9Annotation
9 Selection
Level ~'~ ~
MPI Comm comml,comm2; MPI--Comm_split(&comml,color pid,&comm2) ; if (gpid==0) { ...
~ typea distrib b var x:a ,..
Declaration of types,
m o d u l e s and variables,
Parallel library and/~
user-defined functions.
I dmm dmm-fox std [
II
dmm can
Figure 1. Derivation of the parallel coordination program. The specification program declares multiprocessor tasks as modules and defines their data and control dependencies. It uses a special type concept to declare data types and data distribution types. The transformation applies scheduling, load balancing, and data distribution techniques and the decisions made are included as annotations. The final coordination program contains all information about execution order, processor group sizes, and data distributions. The entire transformation process is accompanied by a parallel cost model to guide the transformations and to obtain an efficient parallel program for a specific parallel machine. 3. TwoL-OF C O M P I L E R STRUCTURE AND IMPLEMENTATION The TwoL open framework compiler implements the core concepts of the TwoL model, supports the guided development of mixed task and data parallel programs, and the porting to
25 different parallel machines. The emphasis lays on fast and easy coding support utilizing the activation of basic modules written in different imperative languages especially C or Fortran. It offers specific interfaces to control the transformation process and to manually revise decisions made by the transformation algorithms. Therefore the intermediate code representation is explicitly stored and open for transformations and annotations.
3.1. Short overview The compilation process starts with a specification language file describing the inherent parallelism of an algorithm. The basic components of a specification are modules which can either be declared (basic modules) or defined (coordination modules). For basic modules a module name and three different types for each parameter to be passed have to be declared. Each module parameter has a data type which corresponds to a data type in the high level target language and a data distribution type which declares how the data are distributed over a processor group and which is used by redistribution functions to establish the data distribution required before starting the execution of a basic module. The third type is the I/O access mode which defines the kind of access to the parameters. Basic modules can be implemented in C or Fortran and are linked to the generated program code in the final compilation step. Coordination modules have to be declared before they can be defined by a coordinating expression which consists of module calls, data dependence operators, and control statements. These expressions are translated into syntax trees which are extended and transformed in the following compilation phases. Based on the coordinating expressions the coordination program is generated by the compiler framework as C code augmented with MPI functions and calls to a data redistribution library interface. The resulting code can be compiled and linked with the basic module implementations using a standard C compiler. The TwoL-OF compiler consists of two consecutive compiler stages. The first stage reads a specification program, generates the intermediate code representation, and outputs two control files for the next stage, one for type definitions and a second file to control tree transformations and annotations. The intermediate code representation comprises different kinds of symbol tables, specification syntax trees for the coordinating expressions, and parallel data dependence graphs to attach data flow information to the syntax trees. Between both stages the control files are augmented with additional specifications supplied by automatic tools or by the user. In the second stage after the intermediate code is read, the type definition and transformation control files are processed. The contents are used to amend symbol table entries and to transform the specification syntax trees into coordination syntax trees. C code with calls to MPI and a data redistribution library is generated after compiler passes to update the data flow information and to introduce variable copying and data redistribution. The compilation process is accompanied by the output of internal data structures as program files which can be postprocessed using the graphviz package by AT&T to generate a graphical visualization. In the next subsections we introduce the specification language and give an example for a parallel specification to illustrate the compilation process.
3.2. Specification language for multiproeessor task programs The specification language allows the expression of data dependencies between module calls by two operators, the ~--operator for data dependence and the II-operator for data independence. Both operators can be used in infix form or as k-ary operator. Furthermore there are several statements to control the program flow especially loops and conditions. The following grammar
26 defines the essential parts for expressing data dependencies and control flow. cexpr stat cexpr_list
--+ I
cexpr >- cexpr I cexpr II cexpr ] II (cexpr_list) I >- (cexpr_list) stat ( cond ) { cexpr } I call loop ] while I if cexpr_list, cexpr ] cexpr
A coordination module defines one coordinating expression (cexpr) which is recursively built up of sub-expressions. To illustrate the compilation process the standard four block matrix multiplication is used here as an example. The algorithm is defined by the following equation:
( all a12 ) ( b l l x hi2 ) _ _ a21 a22 b21 b22
( a 1 1 • b11nt-q12 x b21 all >
To specify the inherent parallelism a specification program containing the following definition of a coordination module (tram) has to be supplied by the user. The declarations of the parameter types for the basic modules used are omitted here. c o m p m o d tmm (a, b, c, n) { m p a r t (a, all, a12, a21, a22, n) >- m p a r t ( b , b l l , b 1 2 , b 2 1 , b 2 2 , n ) ]l (dram(all, bll, cll, n/2) , dram (a12, b21, dll, n/2) , dram (all, b12, c12, n/2) , dmm (a12, b22, d12, n/2) , dmm (a21, bll, c21, n/2) , dmm (a22, b21, d21, n/2) , dmm(a21,b12,c22,n/2) , dmm(a22,b22,d22,n/2)) ~- II ( m m a ( c l l , d l l , c l l , n / 2 ) , m m a ( c l 2 , d l 2 , c l 2 , n / 2 ) , mma(c21,d21,c21,n/2) , mma(c22,d22,c22,n/2)) >- rejoin (cll, c12, c21, c22, c, n)
The module m p a r t subdivides a matrix into four quadratic submatrices and mj o in reverses this process. Addition and multiplication of submatrices are performed by mma and dram, respectively. This specification is processed by the first compiler stage and the coordinating expression is stored as a specification syntax tree with a linked parallel data dependence graph for expressing data flow. Figure 2 illustrates the parallel data dependence graph for the specification example as output from the framework. Together with the intermediate representation which is stored as binary file by the first stage two text files for type definitions and to control transformations are generated. The type definition file is used to specify a C or Fortran data type for the abstract types used in the specification program. If memory allocation is required for a specific data type the corresponding statements can be supplied. Within the second file the user can control transformations and annotations of the specification syntax trees. In its unmodified form the transformation control file expresses the maximum degree of parallelism specified. The following code fragment shows a part of the transformation control file for the block matrix multiplication example: par[ID34] [4] { [0] :ID30 # m m a ( c l l , d l l , c l l , n / 2 ) [i] :ID31 # m m a ( c l 2 , d l 2 , c l 2 , n / 2 ) [2] :ID32 # m m a ( c 2 1 , d 2 1 , c 2 1 , n / 2 ) [3] :ID33 # m m a ( c 2 2 , d 2 2 , c 2 2 , n / 2 )
}
27
m
\
\
I
\
/
Figure 2. Part of the Parallel data dependence graph for block matrix multiplication. The labels enclosed in brackets and prefixed with 'ID' are unique identifiers for the nodes of the corresponding specification syntax tree. They are also displayed in the visualization of the trees and data dependence graphs. The second parameter of the p a r statement specifies the number of processor groups or multiprocessor tasks, respectively. The t a s k statement groups tasks together to be executed consecutively on a specific processor group given as first parameter. Tasks would be separated by commas in that case. The number sign (#) marks the start of a comment which are generated here for the sake of clarity. The following subsection presents the internal structure of the compiler framework in more detail. 3.3. Algorithms and data structures The first compiler stage of the TwoL open framework comprises a syntax-oriented translation scheme to built up specification syntax trees, the construction of symbol tables for data types, data distribution types, variables, and modules, and the creation of parallel data dependence graphs as representations of the data flow. The second stage updates the symbol tables and transforms the syntax trees using information in specific control files provided by automatic tools or directly by the user. After the insertion of the data redistribution operations required the final coordination code is generated. A specification program which contains definitions of coordination modules by coordinating expressions is translated into syntax trees which are realized as C data structures. A tree node comprises a type, a unique identifier, pointers to child nodes, and a union to maintain type specific data. Leaf nodes, which represent module calls, have an additional pointer to a data structure for the parallel data dependence graph. The nodes of this graph represent parameters of module calls and a vertex of the graph defines a true data dependence between two accesses to a specific variable. A parallel data dependence graph is constructed by a depth first sweep over the corresponding specification syntax tree. If the algorithm detects an input parameter reading a particular variable it creates a new node for the parallel data dependence graph and traverses backwards a list of previously visited nodes to find an appropriate output for that variable. The search procedure takes the data dependence information stored in the specification syntax tree into account. When an output is found which does not already have a graph node a new one is constructed and linked with the current one by pointers. These links define a true data dependency between two accesses to a variable. Specification syntax trees are also used to generate a text representation of transformation
28 control structures that is output to the transformation control file. For each ll-operator node the file contains a specific program structure, see for example the control file in Section 3.2. According to these structures the following transformation of the syntax trees is applied by the translation scheme of the second compiler stage. Each II-operator node is replaced by a fork node that is used to insert activations of multiprocessor tasks. The data independence expressed by the tl-operator is then replaced by a fork into multiprocessor tasks. A fork node stores the number of processor groups and has new task nodes as children to define multiprocessor tasks. The former children of the II-operator node have been linked to the particular task nodes representing the processor group which will execute them in the final code. After this transformation the syntax tree defines the parallel execution order of a coordination function. The second compiler stage maintains variables and introduces copies if a variable is accessed in parallel. Based on the information contained in the parallel data dependence graphs data redistribution operations are inserted in the trees as additional nodes. The resulting trees comprise all information to generate the final coordination code. Figure 3 illustrates the internal structure of the TwoL-OF compiler and the main compilation steps. l a Parsing the specification program to built up specification syntax trees and symbol tables.
Specification language program
ISymbol tables Specification syntax tree modules ~,~ data t y p e s Specification syntax tree with linked distribution types paralleldata dependence graph I* variables Type definition Intermediatecode Transformationcontrol program representation program
I
.,.
Updated symbol tables
.....
|
......
Coordination syntax tree with linked paralleldata dependence graph ( ~ C code coordination function
l b Construction of the parallel data dependence graphs. 1c Generation of control files and write out of intermediate code. 2a Processing of intermediate code and parsing of transformation control and type definition files to transform syntax trees and to update symbol tables entries. 2b Maintenance of variable copies and data exchange operations. 2c Generation of coordination program as C code.
Figure 3. TwoL-OF structure (left) and main compilation steps (fight).
4. EXPERIMENTS
To evaluate the TwoL-OF compiler two versions of four block matrix multiplication C = A x B have been implemented, the standard algorithm with 8 matrix multiplies and 4 additions and the Strassen scheme with 7 multiplies and 18 additions [8]. The inner multiplications of two block matrices are done using the Fox algorithm. To determine the overhead both schemes have been implemented by hand and as specification program supplemented with appropriate basic modules. All four versions have been measured on a Linux cluster (CLiC) built of 800 MHz
29 Pentium III PCs interconnected with switched Fast Ethernet. Figure 4 shows the overhead of the compiler-generated version which is given as the ratio between the runtime and the runtime of the hand-tuned version.
Overhead factor for block matrix multiplication on CLiC Matrix size 2400 , 3600 ........... 4800 ............
1.15 1.1 1.05
Overhead factor for Strassen matrix multiplication on CLiC Matrix size 2400 , 3600 ........... 4800 ............
1.15 1.1
7_. . . . . . . . . . . . . . . . . . . . . . . . . . . .
16 36
64
100
5-~_-~..........................
144
196
256
processor number
"-,..
1.05
16 36
64
100
144
196
256
processor number
Figure 4. Overhead factors for the standard scheme (left) and the Strassen scheme (fight) on CLiC. The results show a maximum overhead factor of 1.12 for this examples. This corresponds to first experiences made with other codes and applications. It has been observed that handtuned programs usually require less variable copies and can organize data redistribution more compact. But on the other hand they require a higher development effort and cannot benefit from the automatic detection and insertion of data redistribution operations.
5. RELATED W O R K AND CONCLUSION Many research groups have developed compiler tools and parallel environments which can integrate task and data parallelism. Most of them extent Fortran or High Performance Fortran (HPF) with task parallel constructs [3, 9, 2], see [1] for a good overview. Newer approaches include the coordination of data parallel programs written in skeleton-like or data parallel languages [5], parallel Haskell programming [10], or task and data parallel programming in Java
[4].
In this paper the TwoL open framework compiler as an implementation of the core concepts of the TwoL model was introduced as an open platform for algorithm and application developers. We gave a closer look on the structure of the framework and presented early compilation results. The average overhead factor of compiler generated programs is reasonably small and caused by a higher amount of variable copies required in combination with a slight communication overhead. This factor is expected to be reduced by the development of optimization algorithms for message-packing and for an efficient placement of data redistribution operations. Future work will include the definition of an interface to the binary intermediate representation and support for runtime prediction.
30 REFERENCES
[1] [2]
[3] [4] [5] [6]
[7] [8] [9] [ 10]
H. Bal and M. Haines. Approaches for Integrating Task and Data Parallelism. IEEE Concurrency, 6(3): 74-84, 1998. P. Banerjee, J. Chandy, M. Gupta, E. Hodge, J. Holm, A. Lain, D. Palermo, S. Ramaswamy, and E. Su. The Paradigm Compiler for Distributed-Memory Multicomputers. IEEE Computers, 28(10):37-47, 1995. I. Foster and K.M. Chandy. Fortran M: A Language for Modular Parallel Programming. Journal of Parallel and Distributed Computing, 25(1):24-35, 1995. F. Kuijlman, H.J. Sips, C. van Reeuwijk, and W.J.A. Denissen. A Unified Compiler Framework for Work and Data Placement. In Proc. ofASC12002, pages 109-115, 2002. S. Pelagatti and D.S. Skillicorn. Coordinating Programs in the Network of Tasks Model. Journal of Systems Integration, 10(2): 107-126, 2001. T. Rauber and G. R/inger. The Compiler TwoL for the Design of Parallel Implementations. In In Proc. of 4th International Conference on Parallel Architecture and Compilations Techniques (PACT'96), pages 292-301. IEEE Computer Society Press, 1996. T. Rauber and G. Riinger. A Transformation Approach to Derive Efficient Parallel Implementations. IEEE Transactions on Software Engineering, 26(4):315-339, 2000. V. Strassen. Gaussian Optimization is not Optimal. Numerische Mathematik, (13):354356, 1969. J. Subhlok and B. Yang. A New Model for Integrated Nested Task and Data Parallel Programming. In Proc. ofACM SIGPLANPPoPP 97, pages 1-12, 1997. P.W. Trinder, H-W. Loidl, and R.F. Pointon. Parallel and Distributed Haskells. Journal of Functional Programming, 2(4/5):469-510, 2002.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
31
Distributed Process Networks - Using Half FIFO Queues in CORBA A. Amar a*, R Boulet a, J.-L. Dekeyser a, and F. Theeuwen b aLaboratoire d'Informatique Fondamentale de Lille, Lille 1 Cit6 Scientifique, Bat. M3, 59655 Villeneuve d'Ascq cedex, France b Philips ED & T / Synthesis WAY 3.13, Prof. Holstlaan 4 5656 AA Eindhoven, The Netherlands Process networks are networks of sequential processes connected by channels behaving like FIFO queues. These are used in signal and image processing applications that need to run in bounded memory for infinitely long periods of time dealing with possibly infinite streams of data. This paper is about a distributed implementation of this computation model. We present the implementation of a distributed process network by using distributed FIFOs to build the distributed application. The platform used to support this is the CORBA middleware. 1. INTRODUCTION Kahn Process Networks [6, 7] are well adapted to model many parallel applications, specially dataflow applications (signal processing, image processing). In this model, processes communicate only via unbounded first-in first-out (FIFO) queues. This model has a dataflow flavor and can express a high degree of concurrency which makes it particularly well suited to model intensive signal processing applications or complex scientific applications. This model makes no assumption on the computation load of the different processes and thus is heterogeneous by nature. Distributed architectures provide an attractive alternative to supercomputers in terms of computation power and cost to execute such complex and computation intensive applications. The two main weak points of these architectures are their communication capabilities (relatively high latency) and often the heterogeneity of their hardware. We present in this paper a distributed implementation of the process network model on heterogeneous distributed hardware. The different processing power of the connected computers is a good support for the different computation needs of the networked processes. We have chosen to use the Common Object Request Broker Architecture (CORBA) [9] middleware to handle the communications for its interoperability properties. Indeed, each process of the process network can be written in a different language and run on a different hardware, provided that these are supported by the chosen Object Request Broker (ORB). In addition of the heterogeneity, our implementation presents the following characteristics: *This work has been supported by the ITEA 99038 project, Sophocles.
32 9 automation of data transfer between distributed processes 9 dynamic and interactive linking of the processes to form the data flow 9 hybrid data-driven, demand-driven data transfer protocol, with thresholds for load balancing 9 the implementation was carried out such as to enable a distributed or local execution without any change to the program source. This paper is organized as follows. In section 2, we motivate our approach and we present our implementation. Section 3 describes a process network deployment and distributed execution. The transfer strategies (demand and data driven) are detailed in section 4. And we finally outline our conclusions and plans for future work in section 5. 2. DESIGN AND IMPLEMENTATION 2.1. Related work
The Kahn process network model has been proposed by Kahn and MacQueen [6, 7] to easily express concurrent applications. Processes communicate only through unidirectional FIFO queues. Read operations are blocking. The number of tokens produced and their values are completely determined by the definition of the network and do not depend on the scheduling of the processes. The choice of a scheduling of a process network only determines if the computation terminates and the sizes of the FIFO queues. Some networks do not allow a bounded execution. Parks [ 10] studies these scheduling problems in depth. He compares three classes of dynamic scheduling: data-driven, demand-driven or a combination of both with respect to two requirements: 1. Complete execution (the application should execute completely, in particular if the program is non-terminating, it should execute forever). 2. Bounded execution (only a bounded number of tokens should accumulate on any of the queues). These two properties are shown undecidable by Buck [4] on boolean dataflow graph which are a special case of process networks. Thus they are also undecidable for the general case of process networks. Data-driven schedules respect the first requirement, but not always the second one. Demand-driven schedules may cause artificial deadlocks. A combination of the two is proposed by Parks [ 10] to allow a complete, unbounded execution of process networks when possible. In the context of a distributed execution of a process network, the process execution is inherently asynchronous. We have thus chosen a completely asynchronous scheduling: each process runs in its own thread that is scheduled by the underlying operating system. As explained by Parks and Roberts in [ 11 ], who use a similar scheduling technique, using bounded communication queues allow for a fair execution of the process network. This blocking write when the output queue is full can lead to deadlocks. Determining a priori if the queue length is large enough to avoid such deadlocks is undecidable. We provide a way for the user to modify this length at runtime.
33 Several implementations of process networks are used for different purposes: for heterogeneous modeling with PtolemyII [8], for signal processing application modeling with YAPI [5] and for metacomputing in the domain of Geographical Information Systems with Jade/PAGIS [13]. To our knowledge, only the Jade/PAGIS implementation and the one by Parks and Roberts [ 11 ] are distributed. Parks and Roberts use the Java Object Serialization to automate the distribution of the network processes while we use a central console to deploy the processes. In Jade, all communications proceed through a central communication manager while in Parks and Roberts' and our implementation the processes communicate directly. This allows a greater scalability. Only our implementation allow the coupling of processes written in different languages. 2.2. Design directions The design of our distributed process network implementation was done so as to: 9 enable users to simulate their network model quickly and effectively, 9 keep the source compatibility with the Yapi library: this library developed by Philips [5] implements the process network model for a local execution, Yapi is a C++ library which focuses on signal processing applications, 9 enable the distributed and the local execution without any change to the program source. The idea of the Yapi syntax is to group processes into process networks. The processes communicate via ports. These ports are linked by point-to-point unidirectional FIFO queues. A process network can be seen as a process and used in the same way. This hierarchical construction allows an easy modeling of complex applications. For this study, we have completely reengineered Yapi to be able to distribute any application over a CORBA bus without any change to the application code. In our implementation the communications are hidden to the programmer who can though configure the data transfer parameters. The reader can refer to [3] for more implementation details. 3. PROCESS NETWORK DISTRIBUTION 3.1. Deployment To control the distributed process networks, a console has been developed. It consists of a program which controls the processes connection and execution by the use of a simple language. It also provides a frontend used for monitoring. The presence of a manager program is contrary to the peer-to-peer character of component systems. However, the console is minimal and serves only as collaboration control. All the communications between the components are done without involving the console through the distributed FIFO queues presented in section 3.2. The FIFO links that form the process network are made interactively (or via a script) by this console. The use of a console allows more flexibility in the connection choice and a dynamic control of the components and the communication parameters. The use of an interactive console and the fact that the FIFOs are bounded also allow for an incremental development where computations can start even if the application is not complete. When all the output queues are full, the computation is suspended and can resume as soon as a consuming component is attached to the not-yet-connected output queues.
34
3.2. Distributed FIFOs The FIFO queues are completely distributed, and distributed process networks communicate directly, without a central point contrarily to what is done in Jade [ 13]. These queues implement the blocking read needed by the process network model but, as they are bounded, the write may block also if the FIFO queue is full. A deadlock can appear, but as the execution is fully distributed, deadlock detection is difficult and has not been implemented. To guarantee the code reuse with our implementation, the implementation of the distributed FIFOs must be done without programmer intervention or code change. This was done by encapsulating the distributed FIFOs (the CORBA object) in the FIFOs. The figure 1 shows the structure of the FIFO objects. For the programmer, no difference exists between the local and the distributed FIFO queue. To determine if the FIFO is a distributed FIFO or not, the runtime uses its ports. When a FIFO is local to the program, it has an input and output port. On the other hand, the distributed FIFO is represented by two half FIFOs, the output FIFO queue (producer side) and the input FIFO queue (consumer side). Each half FIFO has one port (an input port for the input FIFO, and an output port for the output FIFO) and should be linked to the other half FIFO. The runtime uses this property to activate the CORBA object only on the distributed FIFOs, and thus, the FIFO distribution is transparent for the programmer.
F,FO
_•
OutputPort
I
write
Data
write
CorbaObjRef
~,FO
I Da-~a 11 writ~Corba readi~__
le ad
o
Distributed FIFO ask
offer
full satisfyRequest sync_ask empty last_request
|
o~.~ ~ - -
(.1
getLength unlink
1
Eo.Oo ~u.
noUfy4Send
link
Distributed FIFO ask
~-<--
~_ read
InputPort
offer
full satisfyRequest sync_ask getLength unlink
empty last_request notify4Send
ilink
Figure 1. Distributed FIFOs implementation
To exchange data between distributed FIFO, we implement a communication protocol based on thresholds. This protocol which will be detailed in section 4 is a hybrid of data-driven and demand-driven transfers.
4. TRANSFER STRATEGIES In this section, we are interested in the data transmission between FIFOs. The transfer strategies were implemented having in mind the minimization of communications between FIFOs, the overlapping of communications by computations, and the load balancing.
35
4.1. Threshold based protocol The communication protocol governs the exchange of data between the FIFOs. A communication can be triggered by any of the two FIFOs in a hybrid data-driven, demand-driven protocol. This protocol is completely distributed, no central authority directs the communication. Each FIFO queue handles its data transmission independently of the rest of the process network. To manage the communications, two thresholds on the number of elements of the FIFO queue have been defined: 9 a maximum threshold (for the producer FIFO queues) which indicate that offering a part of its tokens is necessary to avoid overflowing, 9 a minimum threshold (for the consumer FIFO queues) which indicates that it is necessary to ask the linked producer half-queue for some tokens to avoid idle time. The distributed FIFO IDL interface described bellow defines the methods used for the initialization, the interaction with the other FIFOs, and the recovery of informations. This interface which is implemented by the distributedFifo class is not accessible to the user, but used implicitly by the distributed FIFO queue to communicate with the other distributed FIFO queue and the manager. interface
fifo
base
interface
{ void link(in string refFifo); void unlink(); u n s i g n e d long g e t L e n g t h ( ) ;
}; interface
fifo
int
: fifo
base
interface
{ oneway void a s k ( ) ; oneway v o i d f u l l ( ) ; boolean offer(in eltSequence buffer); oneway v o i d s a t i s f y R e q u e s t (in eltSequence buffer); void empty(); void sync_ask(inout eltSequence buffer); bool notify4Send();
}; The 1 i n k and u n l i n k permit to create or remove a link between two FIFOs. The a s k and s a t i s f y ' R e q u e s t methods are used respectively by the input FIFO to ask for data, and the output FIFO to response to an a s k . s y n c _ a s k is a synchronous version of the a s k method. The f u l l and e m p t y methods are used to indicate the FIFO state, while the o f f e r and n o t 2 f y 4 S e n d are used by the output FIFO to send data to the following FIFO respectively with and without event notification. It should be noticed that communications are vectorized. Indeed we transfer token sequences together and not each token at a time.
36 4.2. Transfer policies The two next paragraphs present the two transfer policies. It can be demand driven with event notification, and data driven without event notification. The other cases are implemented but not used by the runtime (demand driven without event notification by s y n c a s k method and data driven with event notification by n o t i f y 4 S e n d / s y n c _ a s k methods). However, the programmer can modify the transfer strategy by calling the c h a n g e P o l i c y method. Below, we suppose that the two FIFOs F_out and F_in are linked, the first one being the output FIFO and the second one, the input FIFO. Demand driven
When the transfer is demand driven, it is with event notification. When the token number becomes lower than the minimum threshold, the F_in FIFO notifies ( a s k ) the F_out FIFO which responds by sending data ( s a t 2 s f y R e q u e s t ) . The demand driven without event notification protocol was not used because such requests are synchronous and the process would be blocked until the availability of the requested data. With event notification, the process can continue computing if there are still some tokens in its input FIFO (F_in). Data driven
When the transfer is data driven, it is without event notification. When the token number in the FIFO F_out exceeds its maximum threshold, the FIFO sends data ( o f f e r ) to the F_in FIFO. To avoid the overloading of the F_in FIFO, the request returns a result. This result indicates if the receiver FIFO can receive or not other requests from the FIFO F_out. If the F_in FIFO is full, the F_out FIFO will not send any data until it receives a request for data from F_in.The offer requests being asynchronous (return a result), a separate thread manages this communication mode. A data driven protocol with event notification would involves two remote requests, and the use of this mode together with the demand driven protocol with event notification does not guarantee the order of the data reception. All the communications are hidden to the programmer, only the links between the distributed FIFOs determine the data exchange between processes. For the distributed FIFOs, the communications can be triggered in two ways: 1. Inside the process" according to whether the distributed FIFO is an input or an output FIFO, the communication triggering is done in two ways: 9 when a read operation is executed in an input FIFO (figure 2), a pre-read processing is started to ask eventually for data (if the number of tokens in the FIFO is below the minimum threshold). 9 on the other hand, when a write operation is executed in an output FIFO (figure 3), post-write processing is started to eventually send data to the following FIFO (if the number of tokens in the FIFO is above the maximum threshold). 2. Outside the process: this is done directly by the remote invocations from the linked FIFOs which ask for data or send data. However, to adapt the communications to particular applications, some methods have been implemented to give to the programmer the possibility to configure the communications, by
37
.... --
,- . . . . . . . . . . . . . .
..LI .............. ~ (~
..... ~ . . . . . . .
-
- - basic_read
~I
-~ ~I
I i
I
Figure 2. Read and pre-read operations in the distributed FIFO
_,
o.,.
....
~ ,
-
>
(~
:(')
_ _(2)_.....
.... :: 1
Figure 3. Write and post-write operations in the distributed FIFO
setting some communication parameters: minimum and maximum threshold, maximal FIFO capacity, exchange buffer's maximal size. 5. CONCLUSION We have described here our distributed implementation of process networks using CORBA. This implementation is based on the assembly of software components to form a distributed application. Each software component represents a process network which can be hierarchical, and thus exploits the parallelism of the process network model. The CORBA architecture choice is motivated by the handling of the hardware and software heterogeneity. The mapping and the control of the scheduling of the processes is carried out explicitly via a console interface. That makes it feasible to start an application respecting the process network model before its complete implementation. There are a lot of important issues we plan to investigate in the l~ear future. The most important is the dynamicity aspect of the network. Our previous implelhentation of a subcase of the process network model [1, 2] provides more dynamicity, where it is possible to migrate a process or to replace it by another, and our goal is to support these two dynamicity aspects in the general model. To achieve this, two difficulties must be overcome: 9 How to retrieve the internal state of the processes, and how to resume the computation. 9 How to retrieve the contents of the local FIFO queues. Moreover, we are interested in the use of the process network model as an execution model for the applications of multidimensional signal processing based on Array-OL [12], which is a programming language dedicated to systematic signal processing. REFERENCES
[1] Abdelkader Amar, Pierre Boulet, and Jean-Luc Dekeyser. Assembling dynamic com[2]
ponents for metacomputing using CORBA. In Parallel Computing 2001, Naples, Italy, September 2001. Lecture Notes in Computer Science. Abdelkader Amar, Pierre Boulet, and Jean-Luc Dekeyser. Towards distributed process networks with CORBA. Parallel and Distributed Computing Practice on Algorithms, 2003. Special Issue on Parallel and Distributed Computing Practice on Algorithms, to appear.
38 [3]
[4] [5]
[6]
[7]
[8] [9]
[ 10] [ 11 ] [12]
[13]
Abdelkader Amar, Pierre Boulet, Jean-Luc, and Frans Theeuwen. Distributed process networks using half FIFO queues in CORBA. Research Report RR-4765, INRIA, March 2003. Joseph T. Buck. Scheduling Dynamic Dataflow Graphs with Bounded Memory Using the Token Flow Model. PhD thesis, University of California at Berkeley, 1993. E.A. de Kock, G. Essink, W. J. M. Smits, P. van der Wolf, J.-Y. Brunel, W. M. Kruijtzer, P. Lieverse, and K. A. Vissers. YAPI: Application modeling for signal processing systems. In 37th Design Automation Conference, Los Angeles, CA, June 2000. ACM Press. Gilles Kahn. The semantics of a simple language for parallel programming. In Jack L. Rosenfeld, editor, Information Processing 74: Proceedings of the IFIP Congress 74, pages 471-475. IFIP, North-Holland Publishing Co., August 1974. Gilles Kahn and David B. MacQueen. Coroutines and networks of parallel processes. In B. Gilchrist, editor, Information Processing 77, pages 993-998. North-Holland, 1977. Proc.IFIP Congress. Edward A. Lee. Overview of the Ptolemy Project. University of California, Berkeley, March 2001. Object Management Group, Inc., editor. Common Object Request Broker Architecture (CORBA), Version 2.6. http://www.omg.org/technology/documents/ formal/corba iiop. htm, December 2001. Thomas M. Parks. Bounded Scheduling of Process Networks. PhD thesis, University of California at Berkeley, 1995. Thomas M. Parks and David Roberts. Distributed process networks in java. In International Workshop on Java for Parallel and Distributed Computing, Nice, April 2003. Julien Soula, Philippe Marquet, Jean-Luc Dekeyser, and Alain Demeure. Compilation principle of a specification language dedicated to signal processing. In Sixth International Conference on Parallel Computing Technologies, PaCT 2001, pages 358-370, Novosibirsk, Russia, September 2001. Lecture Notes in Computer Science vol. 2127. Darren Webb, Andrew Wendelborn, and Kevin Maciunas. Process networks as a highlevel notation for metacomputing. In Workshop on Java for Parallel and Distributed Computing (IPPS), Puerto Rico, April 1999.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
39
An efficient data race detector backend for DIOTA M. Ronsse a, B. Stougie b, J. Maebe ~, F. Cornelis ~, and K. De Bosschere ~ aELIS Department, Ghent University, Belgium bparallel and Distributed Systems, Delft University of Technology, The Netherlands In this paper, we describe a data race backend we developed for DIOTA. DIOTA (Dynamic Instrumentation, Optimisation and Transformation of Applications) is our generic instrumentation tool, and this tool uses so-called backends to use the information gathered using the instrumentation. Our novel data race backend uses innovative technologies like multilevel bitmaps, snooped matrix clocks and segment merging in order to limit the amount of memory used. The tool was implemented for Linux systems running on IA32 processors and is fully operational. 1. I N T R O D U C T I O N The never ending urge for faster and more robust computers, combined with the existence of cheap processors causes a proliferation of inexpensive multiprocessor machines. Multithreaded applications are needed to exploit the full processing power of these machines, causing a widespread use of parallel applications. Even most contemporary applications that are not CPU-intensive are multithreaded because the multithreaded paradigm makes it easier to develop servers and applications with an MDI (multiple document interface) such as Windows programs to improve responsiveness to actions done by the user, etc. However, developing multithreaded programs for these machines is not easy as it is harder to get a good view on the state of a parallel program. This is caused by the fact that there are a number of threads 1 running simultaneously. Moreover, the very fact that the computation is split into simultaneously executing parts can introduce errors that do not exist in sequential programs. These synchronization errors show up because parallel programs are developed in order to let a number of threads work together on the same problem, hence they will share data. It is of paramount importance that the accesses performed by the different threads are properly synchronised. A lack of synchronisation will lead to data races. More specifically, such a data race occurs when two or more concurrently executing threads access the same shared memory location in an unsynchronised way, and at least one of the threads modifies the contents of the location. As data races are (most of the time) considered bugs, they should be removed. Unfortunately, data races are difficult to find because their occurrence depends on small timing variations. Although it is possible to detect data races using a static approach this is not feasible for nontrivial programs. Therefore, most data race detection tools detect data races dynamically during an actual program execution. 1in this paper we will consider an execution of a parallel program as being a process consisting of N threads executing on a machine with N processors.
40 2. DATA RACE DETECTION As mentioned before, a dynamic data race detector finds data races that occur during a particular execution. In order to detect the conflicting memory operations, one should collect these operations during a particular execution together with information about their concurrency. The concurrency depends on the (order of the) executed synchronisation operations. Basically two (dual) methods exist for detecting data races using collected memory and concurrency information: 9 for each access to a global variable, the access is compared against previous accesses by other threads to the same variable. This requires, for each global variable, information about past accesses. It is obvious that this will lead to a huge memory consumption, especially if each memory location is a potential global variable with life time equal to that of the program itself, as is the case for e.g. programs written in the C language. This method can however be applied for programs with a rigorous object model that is adhered to, such as Java programs as is shown in [4]. 9 for all parallel pieces of code, the memory operations are collected and compared. This method is based on the fact that all memory operations between two successive synchronisation operations (called segments from now on) satisfy the same concurrency relation: they are either all concurrent or not concurrent with a given operation and therefore with the segment containing the latter operation. Given the sets L(i) and S(i) containing the addresses used by the load and store operations in segment i, the concurrent segments i
((L(j)U S(j))N S(i)) r O. Therefore, data race detection basically boils down to collecting the sets L(i) and S(i) for all segments executed and comparing parallel segments. Note that (contrary and j contain racing operations if and only if
((L(i)US(i))AS(j))
U
2"
to the first method) a data race will be detected at the end of the segment containing the second racing instruction, possibly a long time (e.g. at the end of the program) after the actual occurrence of the data race. It is clear that the second method is better suited for programming languages with unconstrained life time of and access to shared variables. As we wanted to develop a tool that is programming language independent, we opted for this method. The remainder of this papers is organised as follows: first we will give a general overview of the infrastructure that supports our implementation of the data race detection tool that is being discussed here. Next, we will explain the paradigms and techniques that we use to detect data races and to make sure that the resulting tool can be used on wide range of real-world applications and with a reasonable slowdown factor. Finally, we will present some experimental evaluation results and the conclusions of this paper, along with our future plans. 3. DIOTA 3.1. General overview
DIOTA (DynamicInstrumentation, Optimizationand Transformationof Applications) is implemented as a shared library for the Linux/80x86 platform. It instruments programs at the machine code level, so it is completely compiler- and language-agnostic and can also cope with
41 hand-written assembler code. It has support for extensions to the 80x86 ISA such as MMX, 3DNow! and SSE and is written in an extensible and modular way so that adding support for new instructions and new types of instrumentation is easy. An environment variable is used to tell the dynamic linker to load the DIOTA library whenever a dynamically linked application is started 2. An i n i t routine allows DIOTA to be activated before the main program is started, after which it can gain full control and start instrumenting. The instrumentation happens gradually as more code of the program (and the libraries that it uses) is executed, so there is no need for complex analysis of the machine code to construct control-flow graphs or to detect code-in-data. The instrumented version of the code is placed in a separate memory region (called the "clone"), so the original program code is not touched and as such neither data-in-code nor the variable-length property of 80x86 instructions pose a problem. Once DIOTA encounters an instruction of which the successor is unknown (such as a jump with a target address that lies in a register or a return instruction), it inserts a trampoline in the clone. Such a trampoline will pass the actual target address to DIOTA every time the instrumented version of this instruction is executed. This way, DIOTA can check whether the code there is already instrumented and if not, do it now. Next, the instrumented version of the target code is executed. Since every exit from an instrumented block leads either to a trampoline or to another instrumented block, DIOTA remains in control. What is extremely important in the light of data race detection however, is that during this process no data is relocated and as such all addresses used by the program remain the same as in an uninstrumented execution, plus or minus a certain delta due to memory occupied by the DIOTA library. As such, all accessed memory addresses can be easily recorded without the need for performing any kind of relocation afterwards. The behaviour of DIOTA can be influenced by using so-called backends. These are small (and not so small) dynamic libraries that link against DIOTA and tell it what kind of instrumentation should be performed. They can ask for any dynamically linked routine to be intercepted and replaced with a routine of their own, ask to be notified of each memory access, of each basic block that is executed and of each system call that is performed (both before and after their execution, so their behaviour can be modified as well as analysed). DIOTA can also handle multi-threaded programs and supports the instrumentation of exception handlers. Further technical details and in-depth discussion on the issues surrounding the interception of dynamically linked routines, can be found in [8]. 3.2. Data race detection b a c k e n d
As described above, the data race detection backend is implemented as a shared library that uses the services provided by DIOTA. The backend request DIOTA to instrument all memory operations and to intercept all p t h r e a d synchronisation operations. The memory operations are intercepted to construct the L(i) and S(i) lists for all segments and the synchronisation operations are intercepted to detect concurrent segments. The problems faced by most contemporary data race detectors is the huge overhead they introduce. The fact that an execution will slow down considerably is primarily caused by the interception of all memory accesses. Indeed, for each memory access a number of operations 2This is the only prerequisite for DIOTA to be able to instrument a program: the program must be dynamically linked. Fortunately, this is the case for all but a few system-maintenanceapplications.
42 will have to be performed by the backend. There exist tools that try to reduce the number of memory operation that have to be checked; e.g. static analysis could reveal that certain memory regions will only be used by one thread. This is of course not possible if we perform data race detection at the lowest level (the machine instruction level, where we could encounter memory operation with unbound addresses, e.g. as a result of the use of a pointer in a C-program), but it is viable for languages were access to variables is strictly controlled and there are no pointers to bypass this protection (e.g. Java [4]). Our tool throws away a small portion of all memory operations: the memory operations executed by the first thread and before a second thread is created are not checked, as they can not be part of a data race. A second reason for the large overhead is the huge memory consumption exhibited by most data race detectors. This is caused by the fact that memory operations have to be compared with memory operations executed in parallel with and in the past of this operation. Keeping track of the old memory operations and the concurrency information requires a lot of memory and we will go to great length to lower the memory consumption, as will be explained in detail in the next few paragraphs.
Collecting and comparing the addresses of the memory operations As mentioned before, two lists of addresses are used for every segment i: the load operations are collected in L(i) and the store operations in S(i). These addresses are collected in a bitmap (one for each list). Such a bitmap contains a 1 on place i if address i was used. As a linear bitmap would require 232/8 = 512 MiB, a multilevel bitmap is used. At this moment we use a 9-level bitmap: the first 8 levels contain page tables while the last level contains the actual bitmap. The highest level is addressed using the 3 highest bits of the addresses. This level contains pointers to page tables at the second level. These page tables are indexed using the next 3 bits and contain pointers to the actual bitmaps. This continues for the next 6 levels, each level indexed using the next 3 bits. The actual bitmaps are indexed using the remaining 8 bits of the address. Although the use of a bitmap means that some information is lost - two or more accesses to the same address count as one - this is not a problem for data race detection. Indeed, if one of these accesses is involved in a race, the subsequent accesses are also involved in a data race. The use of multilevel bitmaps makes it possible to compare the segments in an efficient way.
Detecting and comparing parallel segments In our race detection tool, we use a classical logical vector clock [9] to detect concurrent segments. As vector clocks are able to represent the happened-before [6] relation (they are strongly consistent), two vector clocks that are not ordered must belong to concurrent segments. This gives us an easy way to detect concurrent segments.
Detecting Obsolete Segments For each segment i encountered during the program execution, we build two lists: L(i) and Of course, if we keep all these lists in memory, we will run out of memory. However, it is possible to throw away old information: if it is impossible that a new segment will be parallel with an 'old' segment i, we can throw away the lists L(i) and S(i), effectively freeing memory. Detecting the 'old' segments can be done using matrix clocks.
S(i).
43 Logical matrix clocks are traditionally used to discard information in a distributed environment: the componentwise minimum of the columns yields the maximum number of segments per thread that can be discarded [ 14]. However, in practice we can discard more information than is indicated by logical matrix clocks as logical clocks capture causality, which is one of the weakest forms of event ordering. In a particular execution, all events are executed in some order (not specified by the program), even if they do not have a causal relationship. Classical logical clocks are not able to capture this kind of additional execution-specific ordering. Our data race detection method uses clock snooping [5]. Instead of maintaining matrix clocks, a snooped matrix clock is built each time a segment ends, using the latest vector clock of the processors. All segments that have a vector clock smaller than the componentwise minimum of the snooped matrix clock can be discarded as they will not be needed in the future anymore.
Merging Segments For most parallel applications, logical matrix clocks succeed in throwing away obsolete information. However, there are applications where the segment lists never become obsolete, e.g. because there is little synchronisation between the threads. The lack of synchronisation will preclude segments from becoming causally ordered before new segments, preventing memory from being freed. It is however possible (under strict cirumstances) to lower the memory consumption somewhat by combining segments. This technique [3] is based on the fact that, if all new segments will be ordered equally (parallel with or ordered after) with two segments i and j, we can merge the segments i and j. Merging boils down to removing the segments i and j from the list and adding a new segment j' with L(i') = L(i)U L(j) and S(i') = S(i)U S(j). Of course, we lose some information with merging: if a data race is detected, we no longer know if segment i or j is the cause. Although merging segments belonging to two different threads is possible, at this moment DIOTA only merges successive segments belonging to the same thread.
4. EXPERIMENTAL EVALUATION During the development of our data race backend we used a multitude of small parallel programs and it turned out that our tool was able to find the data races we intentionally introduced. Until now, only a small number of large parallel applications have been benchmarked. One of the benchmarks we used is the Mozilla web browser, a very large, complex and multithreaded application. The table below show some results we obtained with some benchmarks on a dual Celeron processor (500MHZ, 128KiB cache) system running Linux kernel 2.4.19. Four different executions were tested: a normal execution, an execution under control of DIOTA but without adding instrumentation, the same execution but with instrumentation of all memory operations, and finally an execution with the full blown data race detection. Maximum memory overhead was 3.4 x. It is clear that the slowdown is rather high. Fortunately, this is not a real problem since the data race detector can run unattended, e.g. overnight. The effort invested in methods for limiting the memory consumption clearly pays off: the memory consumption is moderate, considering the information that is needed to perform data race detection.
44 program
normal exec. time mozilla 7.50 LU.cont -p4 8.06 fft-p4-m22 11.47 radix-p4-n4194304 6.96 cholesky-p4 inputs/tk29.o 1 0 . 4 3 ocean-p4-n514 15.59 radiosity -p 4 -batch -room 27.50 water-spatial < input4 30.70
no instrum, time slowd, 35.00 4.67x 9.59 1.19x 27.59 2.41x 11.74 1.69x 12.84 1.23x 17.56 1.13x 90.14 3.28x 52.51 1.71x
diota memory instrum, time slowd, 169.00 22.53x 54.15 6.47x 200.37 17.48x 137.39 19.73x 310.74 29.79x 339.06 21.75x 1157.45 42.09x 742.27 24.18x
data race detection time slowd. 401.00 53.47x 85.74 10.64x 393.36 34.31x 244.18 35.07x 581.97 55.80x 667.14 42.80x 6 8 0 5 . 6 1 247.49x 1 5 6 6 . 0 4 51.01x
Table 1 Benchmark results.
5. RELATED W O R K
Although much theoretical work has been done in the field of data race detection [ 1, 2] few implementations for general systems have been proposed. Tools proposed in the past had limited capabilities: they were targeted at programs using one semaphore [7], programs using only post/wait synchronisation [ 11 ] or programs with nested fork-join parallelism [2, 10]. In previous work, we developed RECPLAY [ 12] for Solaris running on the SPARC architecture. This tool was geared exclusively at data race detection and contained an instrumentation engine because this was necessary to gather the accessed memory locations. Later, the instrumentation engine was separated under the name JiTI (Just-in-Time Instrumentation), with the data race functionality remaining as a special purpose backend. Recently, JiTI has been ported to the 80x86 architecture and has since been greatly enhanced and extended, so that another name change was advisable. The DIOTA name captures the (both current and potential) capabilities of the new framework quite well. A major difference between DIOTA and JiTI is that DIOTA also instruments code in dynamic linked libraries. Also, the data race backend of DIOTA is much more powerful than the JiTI version, e.g. segment merging was added recently. The backend is still being actively developed and improved regularly. Most of the previous work, and also our tool, is based on Lamport's happened-before relation. This relation is a partial order on all synchronisation events in a particular parallel execution. Therefore, by checking the ordering of all events and by monitoring all memory accesses, data races can be detected for one particular program execution. Another approach is taken by a more recent race detector: Eraser [ 13]. It goes slightly beyond work based on the happened-before relation. Eraser checks that a locking discipline is used to access shared variables: for each variable it keeps a list of locks that were held while accessing the variable. Each time a variable is accessed, the list attached to the variable is intersected with the list of locks currently held and the intersection is attached to the variable. If this list becomes empty, the locking discipline is violated, meaning that a data race occurred. Unfortunately, Eraser detects many false data races: as Eraser is not based on the happened-before relation it has no timing information whatsoever. For instance, in theory there is no need to synchronise shared variables before multiple threads are created.
45 6. C O N C L U S I O N S A N D F U T U R E W O R K
In this paper, we presented a tool called DIOTA and its data race backend. The tool is able to instrument complex programs and with the help of the backend it can do data race detection on them. The data race detection occurs by dividing the execution into segments and by keeping a history of accessed memory locations for each segment. The segments themselves are ordered through the use logical vector clocks, which are updated by occurrences of synchronisation operations. In the future, we want to improve the speed of the data race backend in several ways. One technique is to remove redundant checks of addresses: once an address has been written to once in a segment, all subsequent read and write operations to the same address are irrelevant as far as data race detection is concerned. Therefore, it is possible to greatly optimise the instrumentation. Simple testing has shown that in e.g. the Mozilla web browser, 60% of the memory accesses use the same address as the previous memory access. Also when looking at loops, it is possible to move a lot of instrumentation code out of the loop body, e.g. the instrumentation of the write operations to loop counters. Another enhancement we propose, is to treat exception handlers properly with regard to data race detection. Currently, memory operations inside exception handlers are considered as being part of the thread in which the exception occurred. However, since exceptions are raised asynchronously, they should be regarded as very short lived separate threads instead. The latest version of DIOTA can be downloaded from h t t p : / / w w w . e 1 i s . U G e n t . b e /
diota/.
REFERENCES [1]
S.V. Adve, M.D. Hill, and R.H.B. Netzer. Detecting data races on weak memory systems. In Proceedings of the 18th Annual Symposium on Computer Architectures, pages 234-243, May 1991. [2] K. Audenaert and L. Levrouw. Space efficient data race detection for parallel programs with series-parallel task graphs. In Proceedings of the third Euromicro Workshop on Parallel and Distributed Processing, pages 508-515, San Remo, January 1995. IEEE Computer Society Press. [3] M. Christiaens, M. Ronsse, and K. De Bosschere. Bounding the number of segment histories during data race detection. Parallel Computing, 28(9):1221-1238, 9 2002. [4] Mark Christiaens and Koen De Bosschere. Trade, a topological approach to on-the-fly race detection in java programs. In Java Virtual Machine Research and Technology Symposium (JVM'O1), pages 105-116. USENIX, April 2001. [s] Koen De Bosschere and Michiel Ronsse. Clock snooping and its application in on-thefly data race detection. In Proceedings of the 1997 International Symposium on Parallel Algorithms and Networks (I-SPAN'97), pages 324-330, Taipei, December 1997. IEEE Computer Society. [6] Leslie Lamport. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21 (7):558-565, July 1978. [7] H.I. Lu, P.N. Klein, and R.H.B. Netzer. Detecting race conditions in parallel programs that use one semaphore. Technical report, Brown University, 1993. [8] J. Maebe, M. Ronsse, and K. De Bosschere. DIOTA: Dynamic instrumentation, opti-
46
[9]
[10]
[ 11 ]
[ 12] [13]
[14]
mization and transformation of applications. In M. Chamey and D. Kaeli, editors, Compendium of Workshops and Tutorials Held in conjunction with PACT'02: Intl. Conference on Parallel Architectures and Compilation Techniques, Charlottesville, VA, 9 2002. Friedemann Mattem. Virtual time and global states of distributed systems. In Cosnard, Quinton, Raynal, and Roberts, editors, Proceedings of the Intl. Workshop on Parallel and Distributed Algorithms, pages 215-226. Elsevier Science Publishers B.V., North-Holland, 1989. John M. Mellor-Crummey. On-the-fly detection of data races for programs with nested fork-join parallelism. In Proceedings of Supercomputing '91, pages 24-33, November 1991. Robert H. B. Netzer and Barton P. Miller. On the complexity of event ordering for sharedmemory parallel program executions. Intl. Conference on Parallel Processing, pages 9397, August 1990. Michiel Ronsse and Koen De Bosschere. Recplay: A fully integrated practical record/replay system. ACM Transactions on Computer Systems, 17(2):133-152, May 1999. Stefan Savage, Michael Burrows, Greg Nelson, Patrick Sobalvarro, and Thomas Anderson. Eraser: A dynamic data race detector for multithreaded programs. ACM Transactions on Computer Systems, 15(4):391-411, November 1997. G.T.J. Wuu and A.J. Bemstein. Efficient solutions to the replicated log and dictionary problems. Proc. 3rd ACM Symp. Principles Distributed Computing, pages 233-242, New York, 1984. ACM Press.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
47
Pipelined parallelism for multi-join queries on shared nothing machines M. Bamha a and M. Exbrayat ~ aLaboratoire d'Informatique Fondamentale d'Orlrans, Universit6 d'Orlrans, BP 6759, 45067 Orlrans cedex 2, France {mostafa.bamha,matthieu, exbrayat }@lifo.univ-orleans. fr The development of scalable parallel database systems requires the design of efficient algorithms, especially for join, which is the most frequent and expensive operation in relational database systems. Join is also the most vulnerable operation to data skew and to the high cost of communication in distributed architectures. Moreover, for multi-join queries, the problem of data-skew is more complicated because the imbalance of intermediate results is unknown during static query optimization. In this paper, we show that the join algorithms we presented in our earlier papers can be applied efficiently in various parallel execution strategies making it possible to exploit not only intra-operator parallelism but also inter-operator parallelism. These algorithms minimize the communication and synchronization costs while guaranteeing a perfect load balancing during each stage of join computation even for highly skewed data. 1. INTRODUCTION Join is the most frequent operation of parallel relational database management systems (PDBMS). It is also the most expensive one, due to its vulnerability to data skew and to the high cost of communications in shared nothing architectures. Research has shown that join is parallelizable with near-linear speed-up on Shared Nothing machines only under ideal balancing conditions. Data skew can have a disastrous effect on performance [ 1] due to the high costs of communications and synchronizations in this architecture. Many algorithms have been proposed to handle data skew for a simple join operation, but little is known for the case of complex queries leading to multi-joins [2, 3, 4]. In particular, the performance of PDBMS has generally been estimated on queries involving only one or two joins [5]. However the problem of data skew is more acute with multi-joins because the imbalance of intermediate results is unknown during static query optimization [6]. Such algorithms are not efficient for many reasons. First, they are not scalable (and thus cannot guarantee linear speedup) because their routing decisions are generally performed by a coordinator processor while the others are idle. Second, they cannot solve load imbalance problems as they base their routing decisions on incomplete or statistical information. Finally, they poorly handle data skew because data redistribution is generally based on hashing data into buckets, while hashing is known to be inefficient in the presence of high frequencies [2]. On the contrary, the join algorithms we presented in [7, 8, 9], use a total data-distribution information in the form of histograms. In this paper, we show that these join algorithms can be
48 applied efficiently in various parallel execution strategies, making it possible to exploit not only intra-operator parallelism but also inter-operator parallelism. These algorithms are used in order to avoid the effect of load imbalance due to data skew, and to reduce the communication costs due to the redistribution of the intermediate results, which can lead to a significant degradation of the performance. The organization of this paper is as follows: we first recall the notions of PDBMS, join operations and load balancing in section 2. We then present parallel execution strategies for multi-queries in section 3, and propose and discuss the use of pipelining strategies together with the Osfa_j oin algorithm in section 4. We finally sum up in section 5. 2. PDBMS : JOIN OPERATIONS AND LOAD BALANCING
2.1. PDBMS, join operations and data skew The join (or equi-join) of two relations R and S on attribute A of R and attribute B of S (A and B of same domain) is the relation T, written R N S, containing the pairs of tuples from R and S verifying R.A = S.B. The sere• - j o • of S by R is the relation S ~< R composed of the tuples of S which occur in the join of R and S. Semi-join reduces the size of relations to be joined and R N S = R N (S K R) = (R ~< S) N (S K R). In PDBMS, relations are generally partitioned among nodes by horizontal fragmentation according to the values of a given attribute. Three methods are used [10]: hash partitioning, block-partitioning (also called range partitioning) and cyclic (also called round-robin) partitioning. The main parallel join algorithms are: Sort-merge join, Simple-hash join, Grace-hash join and Hybrid-hash join [11]. All of them are based on hashing functions which redistribute relations so that tuples having the same attribute value are forwarded to the same node. Local joins are then computed and their union provides the output relation. Their main disadvantage is to be vulnerable to both attribute mlue skew (imbalance of the output of the redistribution phase) and join product skew (imbalance of the output of local joins) [4, 6]. The former affects immediate performance and the latter affects the efficiency of output of pipelined operations in the case of a multi-join. 2.2. Load balancing in parallel join operations The authors of [4] have identified the two best proposed solutions in conventional and sampling-based parallel join algorithms. All are based on the notion of parallel hashing. In the first category, the extended adaptive load balancing parallel hash join [3] sends all tuples with the same attribute value to the same node. As a result, the algorithm may fail to balance the load when few values have a large weight. Moreover, the algorithm ignores the attribute value skew (AVS) of the probe relation and the join product skew UPS). In the second category of algorithms, virtual process partitioning [2] improves on the previous category but only handles AVS for the build relation. It also ignores JPS. To improve on these methods, the authors of [4] have introduced an algorithm, that minimizes the expected AVS by statistically predicting the result of hashing. However, it still fails to correct JPS. The algorithm is based on hashing, and therefore sensitive to the same effect as earlier methods (an important skew will anyway lead to a JPS). We conclude that all existing methods are sensitive to imbalance when applied multiple times because of JPS. To address this problem, we introduced in [7] a deterministic data redistribution algorithm
49 with near-perfect balancing properties, that dynamically computes exact frequency histograms to avoid JPS. A scalable and portable cost analysis was made with the BSP model [12], leading to general predictions about the effect of relation histograms on performance. The analysis suggests a hybrid frequency-adaptive algorithm (f a _ j o i n algorithm) [7], dynamically combining histogram-based balancing with standard hashing methods. The f a _ j o i n algorithm avoids the slowdown usually caused by AVS and the imbalance of the size of local joins processed by the standard algorithms. It handles skews at the cost of extra processing time. We analyzed this overhead both theoretically and experimentally and concluded that it never penalizes overall performance, even in the absence of skew. However, the performance of f a _ j o i n is sub-optimal when computing the join of highly skewed relations because of unnecessary redistribution and communication costs. We introduced in [9] a new parallel algorithm called Osf a _ j o i n (Optimal symmetric frequency adaptive join algorithm) to perform such joins. O s f a _ j o i n improves on f a _ j o i n by its optimal complexity and by the use of semi-joins. Its predictably low JPS and AVS make it suitable for multi-join queries. Its analyze through BSP predicts a linear speedup and an optimal complexity even for highly skewed data. 3. PARALLEL EXECUTION STRATEGIES FOR MULTI-JOIN QUERIES Several strategies were proposed for the evaluation of multi-join queries [5]. These strategies generally depend on the parallel query execution plan. They can use both intra-operator and inter-operator parallelism, and mostly differ in the way they allocate simple joins to different processors and the way they choose an appropriate degree of parallelism (i.e. the number of processors used to compute each simple join). They can be divided into four main categories presented thereafter.
3.1. Sequential parallel execution Sequential parallel execution is the simplest strategy for evaluating, in parallel, a multi-join query. This strategy does not require inter-operator parallelism since simple joins of the query are evaluated one after the other in a parallel way. It does not induce any pipelined parallelism owing to the fact that a simple join cannot be started until all its operands are entirely available. The execution time of each join is then the execution time of the slowest processor. To reach acceptable performance, the join algorithms used in this strategy should reduce the load imbalance among all the processors. O s f a _ j o i n can be used there to reduce the idle time, owing to the fact that it is insensitive to AVS and JPS, and also proved to guarantee a linear speedup. 3.2. Parallel synchronous execution In addition to intra-operator parallelism, parallel synchronous execution uses inter-operator parallelism [13]: several simple joins can be computed simultaneously on disjoint sets of processors. The main difficulty then lies in the allocation of simple joins to the available processors and in the choice of an appropriate degree of parallelism for each join, so that i) the global execution time of each operator should be of the same order to avoid the latencies induced by the imbalance in computations of the different groups of processors, and ii) the execution time of each operator of a given group of processors (i.e. of a given parallel join) must also be of the same order, as the execution time for each join is the execution time of its slowest processor.
50 The load balancing between the groups of processors depends on the effectiveness of the dynamic balancing strategy used at the time of the query optimization, whereas the load balance of the processors in the same group can be carried out by using the join algorithms we introduced.
3.3. Segmented right-deep execution Contrary to a parallel synchronous strategy, a segmented right-deep execution [14] uses, in addition to intra-operator parallelism, pipelined inter-operator parallelism. Pipelined interoperator parallelism is used in the evaluation of the right-branches of the query tree. O s f a _ j o i n could still be used there. Nevertheless, the initial version of O s f a _ j o i n computes joins sequentially, one after the other, and thus has to be modified to handle pipelining efficiently. 3.4. Full parallel execution Full parallel execution [5] is simply a combination of the preceding strategies, that uses intra-operator, inter-operator, and pipelined inter-operator parallelism. In this strategy, all the simple joins, that compose the multi-join query, are computed simultaneously in parallel using disjoint sets of processors. Inter-operator parallelism and pipelined inter-operator parallelism are exploited according to the type of the query tree. Its effectiveness depends on the quality of the execution plan. The initial version of Osfa_j oin is not really adapted to full parallelism Oust like it was not really adapted to segmented right-deep execution), and thus has to be modified to handle pipelining. In the following section we will present a detailed approach to introduce pipelining in Osfa_j oin. 4. PIPELINING OSFA JOIN: CAN WE PIPELINE A DYNAMIC A L G O R I T H M ? . - -
4.1. Principle Pipelining has been successfully implemented with many classical join algorithms. Nevertheless, these algorithms are generally based on hash join techniques and are thus sensitive to AVS and JPS, which are very likely to occur in a multi-join query. For this reason we propose to adapt O s f a _ j o i n (which is insensitive to data skew) to pipelined joins. This task in fact appears to be difficult. Most of pipeline strategies use a static execution plan. The way data is distributed for each operator is fixed at query compilation time. When one processor produces a result tuple, this latter can be immediately transmitted to and used by the next operator, which does not need to have a global view of the whole set of manipulated tuples. This can be repeated over several consecutive join operators. Long pipeline chains can be constituted, their length being mainly limited by processors' availability. On the contrary, O s f a _ j o i n is a dynamic operator; the definitive distribution of data is determined at execution time, once histograms have been built, i.e. after the former operator has finished. This strongly limits the use of pipelines. We thus propose to use two-operator pipeline chains. The first enhancement consists in parallelizing the creation of histograms for source relations (i.e. using inter-operator parallelism). This means that we organize the execution plan so that as many join operators as possible apply either on two source tables, or on one source table and an intermediate result. In this case we can build the histograms for source tables as soon as the
51 query execution begins. This strategy is very similar to the parallelized build phase that is used for classical right-deep trees. We then build the histograms for computed (intermediate) relations as their tuples are produced. In other words, we interlace the tuple production phase of a join operator with the building phase of the next. By the way we do not need to re-scan the results from the disk. Once the histograms are produced for both tables, we can compute the communication template, then distribute data, and finally compute the join. Unfortunately, computing the communication template is the implicit barrier within the execution flow, that prohibits the use of pipeline chains. From the implementation point of view, we must notice that we do not really compute the local histogram on-the-fly (i.e. each time a tuple is produced), but rather wait for a sufficient number of tuples to be available to compute the corresponding histogram.
4.2. Detailed algorithm Let us consider each join is of the form Ri N Si, where Ri is a source relation and S~ is either a source relation (which should only occur for the first join) or an intermediate result. The execution tree consists of t join operators; the final result is stored as St+l. Let us call Rij the fragment of Ri which is stored on processor j. We use n nodes numbered 1 to n. The global algorithm is then: Algorithm 1 Pipelined Osfa_join For each join Ji - ( R i N Si) Par (on each node) j E [1, n] create local histograms for Rij Hist(Rij) broadcast Hist(Rij) to other nodes, and collect all Hist(Rik), k 7~ j build the corresponding part of the global histogram (each node being assigned a given set of values) for Ri Histj (Ri) If i= 1 (first join) Then create local histograms for Sij Hist(Sij) Else wait for Hist(Sij) to be completed by the preceding join phase Endif broadcast Hist(Sij) to other nodes, and collect all Hist(S~k), k ~ j build global histogram for 5'/Histj (Si) compute communication templates according to Histj(S~) and Histj(Ri) (see [7, 8, 9]) broadcast (and collect) the communication templates exchange data across nodes according to the communication template If i=t (last join) Then compute S(i+l)j = (Rij N Sij) Else compute S(~+I)y = (R~j N S~j) and build Hist(S(~+l)j) on the fly Endif Endpar Endfor
52
4.3. Example Figures 1 and 2 represent an annotated query execution tree of a 3-join query T1 N T2 M T3 t~ T4. Source tables are placed on leaves. Numbers indicate the relative order of operations. Referring to the general algorithm, we can identify TJ, T3 and T4 respectively to R1, R2 and R3, and T2 to $1. Figure 1 illustrates an execution where operators are executed sequentially (i.e. a basic use of Os f a _ j o i n , where we do not use inter-operator parallelism). We first build the histograms for T 1 and T2, then compute the communication templates, distribute data, and execute the join. Once the join is over, we build the histograms for T3 and T1 N T2, and so on. Figure 2 illustrates a pipelined execution of the query tree. Building the histograms of all source tables is executed in parallel. As soon as histograms are available for T1 and T2, the communication template of the first join is computed, then data is transfered, and finally the join is executed. During this last operation, result tuples are scanned in order to produce the histogram of T1 N T2. Once this phase is over, we can compute the communication template for the second join, and so on.
6. Compute comm. templates, exchange data, join T4 and $3 into $4 (result)
5. build Hist(T4) T4
5. build Hist(S3) 4. Compute comm. templates, exchange data, join T3 and $2 into $3
3. build Hist(T3) T3
3. build Hist(S2) 2. Compute comm. templates, exchange data, join T1 and T2 into $2
/2
1. build Hist(T1) T1
1. build Hist(T2) T2
Figure 1. Standard execution of a multi-join query using Os f a _ j o i n
4. Compute comm. templates, exchange data, join T4 and $3 into $4 (result)
1. build Hist(T4) T4
3. Compute comm. templates, exchange data, join T3 and $2 into $3, and build Hist(S3) on the fly
T3
2. Compute comm. templates, exchange data, join T1 and T2 into $2, and build Hist(S2) on the fly
/<
1. build Hist(Tl) Tl
1. build Hist(T2) T2
Figure 2. Pipelined execution of a multi-join query using Os f a _ j o i n
4.4. Discussion We showed in section 4.1 that Os f a _ j o i n could hardly be pipelined in a long chain. Nevertheless our two-operator pipeline achieves several enhancements. We first use a parallel construction of histograms for source relations. Moreover, we combine the actual computation of joins together with the production of the next-join histogram, thus limiting the number of accesses to data (and to disks).
53 Conceming the parallel construction of histograms for source relations, we can notice that the degree of parallelism might be limited by two factors: the total number of nodes available, and the original distribution of data. A simultaneous construction of two histograms on the same node (which occurs when two relations are distributed, at least partially, over the same nodes) would not be really interesting compared to a sequential construction. This inter-processor parallelism does not bring acceleration, but should not induce noticeable slowdown: histograms use to be small, and having several histograms in memory would not necessitate swapping. On the other hand, as relations use to be much bigger than the available memory, we have to access them by blocks. As a consequence, accessing one or several relations does not really matter. Our pipeline strategy will really be efficient if join operators are executed on disjoint (or partially disjoint) sets of processors. Intra-operator parallelism is thus limited, and we have to segment our query trees, similarly to segmented fight-deep trees, each segment (i.e. a set of successive joins) being started when the former is over. These two limitations (degree of parallelism of operators and number of "pipelined" operators) would naturally suppose some minor modifications of the general algorithm presented above. 5. CONCLUSION In this paper we overviewed the existing parallel join algorithms, and the execution strategies of multi-joins queries on shared-nothing machines. We have shown that the algorithms, we introduced in earlier papers can be applied efficiently in different parallel strategies. We proposed a pipelined version of the Os f a _ j o i n algorithm. Our approach consists of two elements: parallelizing the scan of source tables and scanning intermediate results at production time, thus limiting disk accesses. We show that long pipeline chains can hardly be constructed due to the dynamic nature of O s f a _ j o in. We are currently conducting tests that tend to validate the superiority of our approach compared to standard hash-join pipeline algorithms. The pipelined version of O s f a _ j o i n is currently implemented within a parallel database prototype, in the framework of the CARAML ~ Grid project. REFERENCES
[1] L. Harada and M. Kitsuregawa, Dynamic Join Product Skew Handling for Hash-Joins in [2]
[3] [4]
Shared-Nothing Database Systems, in Proc. of the 4th Int. Conf. on Database Systems for Advanced Applications, Singapore, 1995,246-255. D. J. DeWitt, J. F. Naughton, D. A. Schneider and S. Seshadri, Practical Skew Handling in Parallel Joins, in Proc. of the 18th Int. Conf. on Very Large Databases, Vancouver, Canada, 1992, 27-40. K. A. Hua and C. Lee, Handling data skew in multiprocessor database computers using partition tuning, in Proc. of the 17th Int. Conf. on Very Large Databases, Barcelona, Spain, 1991, 525-535. V. Poosala and Y. E. Ioannidis, Estimation of Query-Result Distribution and its Application in Parallel-Join Load Balancing, in Proc. of the 22th Int. Conf. on Very Large Databases, Bombay, India, 1996, 448-459.
1CARAMLproject URL http://www.caraml.org
54 [5]
[6] [7] [8]
[9] [10] [ 11]
[12] [13]
[14]
A.N. Wilschut, J. Flokstra and P. M. G. Apers, Parallel evaluation of multi-join queries, in Proc. of the 1995 ACM SIGMOD Int. Conf. on Management of Data, San Jose, USA, 1995, 115-126. H. Lu, B. C. Ooi and K. L. Tan, Query Processing in Parallel Relational Database Systems (IEEE Computer Society Press, Los Alamos, USA, 1994). M. Bamha and G. Hains, A frequency adaptive join algorithm for Shared Nothing machines, Parallel and Distributed Computing Practices 3(3) (1999) 333-345. M. Bamha and G. Hains, A skew insensitive algorithm for join and multi-join operation on Shared Nothing machines, in Proc. 1 l th Int. Conf. on Database and Expert Systems Applications, London, UK, 2000, 644-653. M. Bamha, An optimal and skew-insensitive join algorithm for Shared Nothing machines, Research Report RR-LIFO-2000-05, LIFO, Universit6 d'Orl6ans, France, 2000. D. J. DeWitt and J. Gray, Parallel database systems : The future of high performance database systems, Communications of the ACM 35(6) (1992) 85-98. D. Schneider and D. DeWitt, A Performance Evaluation of Four Parallel Join Algorithms in a Shared-Nothing Multiprocessor Environment, in Proc. of the 1989 ACM SIGMOD Int. Conf. on Management of Data, Portland, USA, 1989, 110-121. D. B. Skillicorn, J. M. D. Hill and W. F. McColl, Questions and Answers about BSP, Scientific Programming 6(3) (1997) 249-274. M.S. Chen, P.S. Yu and K.L. Wu, Scheduling and Processor Allocation for the Execution of Multi-Join Queries, in Proc. of the Int. Conf. on Data Engineering, Los Alamos, USA, 1992, 58-67. M. S. Chen, M. L. Lo, P. S. Yu and H. C. Young, Using Segmented Right-Deep Trees for the Execution of Pipelined Hash Joins, in Proc. of the 18th Int. Conf. on Very Large Databases, Vancouver, Canada, 1992, 15-26.
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
55
T o w a r d s the H i e r a r c h i c a l G r o u p C o n s i s t e n c y for D S M s y s t e m s : an efficient w a y to share data objects L. Lef6vre a and A. Bonhomme a aINRIA / LIP (UMR CNRS, INRIA, ENS, UCB) Ecole Normale Sup6rieure - Lyon, France [email protected] We present a formal and graphical comparison of consistency models for Distributed Shared Memory systems, based on the programmer point of view. We conclude that most consistencies can be categorized into 3 models depending on their flexibility degree (none, 1 or 2). We propose a new consistency model that provides these 3 flexibility degrees : the Hierarchical Group Consistency (HGC) and present its deployment inside the DOSMOS DSM system. 1. INTRODUCTION Distributed Shared Memory (DSM) systems are now a well recognized alternative for the deployment of a large class of applications. Their main challenge is to manage data consistency while keeping good performances. During last decade, a lot of consistencies have been proposed [2, 3, 7, 10]. However, their definition is made from the designer or hardware point of view. Each definition is thereby often dependent on the DSM designer context. By using large scale clusters (hundreds or thousands of nodes), DSM have to face the scalability problem. How to provide scalable solutions for applications needing a virtual shared space with a large number of computing nodes? In particular, one issue that DSM systems have to address concerns minimizing replication (i.e. reducing the number of messages required to keep the memory consistent) and maximizing availability (i.e. increasing the number of local accesses) of shared data (objects). Our goal is to propose a new taxonomy for consistency models, that can be used by a programmer that deals with the implementation of a distributed application on top of a cluster of machines. This taxonomy is programmer-centric. For this, we use the framework introduced by Hu et al. [9], and we show that from the programmer's point of view, there are only 3 main consistency models. The other ones are just various implementations of those models. Based on this observation, we propose a graphical representation of those models. This representation is based on different degrees of flexibility of the consistency : flexibility about consistent moment (like Release Consistency) or about consistent data (like Entry Consistency). This representation outlines that so far, there is no consistency model with flexibility about consistent processes. Hence, we suggest the Hierarchical Group Consistency (HGC). Consistency is only maintained inside a group of processes rather than between all processes. Furthermore, for each group, various consistency rules can be applied, depending on the type of sharing performed
56 inside a group. This group structure is particularly interesting for heterogeneous clusters and allows the programmer to adapt the consistency management to the application (depending on the sharing degree, for example) and to the execution cluster (depending on the communication performances, for example). The Hierarchical Group Consistency has been implemented in the DOSMOS system [5, 12]. Like in most DSM systems based on weak consistency, DOSMOS provides Acquire and Release operations to set critical sections. However, with HGC, DOSMOS also allows the programmer to specify which data to share between which processes. This paper is organized as follows : Section 2 briefly presents the programmer-centric framework used to define consistency models, and introduces a formal comparison of existing models. Then Section 3 is devoted to a graphical 3D representation of the 3 consistency models. Next, Section 4 presents the Hierarchical Group Consistency model. Section 5 describes the hierarchical group deployment in the DOSMOS system. Section 6 concludes this paper and presents future directions. 2. THREE MEMORY CONSISTENCY MODELS FROM THE PROGRAMMER'S POINT OF VIEW
Research on data consistency has always been a hot topic because of its central position in areas like parallelism, operating systems, distributed systems, etc. However, if a lot of consistencies have been proposed for DSM systems, the formal context of the consistency concept has not been clearly specified. As a consequence, comparing consistencies remains a difficult challenge [1, 8]. In this section, we focus on the consistency concept based on the programmer's point of view. That means that a consistency model is defined based on how it is perceived by the programmer rather than how it is managed or implemented by the system. We base our work on the formal framework introduced by Hu et aL [9] in order to compare different models. Section 2.1 introduces this framework and defines the best known consistency within this framework. Section 2.2 shows how to compare these models with this framework. 2.1. Formal definition of consistency models In their article [9], Hu et all define a memory model as follows : Definition 1 A memory consistency model M is a pair (CM, SYNM) where CM is the set of possible memory accesses (read, write, synchronization) and S Y N M is an inter-process synchronization mechanism to order the execution of operations from different processes. The execution order of synchronization accesses determines the order in which memory accesses are perceived by a process. Accordingly, for each program, there are several possible executions. A program execution is defined as follows.
An execution of the program PRG under consistency model M, denoted as EM(PRG), is defined as an ordering of synchronization operations of the program. With the ordering of synchronization operations, the execution of all related operations are also ordered. Thus, we define the synchronization order of an execution.
Definition 2
The synchronization order of an execution EM(PRG) under consistency model M, denoted as SOM(EM(PRG)), is defined as the set of ordinary operation pairs ordered by the synchronization mechanism S Y N M of M. Definition 3
57 Hence, for any consistency model M, we can define CM and SOM(EM(PRG)). CM deals with how the programmer has to program, and SOM gives the rules used to generate the result. The basic Atomic Consistency (AC), the Sequential Consistency (SC) [11], the Release Consistency (RC) [7], the Lazy Release Consistency (LRC) [ 1], the Eager Release Consistency (ERe) [6], the Entry Consistency (EC) [3] and the Scope Consistency (SsC) [10] can all be defined within this framework. Furthermore, based on those definitions, we easily outline that RC, LRC and ERC have the same definition. That means that LRC and ERC are different implementation of the RC model. In effect, from the programmer's point of view, the results are the same for both implementations. Similarly, we state that SsC is a particular case of EC. As a result, in the following, we only consider AC, SC, RC and EC models. Finally, Hu et all. define a correct program as follows :
Definition 4 A program PRG is said to be correct for the consistency model M, iff for any possible execution EM(PRG), all ordinary conflicting access pairs are ordered by either the program order (PO) or by the synchronization order of execution SOM.
2.2. Formal comparison of consistency models In order to compare consistency models, we define the concept of model equivalence. Definition 5 M1 and M2 are said to be equivalent iff :
a) CM~ = CM~, b) a correct program PRG for M1 is also correct for M2, c) and 2 compatible executions EM1 (PRG) and EM2 (PRG) give the same result. EM1 (PRG) and EM2(PRG) are said to be compatible executions if there do not exist 2 synchronization operations (u, v) such that (u, v) E EM1 (PRG) and (v, u) r EM2 (PRG). Theorem 1 The Atomic Consistency model and the Sequential Consistency model are equivalent. Proof : By definition, CAC -- C s c . Furthermore, AC and SC provide a global order for all read/write operations. Consequently, they are both correct for any program. Finally, for any execution Eac, all accesses are ordered. Yet, Esc only orders operations concerning data accessed several times during the execution. Thus, Esc is included in Eac. For the remaining operations that are in EAC and not in Esc, they concern distinct data. The result is, therefore, the same whatever the execution order. AC and EC are equivalent. More generally speaking, we refer to them as the Strong Consistency Model.
Theorem 2 The Release Consistency model and the Entry Consistency model are not equiva-
lent. Proof : Let us give a counterexample. Let us consider the program and execution shown here. If for EC, x is not associated to l l, then PRG is correct for EC (accesses to x are ordered by the synchronization order), and gives
58 the result (a = 0, b - 1). However, for RC, PRG is also correct, but the result is (a = 1, b - 1). Thus EC and RC are not equivalent.
Corollary 1 The Strong Consistency model the Release Consistency model and the Entry Consistency model are three different consistency models. Proof : The strong consistency model does not define any synchronization variables. It is consequently not equivalent to EC or RC. 3. G R A P H I C A L COMPARISON OF THE THREE CONSISTENCY MODELS Once we have outlined 3 distinct models, we suggest to compare them with a graphical taxonomy based on the user's point of view. Considering a correct program for a model M, this taxonomy allows us to answer the following question : At each moment t of a program execution, if a given process Pi accesses a shared data d, will he read the last value written in the shared memory by another process pj ? In case of a positive answer, we say that at moment t, process pi is consistent with the process pj about the data d. Then, we represent each model with a 3D Accesses between visualization of the answer to this question barriers Time remaining a ~ All the Time by using various parameters (t, d, p). Hence, When Critic each model is represented by a volume made between Acquire and Release up of the triplets (t, d, p) for which the answer All Processes is yes. Basically, (t, d, p) is defined with reWho v v Processes linked Other processes spect to 3 axes: with synchronization When (t) : for time intervals of running apAll MemorySpace plication; What k---v----~ v ) W h a t (d) : for shared memory space; Memory Objects linked Memory Space with synchronization Who (iv) : for application processes. As shown in Fig.l, the axis discretization is Figure 1" When, who, what axes. based on shared objects' access patterns. Accordingly, if we normalize the axes, a plain cube means that a process, for all running execution, for all shared data, reads last value written by any other process. The strong consistency model perfectly fits in that definition and graphical representation (Fig. 2). Figures 3 and 4 graphically represent weak consistencies. With Release Consistency, an access performed between Acquire and Release operations is consistent with other processes on all accessible data. In a barrier case, all the processes have the same view of the shared memory. For the remaining execution, there are some conflict risks for any accessible data. The Entry Consistency model gives a slightly different result from the previous one. In effect, if for the barriers, the behavior is the same, it is rather different for an access in a critical section surrounded by an Acquire and Release operation. The process is consistent with the other processes only for the shared data associated with this synchronization.
59 Who
Who
When
/
WhaI
Figure 2. Strong consistency
]
When
/
What
Figure 3. Release consistency Figure 4. Entry consistency
4. TOWARDS A N E W C O N S I S T E N C Y M O D E L ? THE H I E R A R C H I C A L GROUP C O N S I S T E N C Y (HGC)
who Moving from a strong towards a weak consistency implies a decrease in the When axis decrease. The Entry Consistency model also relaxes the What axis. At this stage, only one dimension remains fixed" the processes one. It would be pertinent to [ make this axis flexible. For an application, all processes do not l Whe, use all shared data. Moreover, in order to provide scalable solutions, it could be interesting to synchronize with barriers a / What subset of running processes. Thus, we propose a new consistency model which groups Figure 5: Graphical representogether processes working on the same synchronization vail- tation of a new model with 3 ables. Such a model has the graphical representation shown in flexible axes Figure 5. Basically, the HGC model is similar to EC since data are associated to a synchronization variable. Furthermore, processes are also associated to this synchronization variable. Thus, let us consider a data modification performed inside a critical section managed by the synchronization variable l. Then, those modifications are forwarded only if they concern data associated to l, and only to processes associated to 1. Furthermore, in the HGC model, it is possible to perform some synchronization barrier for only a subset of the processes. Thus, the HGC model can be defined as follows : The Hierarchical Group Consistency model is defined by : - CHCC : {read(x), write(x), Acq(1), Rel(l), Sync(1)} - (u, v) E SOHcc(EHcc(PRG)) iff 3 a synchronization variable I to which u and v are associated such that: u is performed before Rel(1) and v is performed after Rel(1). OR u is performed before Sync(1) and v is performed after Sync(1). Definition 6
Theorem 3 The Hierarchical Group Consistency model is not equivalent to the Release Consistency model nor to the Entry Consistency model Proof : HGC introduces a new synchronization operation (the barrier restricted to a synchronization variable). Consequently, CHCC # CRc and CHcc # Czc. Thus, those models are not equivalent.
60 HGC limits coherence management costs of a shared data to some dedicated and explicitly associated processes. Global communications are restricted to processes really needing to have up to date copies of shared data. In this way, the Hierarchical Group Consistency allows to mix high availability of shared data combined with weak replication [4]. 5. IMPLEMENTATION ASPECT We implement the Hierarchical Group Consistency inside the DOSMOS framework (Distributed Shared Objects MemOry System) [5, 12]. DOSMOS is based on 3 kinds of processes : Application Processes (run application code), Memory Processes (manage memory access and consistency inside a group) and Link Processes (manage memory consistency between groups).. The policy of the DOSMOS system is to manage the consistency of an object only within the group of processes that frequently use this object. Thus DOSMOS introduces a hierarchical view of one's application processes by creating groups of processes that frequently share the same object. We define a group as a pair (G, O) where O is a set of objects and G a set of Application Processes sharing those objects. Each group has a group manager (Link Process : LP) responsible for the inter-group communications. Each group management is independent from the others. Concerning read and write operations to shared data, we classify the accesses into two categories : the intra-group accesses (the accessing process belongs to object's group) and the intergroup accesses between two distinct groups. Figure 6 shows an example of object accesses using group consistency. In this example, we have 2 sites A and B. In each site, processes share the same object (Y for site A and X for site B). The objects X and Y do not have the same consistency management : Release Consistency for Y and Lazy Release Consistency for X. In this context, we describe three different actions : First, one process of site A writes Y, then this process sends the new value of Y to the other members of the site A. The second action illustrates an inter-group access. One process of site/3 wants to read the object Y. Since it does not belong to Y's group, it sends its request to its group manager that forwards it to the group manager of site A. This one forwards again the request to the right process in the group that sends the Y value to the requesting process. The last action concerns an acquire of X : since X is managed in Lazy Release Consistency, the acquirer asks the last acquirer for the new value of X. For an intra-group access, we distinguish two cases : 9 If it is the first access for the process to that object, the group manager sends it a copy of this object; 9 Else, the process already has a local copy. Then, depending on the consistency implementation of the group, the process will directly work on this copy or not. For an inter-group access, the accessing process doesn't have a copy and will never have one during the whole execution of the application. It simply asks for the value of this object (for a read) or sends the new value (for a write) to the group manager.
61 SITE A
SITE B
Action 1 /'
Action 3 .. Acion 2
\
.
Object Y Release Consistency
Object X Lazy Release Consistency
Figure 6. Three actions supporting the group consistency : an intra-write in Release Consistency (action 1), an inter-read (action2), and an acquire in Lazy Release Consistency (action 3)
6. CONCLUSION This paper presents an original taxonomy of memory consistencies based on the programmer approach. This classification shows the interest of a new consistency relaxing un-useful memory management costs. The Hierarchical Group Consistency proposes to group together processes that frequently access the same shared data. This model allows a gain in performance without denying DSM programmability. The Group Consistency model implemented in DOSMOS is an original approach for improving DSM Systems. It allows the system to be adapted to the environment under which it performs (type of processors, communication network, application patterns, object size .... ) Depending on the environment, the policy concerning replication of objects will not be the same and consequently the group structure will be different. The hierarchical Group Consistency is well suited for large scale systems and could perfectly fit for multi-cluster and Grid applications. This model fits perfectly the requirements of various kind of networks. Depending on the latency, the bandwidth or other criteria, we can design an accurate group structure in order to reach the best compromise between replication and availability suited for the studied network. REFERENCES
[1]
[2]
S. V. Adve, A. L. Cox, S. Dwarkadas, R. Rajamony, and W. Zwaenepoel. A comparison of entry consistency and lazy release consistency implementations. In Proc. of the 2nd IEEE Symp. on High-Performance Computer Architecture (HPCA-2), pages 26-37, February 1996. S. V. Adve and K. Gharachorloo. Shared memory consistency models: A tutorial. IEEE Computer, 29(12):66-76, December 1996.
62 [3]
[4]
[5]
[6] [7]
[8]
[9] [ 10]
[ 11] [ 12]
Brian N. Bershad, Matthew J. Zekauskas, and W. A. Sawdon. The midway distributed shared memory system. In 38th IEEE International Computer Conference (COMPCON Spring'93), pages 528-537, February 1993. Alice Bonhomme and Laurent Lef6vre. How to combine strong availability with weak replication of objects ? In ECOOP98 : 12th Conference on Object Oriented Programming : Workshop on Mobility and Replication, Bruxelles, Belgium, July 1998. Lionel Brunie, Laurent Lef6vre, and Olivier Reymann. High performance distributed objects for cluster computing. In 1st IEEE International Workshop on Cluster Computing (IWCC '99), pages 229-236, Melbourne, Australia, dec 1999. IEEE Computer Society Press. John B. Carter, John K. Bennet, and Willy Zwaenepoel. Implementation and performance of MUNIN. ACM- Operating Systems Review, 25(5): 152-164, 1991. K. Gharachorloo, D. Lenoski, J. Laudon, P. Gibbons, A. Gupta, and J. Hennessy. Memory consistency and event ordering in scalable shared-memory multiprocessors. In 16th Annual Symposium on Computer Architecture, pages 15-26, May 1989. L. Higham, J. Kawash, and N. Verwaal. Defining and comparing memory consistency models. In Proc. of the 10th Int'l Conf. on Parallel and Distributed Computing Systems (PDCS-97), October 1997. W. Hu, W. Shi, and Z. Tang. A framework of memory consistency models. Journal of Computer Science and Technology, 13(2), March 1998. L. Iftode, J. E Singh, and K. Li. Scope consistency: A bridge between release consistency and entry consistency. In Proc. of the 8th ACMAnnual Symp. on Parallel Algorithms and Architectures (SPAA "96), pages 277-287, June 1996. L. Lamport. How to make a multiprocessor that correctly executes multiprocess programs. IEEE Trans. on Computers, C-28(9):690-691, September 1979. Laurent Lef6vre and Olivier Reymann. Combining low-latency communication protocols with multithreading for high performance dsm systems on clusters. In 8th Euromicro Workshop on Parallel and Distributed Processing, pages 333-340, Rhodes, Greece, Jan 2000. IEEE Computer Society Press.
Parallel Computing: SoftwareTechnology,Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
63
An operational semantics for skeletons* M. AldinuccP and M. Danelutto b ~Institute of Information Science and Technologies (ISTI)- National Research Council (CNR), Via Moruzzi 1, 1-56124 Pisa, Italy bDepartment of Computer Science, University of Pisa, Via Buonarroti 2, 1-56127 Pisa, Italy A major weakness of the current programming systems based on skeletons is that parallel semantics is usually provided in an informal way, thus preventing any formal comparison about program behavior. We describe a schema suitable for the description of both functional and parallel semantics of skeletal languages which is aimed at filling this gap. The proposed schema of semantics represents a handy framework to prove the correctness and validate different rewriting rules. These can be used to transform a skeleton program into a functionally equivalent but possibly faster version. 1. INTRODUCTION Skeletons have been originally conceived by Cole [8] and then used by different research groups to design high-performance structured parallel programming environments [5, 6, 12]. A skeleton may be modeled as an higher-order function taking one or more other skeletons or portions of sequential code as parameters, and modeling a parallel computation out of them. A skeletal program is a composition of skeletons. Skeletons can be provided to the programmer either as language constructs [5, 6, 7] or as libraries [3, 9, 11 ]. The formal description of a parallel, skeletal language involves at least two key issues: 1) the description of the input-output relationship of skeletons (functional semantics); 2) the description of the parallel behavior of skeletons. The functional semantics enables the definition of semantics-preserving program transformations [2, 4, 1, 10]. These transformations can also be driven by some kind of analytical performance models associated with skeletons [ 13], in such a way that only those rewritings leading to efficient implementations of the skeleton code are considered [2, 1]. Almost all frameworks previously cited have a formal functional semantics. But none of them provide a complete and uniform description of the parallel semantics. In this work we present a schema of operational semantics suitable for skeletal languages exploiting both data and stream parallel skeletons. The operational semantics is defined in term of a labeled transition system (LTS). It describes both functional and parallel behavior in a *This work has been partially supported by the Italian MIUR Strategic Project "legge 449/97" year 1999 No. 02.00470.ST97 and year 2000 No. 02.00640.ST97, and by the Italian MIUR FIRB Project GRID.it No. RBNE01KNFE
64
1. seqf<x,'r>g
e e..seqf<7_>e
2. farm A {x, ~-)e e A {x}o(e,x) "" farm A @}o(e,x) 3. pipe A 1 A 2 (X, T}e
g > A2 "~o(g,x) A1 (X)g "' pipe A 1 A 2
(T)g
4. comp A1 A2 (x, ~-)g g, A2 A1 (x)e "" comp A1 A2 (~-)e 5. map f~ A fd (X, 7"}g
g
' fc (Oz A) fd (X)g "" map f~ A fd (r)g
6. d&c ftc fc A fd <x, 7->e
g
d&c ftc fc A fd (Y>e :
7. whileftcA(x,T)e
h "" •
, d&c ftc f~ A fd <x>e "" d&c ftc fc A fd <7-)e a
e
i# (A~ v)
fc (a (dac ftc fc A/d)) fd (Y)e otherwise
e { whileftcA (A(x)e::@}e) > (x)e :: while ftc /k (7-}g
join
(e>• "" (T>e2 _t_ e2 joinc
A (x>e~ e2> (Y)e3 7Ze A<x>h
g2
, g
fd (X>g g> ~ g, "'" g g>
iff (ft~x) otherwise
relabel
E1 A E1
fc ~
L (~) f~ <x>e e (z>e
Ei - - ~ E~
Vil
<~>~- El
..... En
A " r
3 i , j 1 < i , j < n, gi = ~j :=v i = j ~>
<~>.--E~ ..... E"-.
e E2
> A E2
context
.... e
l..n
@
sp
r
Figure 1. Lithium operational semantics, x, y E value; a, 7- E values; u C values U {e}; E E exp; F C exp*; g, g i , . . . E label; 0 :label • value ~ label.
uniform and general way. We use a subset of the Lithium language as a test-bed to describe the methodology [3]. 2. L i t h i u m F O R M A L DEFINITION
Lithium extends the Java language by providing the programmer with both task parallel and data parallel skeletons [3] All the skeletons process a stream (finite or infinite) of input tasks to produce a stream of results. All the skeletons are assumed to be stateless, static variables are forbidden in Lithium code. No concept of "global state" is supported by the implementation, but the ones explicitly programmed by the user 2. The Lithium skeletons are fully nestable.
2Such as RMI servers encapsulating shared data structures.
65 Each skeleton has one or more parameters that model the computations encapsulated in the related parallelism exploitation pattern. Lithium manages two new types in addition to Java types: streams and tuples that are denoted by angled braces and "~ ~" braces respectively: value ::= A Java value values ::= value I value, values
stream ::= ( values ) ] (c) tuplek ::=~ streamb . . . , streamk
A stream represents a sequence (finite or infinite) of values of the same type, whereas the tuple is a parametric type that represents a (finite, ordered) set of streams. Actually, streams appearing in tuples are always singleton streams, i.e. streams holding a single value. The set of skeletons (A) provided by Lithium are defined as follows:
A::= seq f I farm A I pipe A~ A21 comp A~ A21 map fa A f~ I d&c ft~ fa A f~ I while ftc A where sequential Java functions (f, 9) with no index have type Object ---+ Object, and indexed functions (fc, fd, tic) have the following types: fr :tuplek ---+ stream; fd :stream ~ tuplek; ftc :value ---+ boolean. In particular, fr fd represent families of functions that enable the splitting of a stream in k-tuples of singleton streams and vice-versa. Lithium skeletons can be considered as a pre-defined higher-order functions. Intuitively, seq skeleton just integrates sequential Java code chunks within the structured parallel framework; farm and pipe skeletons model embarrassingly parallel and pipeline computations, respectively; comp models pipelines with stages serialized on the same processing element (PE); map models data parallel computations: fd decomposes the input data into a set of possibly overlapping data subsets, the inner skeleton computes a result out of each subset and the f~ function rebuilds a unique result out of these results; d&c models Divide&Conquer computations: input data is divided into subsets by fd and each subset is computed recursively and concurrently until the ft~ condition does not hold true. At this point results of sub-computations are conquered via the fc function, while skeleton model indefinite iteration. A skeleton applied to a stream is called a skeletal expression. Expressions are defined as follows: exp ::= A stream I/x exp I exp ] fr (c~ A) fd stream. The execution of a Lithium program consists in the evaluation of a A stream expression. 3. Lithium O P E R A T I O N A L S E M A N T I C S
We describe Lithium semantics by means of a LTS. We define the label set as the string set augmented with the special label "_L". We rely on labels to distinguish both streams and transitions. Input streams have no label, output stream are 2_ labeled. Labels on streams describe where data items are mapped within the system, while labels on transitions describe where they are computed. The Lithium operational semantics is described in Figure 1. The rules of the Lithium semantics may be grouped in two main categories, corresponding to the two halves of the figure. Rules 1-7 describe how skeletons behave with respect to a stream. These rules spring from the following common schema: Skel params (x, T)el
e~ >,T(x)e 2 .. Skel params (T)e 3
where Skel E [seq, farm, pipe, comp, map, d&c, while], and the infix stream constructor (a)e~ :: (~-)g~ is a non strict operator that sequences skeletal expressions. In general, f is a function
66 appearing in the params list. For each of these rules a couple of twin rules exists (not shown in Figure 1):
o~) Skel params (x, "r) _L> (C>_L"" Skel params (x, "r)z
~) Skel params <x>e~
> ~<x>~ 2
these rules manage the first and the last element of the stream respectively. Each triple of rules manages a stream as follows: the stream is first labeled by a rule of the kind c~). Then the stream is unfolded in a sequence of singleton streams and the nested skeleton is applied to each item in the sequence. During the unfolding, singleton streams are labeled according to the particular rule policy, while the transition is labeled with the label of the stream before the transition (in this case ~1). The last element of the stream is managed by a rule of the kind ~). Eventually resulting skeletal expressions are joined back by means of the :: operator. Let us show how rules 1-7 work with an example. We evaluate farm (seq f) on the input stream (xl,x2, x3). At the very beginning only the 2~ can be applied. It marks the begin of stream by introducing (c)x (empty stream) and labels the input stream with _k. Then rule 2 can be applied: (c)• "" farm (seq f) (x,,x2,x3)•
_L
, (c)• .. seq f (Xl>0 "" farm (seq f) (x2, x3>0
The head of the stream has been separated from the rest and has been re-labeled (from _L to 0) according to the 69(1, x) function. Inner skeleton (seq) has been applied to this singleton stream, while the initial skeleton has been applied to the rest of the stream in a recursive fashion. The re-labeling function (_9 : label x value --, label (namely the oracle) is an external function with respect to the LTS. It would represent the (user-defined) data mapping policy. Let us adopt a two-PEs round-robin policy. An oracle function for this policy would cyclically return a label in a set of two labels. In this case the repeated application of rule 1 proceeds as follows: (c)• :: seq f (X,>o :: seq f (X3) 0
farm (seq f) (x2, x3>o
0
,
1
> (c)• :: seq f (xx)0 :: seq f <x2>1
::
The oracle may have an internal state, and it may implement several policies of label transformation. As an example the oracle might always return a fresh label, or it might make decisions about the label to return on the basis of the x value. As we shall see, in the former case the semantics models the maximally parallel computation of the skeletal expression. Observe that using only rules 1-7 (and their twins) the initial skeleton expression cannot be completely reduced (up to the output stream). Applied in all the possible ways, they lead to an aggregate of expressions (exp) glued by the :: operator. The rest of the work is carried out by the six rules in the bottom half of figure 1. There are two main rules (sp and dp) and four auxiliary rules (context, relabel, joint and join). Such rules define the order of reduction along aggregates of skeletal expressions. Let us describe each rule:
sp (stream parallel) rule describes evaluation order of skeletal expressions along sequences separated by :: operator. The meaning of the rule is the following: suppose that each skeletal expression in the sequence may be rewritten in another skeletal expression with a certain labeled transformation. Then all such skeletal expressions can be transformed
67
farm (pipe (seq fl ) (seq f2)) (Xl, x2, x3, x4, x5, x6, x7 )
(1)
(C)_L "" farm (pipe (seq f l ) (seq f2)) (Xl,X2, X3,X4,X5,X6,X7)_L
(2)
(e)_L "" pipe (seq f l ) ( s e q f 2 ) ( x l ) o "" pipe (seq f l ) ( s e q f2)(X2)l "" pipe (seq f l ) ( s e q f2)(x3)o "" pipe (seq f l ) ( s e q f2)(X4)l "" pipe (seq f l ) ( s e q f2)(x5)o "" pipe (seq f l ) ( s e q f2)(x6}1 "" pipe (seq fz)(seq f2)(xT)o
(3) (e)• -" seq f2 7~02 seq fl (Xl)O "" seq f2 7~12 seq fl (x2)1 "" seq f2 TZ02 seq fl (x3)o "" s e q f 2 72,.12s e q f l (x4)1 "" seq f2 R02 seq fz (x5)o "" seq f2 7Z12 seq fz @6)1 "" seq f2 7~02 seq fl (x7)o
(4)
(e)_L "" s e q f 2 (fl Xl)02 "" s e q f 2 (fl x2)12 "" s e q f 2 7%2 s e q f l (x3)o "" s e q f 2 7~12 s e q f l (x4)1 "" seq f2 R02 seq fl (xs)o "" seq f2 7Z12 seq fl @6)1 "" seq f2 7~02 seq fl (x7)o
(5)
A_ "" 02 "" 02 "" (f2 (fl m4) >12 "" 02 ""
(6) (f2 (fl Xl), f2 (fl x2) f2 (fl x3), f2 (fl x4) f2 (fl x5), f2 (fl x6), f2 (fl x7) )..L
(7)
Figure 2. The semantic of a Lithium program: a complete example.
in parallel, provided that they are adjacent, that the first expression of the sequence is a stream of values, and that all the transformation labels involved are pairwise different.
dp (data parallel) rule describes the evaluation order for the fc (c~ A) fd stream expression. Basically the rule creates a tuple by means of the fd function, then requires the evaluation of all expressions composed by applying A onto all elements of the tuple. All such expression are evaluated in one step by the rule (apply-to-all). Finally, the fc gathers all the elements of the evaluated tuple in a singleton stream. Labels are not an issue for this rule.
relabel rule provides a relabeling facility by evaluating the meta-skeleton 7~e. The rule does nothing but changing the stream label. Pragmatically, the rule imposes to a PE to send the result of a computation to another PE (along with the function to compute).
context rules establishes the evaluation order among nested expressions, in all other cases but the ones treated by dp and relabel. The rule imposes a strict evaluation of nested expressions (i.e. evaluated the arguments first). The rule leaves unchanged both transition and stream labels with respect to the nested expression.
join rule does the housekeeping work, joining back all the singleton streams of values to a single output stream of values, join, does the same work on the first element of the stream. Let us consider a more complex example. Consider the semantics of f a r m (pipe fl f2) evaluated on the input stream (xl,x2, x3, x4, x5, x6, xT). Let us suppose that the oracle function returns a label chosen from a set of two labels. Pragmatically, since farm skeleton
68 represent the replication paradigm and pipe skeleton the pipeline paradigm, the nested form farm (pipe fl f2) basically matches the idea of a multiple pipeline (or a pipeline with multiple independent channels). The oracle function defines the parallelism degree of each paradigm: in our case two pipes, each having two stages. As shown in Figure 2, the initial expression is unfolded by rule 2~ ((1) --~ (2)) then reduced by many applications of rule 2 ((2) ---~* (3)). Afterward the term can be rewritten by rule 3 ((3) --~* (4)). At this point, we can reduce the formula using the sp rule. sp requires a sequence of adjacent expressions that can be reduced with differently labeled transitions. In this case we can find just two different labels (0, 1), thus we apply sp to the leftmost pair of the previuos expressions : (E)_L "" seqf2 ~02 seq fl (Xl)0 "" seqf2 ~12 seq fl ( X 2 ) l "" seqf2 7~02seq fl (x3)0 ..... (e)_t_ :: seq f2(fl Xl)02 :: seqf2 (fl X2)12 :: seqf2 7Z02seq fl (X3)o : : ' ' " Observe that due to the previous reduction ((4) 4 (5) in Figure 2) two new stream labels appear (02 and 12). Now, we can repeat the application of sp rule as in the previous step. This time, it is possible to find four adjacent expression that can be rewritten with (pairwise) different labels (0, 1, 02, 12). Notice that even on a longer stream this oracle function never produces more than four different labels, thus the maximum number of skeletal expressions reduced in one step by sp is four. Repeating the reasoning the we can completely reduce the formula to a sequence of singleton streams ((5) 4 " (6)), that, in turn can be transformed by many application of the join rule ((6) 4 " (7)). Let us analyze the whole reduction process: from the initial expression to the final stream of values we applied three times the sp rule. In the first application of sp we transformed two skeletal expression in one step; in the second application four skeletal expressions have been involved; while in the last one just one expression has been reduced 3. The succession of sp applications in the transition system matches the expected behavior for the double pipeline paradigm. The first and last applications match pipeline start-up and end-up phases. As expected the parallelism exploited is reduced with respect to the steady state phase, that is matched by the second application. A longer input stream would rise the number of sp applications, in fact expanding the steady state phase of modeled system. 4. PARALLELISM AND LABELS The first relevant aspect of the proposed schema is that the functional semantics is independent of the labeling function. Changing the oracle function (i.e. how data and computations are distributed across the system) may change the number of transitions needed to reduce the input stream to the output stream, but it cannot change the output stream itself. The second aspect concerns parallel behavior. It can be completely understood looking at the application of two rules: dp and sp that respectively control data and stream parallelism. We call the evaluation of either dp or sp rule apar-step, dp rule acts as an apply-to-all on a tuple of data items. Such data items are generated partitioning a single task of the stream by means of an user-defined function. The parallelism comes from the reduction of all elements in a tuple (actually singleton streams) in a single par-step. A single instance of the sp rule enable the parallel evolution of adjacent terms with different labels (i.e. computations running on distinct PEs). The converse implication also holds: many transitions of adjacent terms with different 3Second and third sp application are not shown in the example.
69 labels might be reduced with a single sp application. However, notice that sp rule may be applied in many different ways even to the same term. In particular the number of expressions reduced in a single par-step may vary from 1 to n (i.e. the maximum number of adjacent terms exploiting different labels). These different applications lead to both different proofs and parallel (but functional) semantics for the same term. This degree of freedom enables the language designer to define a (functionally confluent) family of semantics for the same language coveting different aspects of the language. For example, it is possible to define the semantics exploiting the maximum available parallelism or the one that never uses more than k PEs. At any time, the effective parallelism degree in the evaluation of a given term in a par-step can be counted by inducing in a structural way on the proof of the term. Parallelism degree in the conclusion of a rule is the sum of parallelism degrees of the transitions appearing in the premise (assuming one the parallelism degree in rules 1-7). The parallelism degree counting may be easily formalized in the LTS by using an additional label on transitions. The LTS proof machinery subsumes a generic skeleton implementation: the input of skeleton program comes from a single entity (e.g. channel, cable, etc.) and at discrete time steps. To exploit parallelism on different tasks of the stream, tasks are spread out in PEs following a given discipline. Stream labels trace tasks in their journey, and sp establishes that tasks with the same label, i.e. on the same PE, cannot be computed in parallel. Labels are assigned by the oracle function by rewriting a label into another one using its own internal policy. The oracle abstracts the mapping of data onto PEs, and it can be viewed as a parameter of the transition system used to model several policies in data mapping. As an example a farm may take items from the stream and spread them in a round-robin fashion to a pool of workers. Alternatively, the farm may manage the pool of workers by divination, always mapping a data task to a free worker (such kind of policy may be used to establish an upper bound in the parallelism exploited). The label set effectively used in a computation depends on the oracle function. It can be a statically fixed set or it can change cardinality during the evaluation. On this ground, a large class of implementations may be modeled. Labels on transformations are derived from label on streams. Quite intuitively, a processing element must known a data item to elaborate it. The relabeling mechanism enables to describe a data item re-mapping. In a par-step, transformation labels point out which are the PEs currently computing the task. 5. CONCLUSION We propose an operational semantics schema that can be used to describe both functional and parallel behavior of skeletal programs in a uniform way. This schema is basically a LTS, which is parametric with respect to an oracle function. The oracle function provides the LTS with a configurable label generator that establishes a mapping between data and computation and system resources. We use the Lithium - a skeletal language exploiting both task and data parallelism- as a test-bed for the semantics schema. We show how the semantics (built according to the schema) enables the analysis of several facets of Lithium programs such as: the description of functional semantics, the comparison in performance and resource usage between functionally equivalent programs, the analysis of maximum parallelism achievable with infinite or finite resources. To our knowledge, there is no other work discussing a semantics with the same features in parallel skeleton language framework (and, in general, in the structured parallel programming world as well).
70 REFERENCES
[ 1]
[2]
[3] [4]
[5]
[6] [7] [8] [9]
[10]
[ 11 ] [12] [13]
M. Aldinucci. Automatic program transformation: The Meta tool for skeleton-based languages. In S. Gorlatch and C. Lengauer, editors, Constructive Methods for Parallel Programming, Advances in Computation: Theory and Practice, chapter 5, pages 59-78. Nova Science Publishers, NY, USA, 2002. M. Aldinucci and M. Danelutto. Stream parallel skeleton optimization. In Proc. of the 11th lASTED lntl. Conference on Parallel and Distributed Computing and Systems (PDCS'99), pages 955-962, Cambridge, Massachusetts, USA, November 1999. IASTED/ACTA press. M. Aldinucci, M. Danelutto, and P. Teti. An advanced environment supporting structured parallel programming in Java. Future Generation Computer Systems, 19(5):611-626, 2003. M. Aldinucci, S. Gorlatch, C. Lengauer, and S. Pelagatti. Towards parallel programming by transformation: The FAN skeleton framework. Parallel Algorithms and Applications, 16(2-3):87-122, 2001. P. Au, J. Darlington, M. Ghanem, Y. Guo, H.W. To, and J. Yang. Co-ordinating heterogeneous parallel computation. In L. Bouge, P. Fraigniaud, A. Mignotte, and Y. Robert, editors, Proc. of Euro-Par 1996, pages 601-614. Springer-Verlag, 1996. B. Bacci, M. Danelutto, S. Pelagatti, and M. Vanneschi. SklE: a heterogeneous environment for HPC applications. Parallel Computing, 25(13-14): 1827-1852, December 1999. G.H. Botorog and H. Kuchen. Efficient high-level parallel programming. Theoretical Computer Science, 196(1-2):71-107, April 1998. M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computations. Research Monographs in Parallel and Distributed Computing. Pitman, 1989. M. Danelutto and M. Stigliani. SKElib: parallel programming with skeletons in C. In A. Bode, T. Ludwing, W. Karl, and R. Wismiiller, editors, Proc. of Euro-Par 2000, number 1900 in LNCS, pages 1175-1184. Springer-Verlag, September 2000. S. Gorlatch, C. Lengauer, and C. Wedler. Optimization rules for programming with collective operations. In Proc. of the 13th Intl. Parallel Processing Symposium & l Oth Symposium on Parallel and Distributed Processing (IPPS/SPDP'99), IEEE Computer Society Press, pages 492-499, 1999. H. Kuchen. A skeleton library. In B. Monien and R. Feldmann, editors, Proc. of Euro-Par 2002, number 2400 in LNCS, pages 620-629. Springer-Verlag, 2002. J. Srrot and D. Ginhac. Skeletons for parallel image processing: an overview of the SKIPPER project. Parallel Computing, 28(12):1685-1708, December 2002. D. B. Skillicom and W. Cai. A cost calculus for parallel functional programming. Journal of Parallel and Distributed Computing, 28:65-83, 1995.
Parallel Computing: SoftwareTechnology,Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
71
A Programming Model for Tree Structured Parallel and Distributed Algorithms and its Implementation in a Java Environment* H. Moritsch a a School of Business, Economics, and Computer Science, University of Vienna, Brfinner Strage 72, A-1210 Vienna, Austria [email protected] We describe the Distributed Active Tree, a programming model which allows for an objectoriented, data parallel formulation of a whole class of parallel and distributed tree structured algorithms. It provides collective communication with varying synchronization conditions and shared address space through Accessible objects, representing the nodes accessed within a communication operation. We developed a Java implementation and employed it in the parallelization of nested Benders decomposition, a solution technique for large stochastic programming problems such as multistage decision problems. The optimization algorithm is part of a high performance decision support tool for asset and liability management, the Aurora Financial Management System. We discuss the architecture and coordination layer of our implementation and present experimental results obtained on a Beowulf SMP-cluster and on a network of workstations, including a comparison of a synchronous and an asynchronous version of the algorithm. 1. INTRODUCTION Tree structured problems occur in various applications areas, for example in portfolio management and reservoir optimization. In fact, they represent an important class of the problems dealt with in finance. The Aurora Financial Management System under development at the University of Vienna is a decision support tool for asset and liability management employing parallel solvers for large multistage stochastic optimization problems [11]. Using decomposition techniques, the whole optimization problem is represented as a set of small problems defined at the nodes of a scenario tree. By iteratively solving their local problems and updating them using information from neighbor nodes, all nodes contribute to the overall solution. The decomposition method has been implemented in parallel as a distributed active tree structure. The programming model behind allows to formulate a wide class of tree structured algorithms at a high level with transparent remote accesses, communication and synchronization. The remainder of this paper is organized as follows. Section 2 describes the distributed active tree model, and section 3 discusses the implementation as a Java class library on top of Remote Method Invocation. Section 4 presents an application in the field of stochastic optimization, *The work described in this paper was supported by the Austrian Science Fund as part of the Special Research Program No. F011 AURORA "Advanced Models, Applications and Software Systems for High Performance Computing".
72 a tree structured decomposition algorithm, and reports on initial experiments conducted on a network of Sun workstations and on a Beowulf SMP-cluster. Section 5 concludes the paper. 2. T H E DISTRIBUTED ACTIVE TREE M O D E L The Distributed Active Tree (DAT) is a structure which executes a parallel program. The program is defined as a set of cooperating algorithms associated with nodes of a the tree. The DAT is a distributed data structure containing active tree node objects, i.e. nodes have their own threads of control. Whereas in principle they could execute different programs, in this model they execute the same (eventually parameterized with particular properties of the node). As they operate on different data, the DAT programming model is a Single Program, Multiple Data (SPMD) model. The set of tree nodes is distributed onto a set of compute nodes (physical processors), according to a specified mapping defining the distribution of the tree data. The nodes altogether perform a parallel algorithm, thus the whole tree can be as a distributed processing unit (an abstract machine). Tree nodes can be mapped onto compute nodes individually or at the level of complete subtrees. With emphasis on iterative tree structured algorithms, the node activity is modeled as a while-loop. It can contain additional nested loops. Conceptually, all nodes are simultaneously active and exchange data with other nodes. Due to synchronization requirements, a node can be in a wait state. For the coordination among tree nodes, the model provides the abstraction Accessible. It represents the set of tree nodes (element nodes) which are involved in a communication operation, e.g. the successors, the predecessor, or the union of these. Data accesses to other nodes are performed through calling methods of the respective Accessible objects. Data transfered to another node is buffered at the target node. The target.put(data) method copies the object data into buffers at the element nodes of the Accessible target. The source.get(allOrAny) method, called at a node, retrieves data (put by the element nodes of the Accessible source) from the buffer and copies it into local data structures of the node. The get method either operates, depending on the value of its parameter, in a mode in which it waits until data has arrived from all element nodes, or in a mode in which it waits for data from at least one node; this allows for diverse synchronization constraints. Accessing a set of nodes in a single operation provides a kind of collective communication. The communication primitives are transparent with respect to the distribution of the tree: nodes residing on the same compute node (local nodes) are addressed in the same way as nodes residing at a different compute node (remote nodes); the programmer maintains a shared memory view of the whole tree. For the execution of many node activities on one compute node, different scheduling strategies are possible. If every tree node is associated with a thread of the underlying runtime environment, the scheduling is just delegated. Following a different strategy, one single thread cyclically executes one iteration on all tree nodes, then the next iteration, and so on. Combinations of these strategies are possible, e.g. there are m threads at a compute node, each of which is responsible for n tree nodes. 3. I M P L E M E N T A T I O N IN JAVA We developed an implemention in Java because of Java's portability and other features such as the object-oriented programming model, robustness, automatic memory management and support for multithreading and network programming. In particular, we assume the financial
73 application to be used in a heterogenous computing environment. The Java implementation also serves as a basis for a grid-enabled version. At the level of the Java language, the DAT model is defined as a set of interfaces, organized in several packages: a u r o r a , d a t (specification of the tree structure), a u r o r a , c-tat, a l g (specification of the algorithm), a u r o r a . d a t , c o o r d (coordination layer implementation), a u r o r a , d a t . d i s t (tree distribution specification), and a u r o r a , d a t . s c h e d (loop scheduling method); the Java classes implementing these interfaces altogether build a concrete DAT. Every tree node is an instance of a class which implements the A l g o r i t h m N o d e interface and overrides the methods i t e r a t i o n and c o n d i t i o n of the L o o p A c t i v i t y interface, thus specifying the body and termination condition of the main loop. Associated to every tree node there is a coordination object (an instance of a class implementing the Coordination interface), which maintains a buffer for incoming data and provides Accessible objects to the tree node. The scheduling of loops is specified in an implementation of the L o o p S c h e d u l e r interface which operationally defines a mapping from the set of loops to be executed on a compute node onto a set of Java threads. An instance of this class creates the threads, assigns loop iterations to them, and starts them. The tree, including its distribution, is specified in an implementation of the D i s t r i b u t e d T r e e S p e c i f i c a t i o n interface. Using Java code, arbitrary mappings can be defined. At runtime, the DAT comprises, at each compute node, a set of A l g o r i t h m N o d e objects with their corresponding C o o r d i n a t i o n objects, a L o o p S c h e d u l e r - , and a DistributedTreeSpecification object. In the m a i n function at a compute node, these objects are created and combined within a C o m p u t e N o d e object: new C o m p u t e N o d e (number, AlgorithmNode-class-name, Coordinationclass-name, new DistributedTreeSpecification-class (..) ) . r u n (new LoopSchedulerclass (..) ). The implementation of the coordination forms a separate layer in the DAT below the algorithm layer, dealing with all intra- and inter-processor communication and-synchronization. It will employ underlying mechanisms such as Sockets, RMI, CORBA, and even MPI (via the Java Native Interface). Via the C o o r d i n a t i o n interface, different coordination implementations can be used by an algorithm. We developed an implementation on top of Java Remote Method Invocation (RMI). The class R D C (RMI-based DAT Coordination) has a s y n c h r o n i z e d method add, which appends new data to the input buffer. Within the p u t operation, local nodes are accessed through shared memory via calls of the a d d method at the coordination objects of the target nodes. Remote nodes are accessed through calling (via RMI) a remote method of the target C o m p u t e N o d e object, which then locally calls the a d d method at the target coordination objects. We describe the synchronization of tree nodes and its implementation as a collection of state dependent actions using Java's built-in synchronization features [7]. Every tree node performs its activity in a separate thread. With regard to coordination, the logical state space of a node (more precisely, of the coordination object) consists of the following states. In the Active state, the node is performing computations of the node algorithm. When it starts waiting for data from other nodes, it enters the Wait state. When new data has arrived, it enters the Transfer
74
add ~ [
_~rI
i
add ( ] [element][" TransferJ( :"~ [element&&
Active a)
la~176 >
public class RDC implements Coordination { 9. . public class RDA implements Accessible { 9. . p u b l i c v o i d get ( b o o l e a n a l l O r A n y ) { s t a t e C h a n g e d (this, a l l O r A n y ) ; s y n c h r o n i z e d (RDC. this) { if (state == WAIT) try { RDC.this.wait(); } c a t c h ...
} do {
t r a n s f e r (this) ; s t a t e C h a n g e d (this, false) ; } w h i l e (state == T R A N S F E R ) ; }//get } }
b)
Figure 1. a) Logical states of the coordination object b) Synchronization within the get: operation
state. In this state, the buffer contents are copied into local data structures. States are left either when the activity associated with a state is finished, or when the a d d method has been called, i.e. another node has added data to the buffer. The statechart diagram in Figure 1 shows the additional substates ActiveNew and TransferNew, which are entered when new data has arrived during the state activity. This allows for optimizations: in the case of Active, Wait can be skipped, and in the case of Transfer, the new data will just be considered during the current g e t operation. In addition, two guard conditions control the state transitions. Incoming data is regarded relevant only if the node, which has put it into the buffer, is an element of the Accessible performing the g e t operation. In the case of a l 10rAny== t r u e , data is regarded available only if arrived from all element nodes of the Accessible. The transition diagram is implemented in the synchronized method R D C . s t a t e C h a n g e d , which writes the instance variable s t a t e ; it is called in the g e t method, and, from the source node thread, at the end of the a d d method. The code excerpt in Figure 1 shows the implementation of the g e t operation. If the state has changed to Wait, the node thread is caused to wait; s t a t e C h a n g e d calls n o t i f y just in the case that Wait has been left. The s y n c h r o n i z e d method RDC. t r a n s f e r copies the buffered data objects into the local data structures of the A l g o r i t h m N o d e object. These are not visible to the coordination object, because they are in the algorithm layer; the copy operations are methods of the algorithm data objects and, via a conversion mechanism, called by t r a n s f e r .
4. APPLICATION EXAMPLE: STOCHASTIC OPTIMIZATION A multistage decision problem, for example in portfolio optimization, is defined as a set of node problems, each of which is associated with a node of a scenario tree describing the possible developments of the environment. At every node, based on the scenario, a node specific objective function is formulated. The optimization of a combination of all node specific functions is the ultimate goal:
75
class BendersNode implements AlgorithmNode {
[.................. nil. C updateR,Q ~'~
+
I when(newCutornewSolution) l
LP ip;
+
C so've'P ) [infeasible]+
[feasible]
+ ,
master
i
l ........... > . . . . .
compute C feasibi
S o l u t i o n master; Vector Q[], R[]; Coordination coord; Accessible from; boolean synchronous; boolean forward;
- [objVal> master.objVal]3~
d
~SIRV~
(forward ? coord.getPredecessor() : coord.getSuccessors())
- coord.getAll()); // update master, R, Q ... ip = n e w L P ( m a s t e r , Q , R ) ; // create and solve LP ... coord.getSuccessors().put(solution); // send s o l u t i o n ... coord.getPredecessor () .put (cut) ; // send cut
Ise]
from.get(synchronous);
]
a)
// sweep direction
// linear program object
// loop body of the decomposition algorithm public void iteration(int iterationNumber) { S o l u t i o n solution; // s o l u t i o n of LP Cut cut; // cut for master from = (synchronous ?
{cardinality=#slaves]
[else] ~ (~ [termination]
// s o l u t i o n from master // cuts from slaves // coordination object // source of get o p e r a t i o n // is synchronous version
}
b)
Figure 2. a) Node algorithm for nested Benders decomposition b) DAT implementation
Minimize ~
f~ (z~)
hEN"
V(n E iV') {
TnxpredC (n)xnSn + dnxn = bn
where N" denotes the set of nodes in the tree. A node n E N" is associated with a local objective function f~ wrt. decision variables zn. An, bn, and Tn describe the constraints representing the dependency on the decision variables at the predecessor node pred(n), Sn describes constraints local to node n, for example budget constraints. In the case of the root node, Tn = 0. In the nested Benders" decomposition method, every node performs an iterative procedure, acting as a master, slave, or both of them [ 1]. The master solves a linear program, sends the solution to the slaves (the successor nodes), and receives from them additional constraints (cuts) which will improve the master's solution in the next iteration. In the synchronous version of the method, each node receives the solution from its master, builds and solves the local problem, then sends the solution to its slaves, and waits for cuts from every slave. The solution process is a sequence of forward- and backward sweeps over the whole tree. In the asynchronous version, every node waits until it has received data from at least one of slaves or from the master [9]. Figure 2 shows the asynchronous node algorithm and fragments of the algorithm layer code for both versions. Note that the Accessible variable f r o m is used to express the different cases of getting data from neighbor nodes. Figure 3 shows the results of initial experiments with a multistage financial portfolio optimization problem, and tree sizes from 127 nodes to 255 nodes. Every tree node runs as a separate thread; the nested Benders decomposition code calls within a synchroni z e d block, via JNI and C, the E04MFF NAG Fortran LP routine. For the distribution of the tree, the root node and its descendents, up to a specific depth in the tree, are mapped to the "root" processor. The subtrees emanating from the nodes at that depth as a whole are mapped to the remaining
76 10,0 9,0 8,0 7,0 6,0
5,0 ~ 4,0 3,0 2,0 1,0 0,0
8,0 A__ ........... A "-.. "'- .. "'-
(3-(3127 ~ 127 sync a - w 255 ~ 255 sync A-A511 .. -..
~5,0
-,~
o=4,0
[] . . . .
7,0
A.. ..........
6,0
""-...... tSl..
". . . . .
(3- 9 127 ~ 127 sync G-D 255 ~ 255 sync A-A511 ~ 511 sync
~3,0 O- . . . . . . . . . . . . . . . . . . .
2,0
~ ....................
"-... (3- . . . .
~ ....................
1,0 1
3 compute nodes
a)
5
0,0
1
3 compute nodes
5
b)
Figure 3. Execution times on a) network of Sun workstations b) Beowulf cluster compute nodes; they require communication with the root processor only. The initial size of the local constraint matrices were 7 x 6, the values for the depth parameter d of the distribution of nt tree nodes onto n~ compute nodes were chosen (nt]nc, d) = (127]5, 4), ({12713 , 255]5}, 5), and ({255]3,511]5}, 6). The results expose some properties of the algorithm. In the synchronous version, the tree nodes perform on average less iterations than in the asynchronous one (1); this is reflected in shorter execution times on a single compute node. In the asynchronous version, the tree nodes spend less time in waiting for new data (2). When running in parallel, a compute node as a whole is idle, when all of its tree nodes are waiting. Shorter processor idle times result in larger speedups for the asynchronous version. Still, these are rather small due to small node problem sizes, resulting in weak computation/communication ratios; we expect better numbers for larger problems (which currently suffer from numerical instabilities). The number of iterations per tree node increases also with the tree size (3). The additional increase in the number of communication operations is seen as one reason for longer execution times of the asynchronous version with larger trees in parallel. In addition, low level effects such as the thread scheduling overhead of the particular runtime system (JVM and operating system) have to be taken into account. 5. CONCLUSIONS In this paper we have presented the Distributed Active Tree programming model, which allows the application programmer to express tree structured iterative algorithms at a high level in a natural way. The model can be implemented with a variety of protocols, communication mechanisms, including web services and grid technology. We described an implementation on top of Java/RMI, with the sole use of Java's multi threading, communication and synchronization mechanisms. As a case study, a parallel decomposition technique for solving large scale stochastic optimization problems has been implemented, in a synchronous and in an asynchronous version. Within the Aurora project, the nested Benders decomposition has been parallelized using OpusJava [6], and the DAT with the coordination implemented on top of JavaSymphony [5]. An alternative optimization algorithm is described in [2], parallel decomposition techniques
77 and their implementation in [4, 10, 13, 14]. For writing high performance applications in Java, language extensions have been defined. Spar [12] provides extensive support for arrays such as multidimensional arrays, specialized array representations, and tuples. It supports data parallel programming and allows via annotations for an efficient parallelization. Titanium [ 15] is a Java dialect with support for multidimensional arrays. It provides an explicitly parallel SPMD model with a global address space and global synchronization primitives. HPJava [3] adds SPMD programming and collective communication to Java. DAT does not change the syntax or semantics of Java and is specifically targeted at a high level formulation of tree structured iterative algorithms and a highly modular architecture. A classification of language extensions, libraries, and JVM modifications for high performance computing in Java is given in [8]. The implementation of the nested Benders decomposition algorithm is subject to optimization along various dimensions, such as the tree distribution (including dynamic pruning and rebalancing of the tree), the loop scheduling strategy, the underlying communication mechanism, LP solution techniques (warm start of the LP solver), and the mapping of scenario tree nodes to local problems. The Distributed Active Tree is a tool to implement and combine variants of all contributing parts, to study the interplay of effects and to achieve a high performance application. REFERENCES
[1] [2]
[3] [4] [5]
[6]
[7] [8]
[9]
[10]
J.F.Benders. Partitioning procedures for solving mixed-variable programming problems. Numer. Math., 4:238-252, 1962. S.Benkner, L.Halada, M.Lucka. Parallelization Strategies of Three-Stage Stochastic Program Based on the BQ Method. Parallel Numerics'02, Theory and Applications. Ed. R.Trobec, P.Zinterhof, M.Vajtersic, A.Uhl, pp.77-86, October,23-25, 2002. B.Carpenter, G.Zhang, G.Fox, X.Li, Y.Wen. HPJava: data parallel extensions to Java. Concurrency: Practice and Experience 10,11-13: 873-877, 1998. M.A.H.Dempster, R.T.Thompson. Parallelization and aggregation of nested benders decomposition. Annals of Operations Research, 81:163-187, 1998. T.Fahringer, A.Jugravu, B.Di Martino, S.Venticinque, H.Moritsch. On the Evaluation of JavaSymphony for Cluster Applications. In Proceedings of the IEEE International Conference on Cluster Computing (Cluster2002). September 2002, Chicago, Illinois. E.Laure, H.Moritsch. Portable Parallel Portfolio Optimization in the Aurora Financial Management System. In Proceedings of SPIE ITCom 2001 Conference: Commercial Applications for High-Performance Computing. August 2001, Denver, Colorado. D.Lea. Concurrent Programming in Java. Addison-Wesley, Reading, Mass., 1997. M.Lobosco, C.Amorim, O.Loques. Java for high-performance network based computing: a survey. Concurrency and Computation: Practice and Experience (Ed. Geoffrey Fox), John Wiley & Sons, 14:1-31, 2002. H.Moritsch, G.Ch.Pflug, M.Siomak. Asynchronous nested optimization algorithms and their parallel implementation. In Proceedings of the International Software Engineering Symposium. Wuhan, China, March 2001. Soren S.Nielsen and S.A.Zenios. Scalable parallel Benders decomposition for stochastic linear programming. Parallel Computing, 23:1069-1088, 1997.
78 [ 11] G.Ch.Pflug, A.~;wi~tanowski, E.Dockner, H.Moritsch. The AURORA Financial Management System: Model and Parallel Implementation Design. Annals of Operations Research, 99:189-206, 2000. [12] C.van Reeuwijk, F.Kuijlman, H.J.Sips. Spar: a set of extensions to Java for scientific computation. Concurrency and Computation: Practice and Experience 15,3-5: 277-297, 2003. [ 13] A.Ruszczynski. Parallel decomposition of multistage stochastic programming problems. Math. Programming, 58: 201-228, 1993. [ 14] H.Vladimirou, S.A.Zenios. Scalable parallel computations for large-scale stochastic programming. Annals of Operations Research, 90:87-129, 2000. [15] K.Yelick, L.Semenzato, G.Pike, C.Miyamoto, B.Liblit, A.Krishnamurthy, P.Hilfinger, S.Graham, D.Gay, P.Colella, A.Aiken. Titanium: a high-performance Java dialect. Concurrency and Computation: Practice and Experience 10,11-13: 825-836, 1998.
Parallel Computing: Software Technology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
79
A Rewriting Semantics for an Event-Oriented Functional Parallel Language F. Loulergue a aLaboratory of Algorithms, Complexity and Logic, 61, avenue du G6n6ral de Gaulle, 94010 Cr6teil cedex, France This paper presents the design of the core of a parallel programming language called CDS*. It is based on explicitly-distributed concrete data structures and features compositional semantics, higher-order functions and explicitly distributed objects. The denotational semantics is outlined, the (equivalent) operational semantics is presented and a new realization of the latter is given as a rewriting system. 1. I N T R O D U C T I O N Resource-aware programming tools and programs with measurable utilization of network capacity, where control parallelism can be mixed with data parallelism are advocated. In [8, 5] we proposed semantic models for languages whose semantics is functional and whose programs are explicitly parallel. Such languages address the above-stated requirements by expressing data placement, and hence communications explicitly, allowing higher-order functions to take placement strategies and generic computations as arguments, allowing higher-order functions to monitor communications within other functions, and yet avoid the complexity of concurrent programming. We also have introduced the elements of an explicitly-parallel functional language CDS*: denotational semantics, operational semantics and full abstraction result. It is inspired by Berry and Curien's sequential language CDS [1] but uses Brookes and Geva's generalized concrete data structures (gcds) [2], for compatibility with parallel execution. Here we present a detailed rewriting semantics. Unlike Multilisp [6] or CD-Scheme [10], CDS* causes no dynamic process creation in order to facilitate performance prediction. Moreover all the possible events of a CDS* program are declared statically together with their physical processor (or static process) location. We call this feature explicit processes and share it with Caml-Flight [4] and the proposed BSP library standard [7]. CDS* improves on Caml-Flight and BSP by its compositional semantics. User-defined functions define dependencies between (explicitly-located) events and thus prescribe the communications generated by their application. As observed by Berry and Curien, a program of functional type on concrete structures can observe event dependencies inside a functional argument and thus compare different algorithms before applying them. In our context, this means that a second-order function can compare first-order function's load-balancing and communication properties before applying them. This goes one step beyond the proposal of Mirani and Hudak [9] which was to use explicit and programmable schedules but to obtain cost information from system calls. Compared with data-flow programming CDS* is a gener-
80
alization by its inclusion of higher-order functions but it is beyond the scope of this paper to make an exact comparison. Let us simply observe that connecting deterministic programs with streams is a special case of programming with section. The following sections present the language's denotational and operational semantics and then the rewriting semantics. 2. CDS* AND G E N E R A L I Z E D C O N C R E T E DATA STRUCTURES W I T H INDICES. A CDS* program is made of type definitions, followed by term definitions and a request to evaluate a given term. The definition of a type inference system, ofparameterized and polymorphic types is important open problem. As a result our (strong) typing is explicit and monomorphic. A CDS* type is a gods with explicit process indices. A gods is a set of cells with allowed values for each one, seen as a game to be played. Certain cells can be filled at any time and others can only be filled after being enabled by specific finite sets of filling events. A valid gods configuration is called a state and the object of the game is to compute states monotonically: never erasing or changing the value given to a cell. As such, states are generalized traces/streams. A program of functional type r --+ r ' describes a continuous function from states of gods r to states of gods r'. Program syntax is almost standard for a functional language except for one crucial difference: there is no A-binding. Elementary terms are simply an enumeration of finite states, as sets of events. Now unlike standard functional programs which denote abstract functions, a CDS* program denotes a function between concrete domains which can be (concretely) encoded by the state of a special-purpose exponential gods. As a result, elementary terms of type r -+ r' enumerate so-called exponential events which are in fact functional dependencies between (sufficient) input events and (necessary) output events. A generalized concrete data structure is a tuple ((C, _<), V, E, ~-) where C is a countable and partially ordered set of cells, V is a countable set of values, E c_ C • V is a set of events. An event (c, v) is often written cv. The enabling relation ~ is between finite sets of events and cells. It induces aprecedence relation on cells: c << c' iff By, v. y U {cv} ~ c'. The enabling relation must be well-founded. The set of events and the enabling relation are upwards-closed with respect to cell ordering. Namely cv E E, c <_ c' ~ c'v C E and y F- c, c _< c' ~ y F- c'. We assume a finite and statically-available set I of indices. A gcds with indices is a gcds and a total function A from the set of cells to the set of indices, called the location function. A cell c is called initial if it is enabled by the empty set of events (written ~- c). A cell c is filled in a set of events y if 3v. cv E y. Write F(y) for the set of cells filled in y. If y k c, then y is an enabling of c. Let E(y) be the set of cells enabled in y, and call A ( y ) - E ( y ) - F ( y ) the set of cells accessible from y. If y ~ c and Vd E F(y), a(d) - A(c), then the enabling y F- c is local, else it is synchronizing. Let M, M', N denote gcds from now on. A state of M is a set of events x c_ EM which is functional and safe, namely where CVl, cv2 E x =~ Vl = v9 and c c F ( x ) =~ By c_ x. y F-~ c. States must be upwards-closed with respect to cell ordering: cv E x, c <_ c' =~ c'v c x. The generalized concrete domain or simply domain associated with M is the poset (7)(M), C_ ) where 7)(M) is the set of states of gcds M. In the mathematical description we sometimes leave A implicit, although it is understood that the concrete location of every event is that of its cell. Because all types-domains are concrete, the usual type constructors product and arrow have
81 concrete realizations. The product construction is straightforward. Given two gcds with indices M1, M2, we define the gcds with indices M1 x M2 by renaming the cells c of M~ into c.i (i = 1, 2) and by taking the union (using the cell renaming this union will be a disjoint union) of cell ordering, values, access relation, function of location of the two gcds with indices. As expected, and easily verified, the domain of M1 x M2 is isomorphic to the cartesian product of the domains of M1 and M2. The definition of a correct exponential is non-trivial and was in fact a central issue in the works of Berry and Curien and later Brookes and Geva. Let two gcdss with indices M, M' be given and let us call them the source structure and the target structure. The exponential M ---+ M' is (C, <, V, E, F) where C = D~in(M) x Ca4,, and 7)~in(M) is the set of finite states of M ordered by inclusion. (z, c') will be abbreviated to zc'. The finite sets of cells are ordered by inclusion and the cells of the target retain their order: < - C x _
(1).
Clearly (l) defines, for given functional state a and argument state y, a unique set of events. Moreover this set of events can be verified to be functional and safe [2]. This is therefore used as the definition of function application in CDS*. Finally, there is a type constructor specific to gcds the graft, which adds great flexibility in practical applications. Its use is illustrated in [3]. Due to lack of space we do not give a concrete syntax for CDS* and we refer to [5]. This syntax allows to define (for the moment only finite) types (gcds with indices) by enumeration of the cells and enablings and by combining types using product, sum or arrow. The values and functions are named states given by enumeration of their events. Terms are function names, application of one term to another, composition of terms, pairs or couples of terms, fix-point, currying or uncurrying of a term (of correct types of course).
3. OPERATIONAL SEMANTICS We now define the formal operational semantics of CDS* which is fully abstract with respect to the denotational semantics outlined above [5]. The operational semantics manipulates memo terms, terms which memorize parts of their evaluation. A memo term is a CDS* program term some of whose syntactic nodes are tagged with tables. Tables, initially empty, store current approximations of auxiliary states. For example, a function-application node is tagged with a table approximating the (state denoted by) the right sub-term i.e. the argument of the application. Let us recursively define the memo terms. States are memo terms. Couples, pairs, currying and uncurrying of memo terms are memo terms. If T and U are two memo terms, x is a state of the same type as U, then [T.U, x] is a memo term, where x represents the current approximation of U and is called a table. I f T is a memo term and x is a state ofthe same type as T's input, then
82 [ f i x ( T ) , x] is a memo term. If T and T' are memo terms and F is a set of pairs of states (x, x') where x (resp. x') has the same type as T's (resp. T"s) input, then [T' o T, F] is a memo term. x is an approximation of the input of T and x' is an approximation of T ( x ) . CDS* expressions are memo terms without tables x and F. We then define an inference system on a set of requests T?c and answers T!v where T is a memo term, c a cell of the same type and v a value. The rules are presented in figure 1. In the rules AP1, AP2, the table x represents the current approximation of the input U. The union II is defined over memo terms whose pure-term parts are equal. Its result IIcUc has the same pure-term part as its arguments Uc and has as approximation-part, the union of approximation parts of Ur and recursively. In the rules for composition o f T ' o T, we have an input state x and a table F of pairs (z, z') where z is a set of input events and z' a set of output events for T.z (i.e. input events for T'). From x and F, two states y, y' are defined as follows, y is the set of input events found in F and relevant w.r.t x (i.e. element of x). y' is the set of output events found in F and relevant w.r.t. x. More precisely: y = U{zl 3(z, z') E F. z c_ x} and y ' = U{z'l 3(z, z') c F. z C_ x}. By construction, for each pair (z, z') of F, z' represents an approximation of the input to T'. Rules AP2, F2, C2 and C3 are the only rules who introduce sub-requests which will be executed in parallel. These sub-requests are determinate by a set A ( x ) , which is a set of cells which are accessible from x (to follow access conditions). Rules CL1 and P1 are for couples and pairing respectively and there exists also symmetrical rules CL2 and P2 (not shown here).
Figure 1. Operational semantics
T?xc' > Tl!v' T?c E> Tl !v (T, U)?c.1 > (T1, U)!v [CL1] (T, U} ?x(c'.l) [> (T1, U)!v' [P1] T?x(x' c") > Tl !V" T?(x, x')c" > T~!v" [CUR] uncurry(T) ? (x, x')c" [> uncurry(T1) !v" [UNC] curry(T)?x(x'c") > curry(T1)!v" T?xd > T~!v' U ' = u~U~ y = u~{r I r e A(z) & U?c > Ur lAP2] [T.U, x]?c' E> [TI.U, x]!v' [AP1] IT.U, x]?c' E> [T.U', x U y}]?c' T?xc > Tl !V [FU [fix(T),x]?c > [fix(T~),x]!v T1-- UClTca, y-U Cl(Yc~U{ c1Vl })[c1 cA(x)&[fix(T),x]?cl[>[fix(T~ ),Yr ]!Vl [fix(T), x]?c [> [fix(T1), y]?c IF2] T'?y'r ~ T~!v" T1 = U~,T~, y~ = U~,{dv'} I d e AC(y') & T?yc't> Tr [c2] [T'oT, F]?xc"[>[T~oT, F]!v ''[C1] [T'oT, F]?xc"[>[T'oT1,FU(y,y'Uy~l)]?xc '' Yl Uc{CV} I c e AC(y) & x?c c>x!v cv 9 up~(~) x - T a ~>/3 /3 > T fTRN] IT' o T, F]txc" > [T' o T, F (2 { (y U Yl, Y') }]?xc" [C3] x?c > x!v [E] a > ,7 -
"
-
4. T O W A R D S AN A B S T R A C T M A C H I N E : R E W R I T I N G S E M A N T I C S Like natural semantics of the usual functional languages, the operational semantics cannot be used in practice. Thus we investigate a new semantics which is to an implementation. The
83 rewriting semantics is defined by the figures 2, 3 and 4 below representing possible transitions from left columns to right columns. The global data structure is a multi-set of tasks together with a function (set of pairs) mapping syntactic nodes of the term being evaluated to tables. In the figures the columns contain only the interesting tasks for the rules and it must be understood that the remaining tasks of the multi-set are left unchanged. There are two main forms of tasks. The first component of tasks is the name of a cell together with the term whose type contains it. The last component of tasks is always a m o d e : either a value computed for the given cell, a failure marker 2_ to indicate that the cell could not be filled, or a request mode. A simple request mode "?" simply means a general request to fill the cell. An indexed request mode ?~ means a request with the added information that the value sought is also that of cell c. As an example of this last kind of mode, see rule CL2 in the first figure. A last kind of mode ?tr indicates that a s l i c e of state ( A ( x ) in the operational semantics) is currently being filled. A task with this mode has a third, middle, component. This middle component stores a set of cells remaining to fill and the current result of the attempt at filling the slice. This result is either failure on all cells • success at least with one cell !. A special kind of task is also used (in figure 4) for filling the so-called F-tables when evaluating a composition term. The following definition is needed. V(x c) - / v iff cv E x '
.
_J_iffcv ~ x
In general, transition rules come in pairs: a call for evaluation and a return with the result. For example CL 1 with CL2. Rules in figure 2 merely follow the syntactic structure of terms. For example rule ST must be read as: if the multi-set of tasks contains the task (c~, ?) then it will be replaced by the task (c, v) if cv e x and by the task (c, 2-) otherwise.
Figure 2. Rules without table ]Tasks ST (cx,?) CL1 (C.I(T1,T2), 7) CL2 (C.I(T1,T2), ?.CT1 ) U (CT1,//) CR1 (C.2(T1,T2), 9.) CR2 (C.2(T1,T2), ?.CT2 ) ~ (CT2, v ) PL1 (c.I
Tasks' (~x,v(~,~)) (C.I(T1,T.2) , 7CT1 ) U (CT1 , ?) (C.I(TI,T2), Y)
(C.2(T1,T2), ?.CT2 ) W (CT2,9) (C.2(T1,T2), v)
(c.I,?~T~) w(cT~,?) (C.I
(X(X'C")cur(T),U) ((x, x')%nc(T),7~(~'c")T)U (~(~'~")T,7) ((x, x')~,c(T),")
Rules in figure 3 implement the definition of concrete function application. The evaluation
84 proceeds in two phases. First AP1, then AP2, then AP3 or AP4. AP2 asks the global system whether the current approximation of U (namely x) is sufficient to fill the result cell in T . U . If successful then complete with AP3. If not, then AP4 requests to increment the table: AP5, AP6, AP7. If this was a success, AP8 resumes and otherwise AP9 aborts.
Figure 3. Rules for application Tasks AP1 (~,.v, ?) AP2 (C(F.U, 71) AP3 (c'T.~, ?x~'T) | (J~, v) AP4 (~r.u, ?x~)w (xc~,• AP5 AP6 AP7 AP8 AP9
(dr.a, (t u ~,m), ?t~) +u (cu, v) (c~.v,(tUc, m),?~r)W(cU,• (~.~, (~,!), 7~) (c~.u, (~,_L),?~)
Tables
Tasks' (~. v, 71)
Tables'
7-U(T.U,x)
(C~T.u,?XC~)W(XC~T,?) (~'~.~, v)
7-tO(T.U,x)
(~.u, .%) ~-u (T.u,x) 7-t2 (T.U,x)
(C~F.U,(Uc EA(x)C,-1-),?tr) WceA(x)(CU, 7)
(C~r.v,(t, !), ?~) 7-U(T.U,x) (c~,.v,(t,m),7~r) (~.~, 71) (~.u, •
7-u (T.U,~) T u (T.g,x u (cu, v))
7-U(T.U, xU(cu,_L))
Rules are needed for the fix-point. There are similar to those of figure 3, the difference is that the table x is an approximation of the fix-point (and that T . U must be replaced by f i x ( T ) everywhere). Rules in figure 4 implements the composition.The evaluation proceeds in three phases. First, rule C2 requests to obtain the approximation y' of T.x from the F-Table. Then rule C3 uses this approximation to get the value of the cell y'c" in term T'. If the value can not be obtained, then C3 requests to increment the table. C6-C 11 aim at increment the approximation of the output of T. In case of success, C 10 resumes, in case of failure, C 11 requests to increment the input part of T with rules C 12 to C 17. If the incrementation successes C 16 requests a new approximation of the output part, otherwise the whole request fails with rule C 17.
5. CONCLUSIONS
CDS* integrates concrete higher-order functions with parallel execution and this combination of features opens the possibility of improving the state of the art in declarative parallel programming. Open questions are of two kinds. Programming convenience would greatly improve if a polymorphic type system can be designed. Efficient implementation remains to be shown but the rewriting system presented here constitutes the first draft of a distributed abstract machine, all of its actions being even-driven and having therefore explicit locations. We believe that transparency with respect to communications is the key to effective parallel programming. Our main goal is to prove this kind of transparency compatible with compositional semantics, higher-order functions, and reflexivity.
85 Figure 4. Rules for composition and F-tables
1
1 ~a~k~ C1 C2 C3 C4 C5 C6 C7
] Tbl
"91)
(XC~,oT,9.)
(XCT, oT,
( X C ~ t o T , ?fzl) ~ ( ( T ' o T , x ) ,
(Xr
?tl) ~ ((T , o T, x),r X C TIt, o T , ?q,t tt , V) "~ ott ~T' ) ~ ~(Y t CT, X C T , o T , ?qtlt~ll If s[ l CT,,
C14 C15 C16 C17
(Cx,V)~(xc~,oT,(tUc,(y, yt),m),?tr3) (Cx,-[-)W{-CTrot , (tUc,(y, yt),m),9, tr3) (~'oT' (~' !)' "~) (X~'oT' (~' (Y'Y')' • "~'~) ((T, zUzl),FU(z,
zt),(y,y'))
,,
~,, ~,,
,~,,
(x%,oT, .~ ~T,) w (y ~T"
(y,B'))
C9 C10 Cll C12 C13
FTBL1
Tbl'__i
(x~" ?~) CT,oT,
(XC~,oT,72) (ZC~r,oT, %2) | ((T' oT, x),e,(y,y')) (-~" :1; T r o T ~ (tu~' , (y,v'),~) , ?~2) (~d~, ~') (xC~,oT, (t u d, (~, r m), ?~) w (ST, • ('~'oT' (~' (Y'Y')' !)' ?*~) (X~(~,oT,(~, (Y,Y'), • ?~) (~'oT' 73) (XC~,oT,?.t3) ~J((T' o T, x),G (y,y'))
C8
Tasks'
.
75
F, e) ?)
(xC~,oT, ~) it (XCT, oT,
?2)
(XC~,oT,?t2)~ ((T' T,x),F,e) (J~,oT, (u~,d, (y,y'),• ?~) o
~J
U c t E A ( y , ) ( Y C t T , ?)
(~,oT, (t, (y, y'), !), 7 ~ )
~(v')
(xc"T, o T ~ (t, (y,y') m), 7~2)
7-2(•
(XC~,oT,?I)
=r~ .1_
(XC~.,oT,73) (xc~,oT,?t3)t~((T' oT, x),F,Q (xc" T r o T ~ (u~c,(v,y') z),?t~3) u U~eA(~)(c~,?) (x~,oT, (t, (~, y'), !), 7~T3) (xd~,or, (t, (y, y'), .~), v~)
7-3(,) ~(J_)
(xd~,o~,• ((T, zUzl)'F'(YUz'ytUz))
1
FTBL2 ((T,z),Fu(zuzl,z'),(y,y')) ((T,z),f,(y,y')) [ whereT1 = T U ( T ' oT, F),T2(m) = "-l-U(y,y'U(c',m))andTa(m) = T U ( y U ( c , m ) , y ' ) .
[ [
REFERENCES
[1] [2]
[3]
[4]
[5] [6]
[7]
G. Berry and R-L. Curien. Theory and practice of sequential algorithms: the kernel of the applicative language CDS. In J. C. Reynolds and M. Nivat, editors, Algebraic Semantics, pages 35-84. Cambridge University Press, 1985. S. Brookes and S. Geva. Continuous functions and parallel algorithms on concrete data structures. In S. Brookes, M. Main, A. Melton, M. Mislove, and D. Schmidt, editors, MFPS'91, number 598 in LNCS, pages 326-349. Springer, 1991. R-L. Curien. Categorical Combinators, Sequential Algorithms and Functional Programming. Birkh~iuser, Boston, second edition, 1993. C. Foisy and E. Chailloux. Carol Flight: a portable SPMD extension of ML for distributed memory multiprocessors. In A. W. B6hm and J. T. Feo, editors, Workshop on High Performance Functional Computing, Denver, Colorado, April 1995. Lawrence Livermore National Laboratory, USA. G. Hains, F. Loulergue, and J. Mullins. Concrete data structures and functional parallel programming. Theoretical Computer Science, 258( 1-2):233-267, 2001. Jr. Halstead. Multilisp: A Language for Concurrent Symbolic Computation. A C M Transactions on Programming Languages and Systems, 7(4):501-538, october 1985. J.M.D. Hill, W.F. McColl, and al. BSPlib: The BSP Programming Library. Parallel Computing, 24:1947-1980, 1998.
86 [8]
F. Loulergue and G. Hains. Parallel functional programming with explicit processes: Beyond SPMD. In C. Lengauer, M. Griebl, and S. Gorlatch, editors, Euro-Par "97, number 1300 in LNCS, pages 530-537. Springer, 1997. [9] R. Mirani and P. Hudak. First-class schedules and virtual maps. In 7th Annual Conference on Functional Programming Languages and Computer Architecture (FPCA "95), La Jolla, California, 1995. ACM. [ 10] C. Queinnec. A concurrent and distributed extension of Scheme. In D. l~tiemble and J.-C. Syre, editors, PARLE'92, number 605 in Lecture Notes in Computer Science, Paris, 1992. Springer.
Parallel Computing: SoftwareTechnology,Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All fights reserved.
87
R M I - l i k e c o m m u n i c a t i o n for m i g r a t a b l e s o f t w a r e c o m p o n e n t s in H A R N E S S M. Migliardi ~ and R. Podesta ~ ~DIST-University of Genoa, Via Opera Pia 13, 16145, ITALY {om, ropode }@dist.unige.it Tel. +39 010 3536549 Fax +39 010 353 2154 1. INTRODUCTION The foundation of the HARNESS [ 1] experimental metacomputing system is the concept of dynamic reconfigurability of networked computing frameworks. This reconfigurability goes beyond simply involving the computers and networks that comprise the virtual machine, and includes the capabilities of the VM itself. These characteristics may be modified under user control by accessing a set of Java based kernels and via an object oriented [2] "plug-in" mechanism that is the central feature of the system. While the usefulness of being capable to enroll and release computational resources at runtime has been already proved in the past by metacomputing systems such as PVM [3], the plug-in paradigm appeared in the past only in sequential environments and applications such as Netscape Navigator and Adobe Photoshop. On the contrary, HARNESS extends this concept to a parallel, distributed environment providing to users the capability to control, monitor and manage the pluggability of a distributed virtual machine as a whole. Recently, the scope of distributed programming has enlarged from clusters of computers connected to a LAN toward dynamically evolving sets of resources interconnected through the Internet. In this new scenario the capability of software components to migrate from one node of the virtual machine to another has become paramount. As a matter of fact, while in an environment where all the nodes belong to and are managed by a single entity, the disappearance of a node is relatively infrequent, in an unstructured environment such as the Internet the nodes may exhibit a very short staying time with frequent join and barely announced leave events. For this reason, it is necessary to build applications as sets of interacting migratable components capable of surviving the disruption caused by unpredictable component location. n a Java based environment, such as it is HARNESS, one of the most convenient mechanism to allow components to interact is RMI [4]. RMI is a fully object oriented remote procedure call mechanism, however, it does not support component migration as it ties the reference of the remote object to its Internet address. This fact makes extremely difficult to use RMI in an environment where components need to often migrate from one node to another. In this paper we describe a user controlled task migration mechanism and an RMI-like method invocation mechanism that we developed for a migratable components support suite in the HARNESS framework. The paper is structured as follows: in section 2 we give an overview of the HARNESS system architecture; in section 3 we describe how HARNESS supports user controlled migration of multi-threaded tasks; in section 4 we describe how we implemented a migration
88 independent RMI-like method invocation; finally, in section 5, we provide some concluding remarks. 2. OVERVIEW OF THE HARNESS METACOMPUTING SYSTEM In figure 1 we show the abstract layers of the HARNESS system architecture.
e.
~ J 4~
Level 3
re~ource~emdledilltheDVM
SclvR;cs are p l u ~ ] --in m form the
H~=rog~eous Compulaliotml
'
lhe D V M
Level
Figure 1. Abstract model of a Harness virtual machine The fundamental abstraction in the HARNESS metacomputing framework is the Distributed Virtual Machine (DVM) (see figure 1, level 1). Any DVM is associated with a symbolic name that is unique in the HARNESS name space, but has no physical entities connected to it. The HARNESS name space is a mapping of the symbolic DVM name into an access point that Heterogeneous Computational Resources can contact to enroll into the DVM. There are at present two implementation of the HARNESS name space, the first is based on multicast and the hashing of the DVM name onto port numbers and the second is based on a HARNESS name server. Heterogeneous computational resources may enroll into a DVM (see figure 1, level 2) at any time. At this level the DVM is only able to provide two services: loading of plug-ins and status tracking and management. To satisfy the needs of users and applications the heterogeneous computational resources enrolled in a DVM need to load plug-ins (see figure 1, level 3). A plug-in is a software component implementing a specific service. By loading plug-ins a DVM can build a service baseline that is consistent to the needs of the application
89 over a large set of heterogeneous resources(see figure 4, level 4). Users may reconfigure the DVM at any time (see figure 1, level 4) both in terms of computational resources enrolled by having them join or leave the DVM and in terms of services available by loading and unloading plug-ins. The main goal of the Harness metacomputing framework is to achieve the capability to enroll heterogeneous computational resources into a DVM and make them capable of delivering a consistent service baseline to users. This goal require the programs building up the framework to be as portable as possible over an as large as possible selection of systems. The availability of services to heterogeneous computational resources derives from two different properties of the framework: the portability of plug-ins and the presence of multiple searchable plug-in repositories. Harness implements these properties mainly leveraging two different features of Java technology. These features are the capability to layer a homogeneous architecture such as the Java Virtual Machine (JVM) [5] over a large set of heterogeneous computational resources, and the capability to customize the mechanism adopted to load and link new objects and libraries. However, the adoption of the Java language as the development platform for the Harness metacomputing framework has given us several other advantages: an OO development environment, a clear and consistent boundary for plug-ins (each plug-in has to appear as a Java class), a natural environment for developing additional services both in a passive, library-like flavor and in an active thread-enabled flavor, a generic methodology to transfer data over the network in a consistent format (Java Object Serialization [6]) and a way to tune the trade-offbetween portability and efficiency for the different components of the framework (JNI [7]). 3. HARNESS USER C O N T R O L L E D MIGRATION OF MULTI-THREADS TASKS Our implementation of task migration support in HARNESS has three main requirements: 9 keeping strict compatibility with the standard Java 2 platform by requiring no changes to the JVM 9 supporting migration of multi-threads tasks 9 generating as limited as possible an additional burden for programmers of mobile tasks. The first constraint is imposed by the fact that one of the main goals of HARNESS is portability over a set as large as possible of heterogeneous systems. Thus it was necessary to renounce to a strong migration model and to limit our design to a weak migration model. In fact, modifying the JVM to support serialization of the JVM stack would have completely nullified the portability advantage obtained adopting the Java language by requiring users to run a special version of the JVM itself. For this reason we implemented a weak migration mechanism that requires applications to adhere to a set of rules in order to be migratable. These rules are: 9 tasks must be modeled as one or more finite state machines (one finite state machine per each running thread); 9 in every instant in which migration is allowed threads must store their complete status in non-automatic, non static variables, i.e. instance variables; 9 a task must extend the Migratable abstract class;
90 9 very thread that can be migrated must be obtained using the method createThread of the Migratable abstract class, not using the constructors of the Thread class. A task that extends the Migratable abstract class needs an execution environment in the physical nodes of the DVM. Such an environment is provided by the implementation of the functional interface named Arena. This implementation is a HARNESS plug-in that can be loaded onto each node belonging the DVM according with general programming model of the framework. One node can host one or more Arenas. However, Arenas are able to managing multiple Migratable, thus having more than one Arena in a single node is redundant unless different Migratable tasks need to strictly separate their execution environments. Modeling a thread, as a finite state machine does not represent a significant burden for the programmers, on the contrary, a finite state machine seems to be a very natural way to model the evolution of a sequential entity such as a thread. A multi-threaded task can be realized simply defining a different finite state machine for each type of thread involved in the computation. The restriction of storing the status of tasks in instance variable does not mean that use of automatic variable is forbidden. On the contrary this rule only requires that before acknowledging that it is ready for migration a thread must transfer all the values that are part of its status from automatic variables to instance variables. Any part of the status that is not stored into instance variables will not migrate with the task. Migration can be requested by any thread in a task at any time during the task execution. However, migration will only actually take place as soon as all the migratable threads (i.e. the ones that have been created with the createThread method) have acknowledged their availability by calling the canlContinue method. Besides, entities outside of the task can request the migration of the migratable task, in fact the requestMigration method is public. In order to start a migratable task into a HARNESS DVM, it is necessary to have at least one Arena plug-in available in the DVM. More in detail, each node of the DVM that wishes to be capable of hosting migratable tasks needs to load ate least one instance of the Arena plug-in. If a node is used mainly for providing other services and does not need to host migratable software components it does not need to load the Arena plug-in. The process of injecting injecting a multi-thread task into an Arena plug-in is composed of the following steps: 1. get the name of the DVM 2. get the reference to any Arena plug-in in the DVM 3. instantiate an object that implements the Migratable abstract class 4. invoke the start method of the Arena whose reference has been obtained passing the Migratable instantiated at step 3 as a parameter. The retum value of the start method is an ID that is unique at HARNESS DVM level. This ID allows tracking the task regardless of its migrations from one node of the DVM to another. The Arena plug-ins and the Migratable class leverages the HARNESS event system to keep track of the node where the migratable task is currently hosted. The current implementation of the HARNESS event system is based on a star topology that poses some performance limitation. In fact, the system is not capable of delivering more that five hundred events per second in a DVM
91 with twenty active nodes. However, the plug-in based architecture of the HARNESS system allows adopting a new, more performant event service if the need arises. An Arena plug-in advertises the reception of a migratable task while a Migratable advertises his leaving an Arena. These events are distributed by HARNESS as user events with a specific Java type, namely the MigrationEvent type. This binding between event type and Java type makes extremely easy for application interested in the movement of migratable tasks to subscribe this specific kind of event without the need to transmit additional, protocol specific event type tags. Thus applications can keep track of migration event to follow the movements of migratable tasks and pin-point their location if they need to. Arenas process migration event provide a location service for migratable tasks by means of a lookup method. An entity can request any Arena the location of a migratable task identifying it by means of the unique ID value that is generated as soon as a migratable task is injected into any of the Arena plugins loaded onto a HARNESS DVM. The ID value is unique in the whole HARNESS DVM and cannot be changed after it has been set into a migratable task. The lookup method returns the HARNESS handle of the Arena where the migratable task is currently running. Note that a HARNESS handle is a string with a specific format, namely //hostname/DVMname/plug-inName/instance#. This string can be used inside a harness DVM to request services to a plug-in as it identifies unambiguously the HARNESS plug-in it is associated to. The association between the unique task ID and the Arena handle allows the run-time to discover the node of the HARNESS DVM where a migratable task is currently residing. 4. THE RMI-LIKE M E T H O D I N V O C A T I O N M E C H A N I S M F O R MIGRATABLE TASKS IN H A R N E S S
An entity can interact with a migratable task in several different ways. In fact, HARNESS does not impose a single mechanism to interact with migratable task, on the contrary migratable tasks are free to support either direct temporally and spatially synchronous interaction such as RMI, or indirect interaction such as messages posted to a JavaSpace as well as both. owever, Arenas provide an additional mechanism for entities to interact with migratable tasks. This mechanism uses reflections together with the locator service provided by Arenas to allow applications to invoke any of the public methods of a migratable task regardless of their current location. The most important part of this mechanism is the public method invoke provided by the Arena plug-in. This method allows the RMI-like method invocation and operates following these step: 9 it obtains the current task handle using the task ID as a key in the lookup table where they are stored; 9 it matches the Arena handle against the task handle; 9 IF the task and the Arena are residing on the same host - THEN 9 a Migratable object is initialized with the instance of the task obtained by looking up the local database of migratable objects using its ID as a key (this step follows the rule shown in the previous section of the paper). An object Method
92 keeps the method name and the formal parameter that are provided like method invoke input. Finally, through the Java reflection, the requested method is invoked with the actual parameters. - ELSE 9 If the task is residing into another Arena, loaded onto another node of the HARNESS DVM, the invoke method takes the RMI reference of the actual Arena owner of the task and then calls its method invoke operating like shown. In figure 2 we show the core code of the implementation of the invoke method of the class Arenalmpl and its abstract interface Arena.
p~lic
{
interface Arena extends j a v a . r m i . ~ m o ~
p ~ l i c void receive(Object thaW) throws j a v a . ~ i . ~ t e E x ; c e p t i o n ; p ~ l i c void start(Object thaW} throws java~rmi ~ t e E x c e p t i o n ; p ~ l i c Object invoke ( S ~ i n g ID, String m ~ o ~ a m e , Class [] fo~iPar~~rs, Object[ ] actualParame~rs} ~ r o w s java. r ~ . ~ t e ~ a e p t i o n ; p ~ l i c java util. L i n k e d ( s t Is () ~ z o w s java ~rmi. ~moteException ~ p ~ l i c java.~util. L i n k e d ( s t i a ~ 1 () ~ r o w s java. rmi. Remote~ception; p ~ l i c void test (Java.io.Serializ~!e test) throws java. ~ i . ~ m o teExceptio.;
p ~ l i c Object invoke(Stri.g ID, String ~ o ~ ~ , actual) ~ r o w s java, rmi. Remo~Exception
Class[] f o c a l
Object[]
/* (,,~:)* t
if (source. e ~ a l s (myHan~e. toString () ) )
{
~ g r a t a b l e obj - null synchronized (migran ~ s )
}
Obj = (~gra~ble)migrantes, get (ID) ;
/, ( ..... ) */
Class clazz = obj,getClass () ; Method method -- c i a z z . g e t M e t h o d ( m e t h o ~ ~ , / * ( .....)*t retvai ~ method.invokeiobj, actual) ;
formal)
9
/* I,,~) */ else
{
~ena
o w n e r .=. ( A r e n a ) H _ S c o r e , g e t ~ I R e f e r e n c e
t * I-~)*t
t * (~,:,) * t
}
retval
= o~er.
invoke
lID
met,~o~:~,
formal,
( s o u r c e ) ": autual)
;
}
Figure 2. The Arena interface and the core code of the invoke method of Arenalmpl
93 Therefore, an external application can call the invoke method of an Arena passing as parameters the migratable task ID, the requested task method name, the array of formal parameters types and the array of actual parameters. The Arena checks if the requested task is locally present, if it is then it generates a local function call and directly returns the obtained results. If the requested migratable task is not locally available then the Arena checks a lookup table to see if another Arena has currently the requested task. If it finds the requested task it requests the invocation service to the remote Arena. If the requested task is not found on any of the Arena in the DVM then the local Arena assumes that the task is currently migrating and waits for an Arena to signal reception of the requested task. The wait time is limited by a time-out to prevent blocking the caller entity forever. Another important limitation imposed to the waiting time of a caller is the number of redirections performed by Arena plug-ins. In fact, our approach is not capable of preventing a Migratable from leaving an Arena before the actual invocation of the method took place. To avoid a livelock effect where the invocation message keeps following the Migratable movements without actually ever reaching it we introduce a time-to-live concept for all the redirected calls. Following the c l i e n t - server model, we can call the external entity client application and the migratable Task the server application. Such a server has the particularitity of not having a fixed location, as he can migrate from one HARNESS DVM node to another without disrupting its clients. This kind of invocation allows programmers to get over the limitation of standard RMI invocation: the constraint of having a fixed binding between the remote object and its Internet address. With the introduction of the Arena invocation service, the callers are freed from this problem, in fact any component that needs to obtain a service from a migratable component can do it regardless of the callee current location. The task of locating the callee is delegated to the implementation of the Arena abstract class, the ArenaImpl class, by means of its public method invoke. Therefore, the invocation remains transparent to the external application. The package supporting task migration in HARNESS is currently in beta release. A distribution that includes sources of both the framework and a test application is available. This distribution demonstrates the location-unaware invokation of a public method of a Migratable that store the history of its random migrations.
5. CONCLUDING REMARKS RMI is a very convenient mechanism for leveraging services provided by software components independently from their location. However, its fight binding of services? end-points to IP addresses prevents programmers from using it in presence of migration mechanisms. In this paper we have described how the HARNESS framework supports a mechanism for user controlled migration of multithread tasks and an RMI-like mechanism to obtain services from migratable software components. The modular, object oriented nature of HARNESS system allows on demand mix and match of components or complete applications adopting programming models that are different from its native one. This process is supported by means of compatibility suites that can be plugged into the system if the need for such a programming model arises. In this paper we described the main features of our migratable software components compatibility suite and, more in detail, its capability to support weak migration for multi-threaded
94 tasks and a location-free RMI-like communication mechanism. In order to qualify as a migratable component a task must follow a simple set of rules that pose only a limited burden on the programmer?s shoulders. Such a software component is allowed to migrate freely among a HARNESS DVM nodes and is also allowed to have public methods that can be invoked from extemal entities in an RMI-like fashion regardless of the callee current location. Our implementation of this mechanism allows the programmer to call the methods belonging to a migratable component with no knowledge whatsoever about its current location; as a matter of fact, the migratable component can reside on any HARNESS DVM node. Our implementation achieved following results: /) it overcame the inherent limitation of RMI that binds tightly a service end-points with IP addresses, ii) it allowed programmers to adopt a convenient, RMI-like communication mechanism between migratable components. The comerstones of our implementation are the capabilities of the Arena HARNESS plugin. This plug-ins keeps track of the state and location of the migratable components leveraging the event services provided by the HARNESS system. The current, star based implementation of the HARNESS event service may pose scalability problems for more than five hundred events per second. However, the plug-in based architecture of HARNESS allows substituting the current implementation with a more performant one if the need arises. REFERENCES [1]
[2] [3]
[4]
[5] [6]
[7]
M. Migliardi, V. Sunderam, A. Geist and J. Dongarra, Dynamic Reconfiguaration and Virtual Machine Management in the HARNESS Metacomputing Framework, proc. Of ISCOPE98, Santa Fe (NM), December 8-11, 1998. G. Booch, Object Oriented Analysis and Design with Applications, Second Edition, Rational, Santa Clara, Califomia, 1994. A. Geist, A. Beguelin, J. Dongarra, W. Jiang, B. Mancheck and V. Sunderam, PVM: Parallel Virtual Machine a User?s Guide and Tutorial for Networked Parallel Computing, MIT Press, Cambridge, MA, 1994. Sun Mycrosystems, Remote Method Invocation Specification, available on line at http://java.sun.com/products/jdld1.2/docs/guide/rmi/index.html, July 1998. T. Lindholm and F. Yellin, the Java Virtual Machine, Addison Wesley, 1997. Sun Mycrosystems, Object Serialization Specification, available on line at http://j ava. sun. com/products/j dlJ 1.2/doc s/guide/serialization/index.html. S. Liang, The Java Native Interface: Programming Guide and Reference, Addison Wesley, 1998.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
95
Semantics of a Functional BSP Language with Imperative Features F. Gava a and F. Loulergue a ~Laboratory of Algorithms, Complexity and Logic, 61, avenue du G6n6ral de Gaulle, 94010 Cr6teil cedex, France The Bulk Synchronous Parallel ML (BSML) is a functional language for Bulk Synchronous Parallel (BSP) programming, on top of the sequential functional language Objective Caml. It is based on an extension of the A-calculus by parallel operations on a parallel data structure named parallel vector, which is given by intention. The Objective Caml language is a functional language but it also offers imperative features. This paper presents formal semantics of BSML with references, assignment and dereferencing. 1. I N T R O D U C T I O N Declarative parallel languages are needed to ease the programming of massively parallel architectures. We are exploring thoroughly the intermediate position of the paradigm of algorithmic skeletons [ 1, 7] in order to obtain universal parallel languages whose execution costs can be easily determined from the source code (in this context, cost means the estimate of parallel execution time). This last requirement forces the use of explicit processes corresponding to the parallel machine's processors. Bulk Synchronous Parallel (BSP) computing [ 11 ] is a parallel programming model which uses explicit processes, offers a high degree of abstraction and yet allows portable and predictable performance on a wide variety of architectures. Our BSML [6] can be seen as an algorithmic skeletons language, because only a finite set of operations are parallel, but is different by two main points: (a) our operations are universal for BSP programming and thus allow the implementation of more classical algorithmic skeletons. It is also possible for the programmer to implement additional skeletons. Moreover performance prediction is possible and the associated cost model is the BSP cost model. Those operations are implemented as a library [5] for the functional programming language Objective Caml [8]; (b) the parallel semantics of BSML are formal ones. We have a confluent calculus, a distributed semantics and a parallel abstract machine, each semantics has been proved correct with respect to the previous one. Our semantics are based on extension of the A-calculus, but our BSMr,1 i b library (a partial implementation of the complete BSML language) is for the Objective Caml language which contains imperative features. Imperative features can be useful to write (sequential) programs in Objective Caml (for example, many interpreters of A-calculus written in Objective Caml use a imperative counter in order to generate fresh variable names). Sometimes imperative features are also needed to improve efficiency. Thus to offer such expressivity in BSML we have to add imperative features to BSML. In the current version of the BSML1 i b library the use of imperative features is unsafe and may lead to runtime errors. This paper explores formal
96 semantics of BSML with imperative features. We first describe functional bulk synchronous parallel programming and the problems that appear with the use of imperative features (section 2). We then give two formal semantics of BSML with imperative features whose parallel executions are different (section 3) and conclude (section 4).
2. P R E L I M I N A R I E S
We assume here some knowledge of Bulk Synchronous Parallelism (BSP) [9]. There is currently no implementation of a full Bulk Synchronous Parallel ML language but rather a partial implementation: a library for Objective Caml. The so-called BSML12b library is based on the following elements. It gives access to the BSP parameters of the underling architecture. In particular, it offers the function b s p _ p 9u n i t - > i n t such that the value of b s p _ p ( ) is p, the static number of processes of the parallel machine. The value of this variable does not change during execution. There is also an abstract polymorphic type ' a p a r which represents the type of p-wide parallel vectors of objects of type ' a, one per process. The nesting of p a r types is prohibited. Our type system enforces this restriction [4]. In our framework, indeterminism and deadlocks are avoided and it is possible to easily prove programs using the Coq proof assistant [2]. The BSML parallel constructs operate on parallel vectors. Those parallel vectors are created bymkpar: (int->'a)->'a p a r so that ( m k p a r f) stores (f i ) on p r o c e s s i f o r i between 0 and (p - 1). We usually write f as f u n p i d - > e to show that the expression e may be different on each processor. This expression e is said to be local. The expression ( m k p a r f ) is a parallel object and it is said to be global. A BSP algorithm is expressed as a combination of asynchronous local computations (first phase of a super-step) and phases of global communication (second phase of a super-step) with global synchronization (third phase of a super-step). Asynchronous phases are programmed with m k p a r and with a p p l y - ( ' a - >' b) p a r - >' a p a r - >' b p a r . The expression a p p l y ( m k p a r f) ( m k p a r e) stores (f i ) (e i ) on process i. Neither the implementation of BSML1 i b , nor its semantics prescribe a synchronization barrier between two successive uses of a p p l y . put- (int->'a option)par->(int->'a option)par allows to express communications and synchronizations, where ' a o p t i o n is defined by: type 'a
option=None]Some
of 'a.
Consider the expression: p u t ( m k p a r ( f u n i - > f si) ) (*) To send a value v from process j to process i, the function f sj at process j must be such as (f sj i ) evaluates to Some v. To send no value from process j to process i, ( f s / i ) must evaluate to None. Expression (*) evaluates to a parallel vector containing a function fdi of delivered messages on every process. At process i, (fdi j ) evaluates to None if process j sent no message to process i or evaluates to Some v if process j sent the value v to the process i. The full BSML language would also contain a global synchronous conditional operation. This i f a t - ( b o o l p a r ) * i n * ' a * ' a - > ' a operation is such that i f a t (v, i , v l , v 2 ) will evaluate to v l or v2 depending on the value o f v at process i. But Objective Caml is an eager language and this global synchronous conditional operation can not be defined as a function.
97 That is why the core B S M L l i b library contains the function a t 9b o o l p a r - > i n t - > b o o l to be used only in the global expression: i f ( a t v e c p i d ) t h e n . . . 6 l s e . . . where ( v e c - b o o l p a r ) and ( p i d - i n t ) . i f a t expresses communication and synchronization phases. Global conditional is necessary of express algorithms like: Repeat Parallel Iteration Until Max of local errors < c. Without it, the global control cannot take into account data computed locally. Objective Caml offers to the programmer an important extension of functional languages: imperative features. They have been added to functional languages to offer more expressiveness. Classically, this modification is added to functional languages by the possibility of assignment and allocation of a variable or a data structure. The idea is to add references. A reference is a cell of the memory which could be modified by the program. One creates a reference with the allocation's ref(e) construction which gives a new reference in the memory initialized to the value of e. The value kept by the reference is called the stored value. To use and read the stored value (dereferencing), we need an operation, written l, to extract it. Finally, we can modify the content of our reference by replacing this value by another. This operation is called assignment and written: el := e2. We use the same notations as Objective Caml. A reference binding to an identifier in a functional language is like a variable in an imperative language. Imperative features are not a trivial extension of functional language. First, the value of an expression changes with the values of the free variables. If these variables have a known value, the evaluation of the sub-expressions could be done independently. For imperative language, it is not the case (and also for imperative extensions)" the evaluation of a sub-expression could modify a reference by an assignment and thus affects the evaluation of the other sub-expressions which used this reference. A secondly difficulty came from the shared references which could not pass the well-known and classical/%reduction of the functional languages. Take for example: let r=ref 2 in r-=[r*!r; (!r+l). The instances of r are for the reference, allocated by the r e f 2 sub-expression. If we make a natural/3-reduction to our expression, we would have: ( r e f 2) -= ! ( r e f 2 ) * ! ( r e f 2 ) ; ( ! ( r e f 2 ) + 1 ) which allocates four different references and do not have the same behavior. To extend the dynamic semantics of functional languages and keep out the problem of the shared allocations, locations (written 0 and store (written s) have been added. A store, is a partial function from locations to values and a reference is evaluated to a location. In the following, we give the reduction of expressions starting from an empty store. First, the left sub-expression of the 1 e t construction is evaluated and a new location g is created in the store. Second, the/%reduction can be applied and finally the fight sub-expression of the 1 e t construction is evaluated with the classical rules of a functional language:
-~
let r = ref(2) in r :=!r,!r; (It + 1) / letr-(inr:=!r,!r;(!r+l) /
0 {~2}
e := 2 9 2; (!e + 1)
{e ~ 2}
/
--~ ( l ( + l ) -~ ( 4 + 1 ) -~ 5
/ / /
{(~-+4} {e~4} {e ~ 4}
2.1. BSML with imperative features BSML is a parallel functional language based on BSP whose architecture model contains a set of processor-memory pairs and a network. Thus in the current implementation each processor
98 can reach its own memory, and it causes problems. Take for example, the expression: let if
a = ref(O) in let d a n g e r = m k p a r ( f u n p i d rood 2=0) in (at d a n g e r !a) t h e n el e l s e e2
pid
-> a - = p i d ;
First, this expression creates a location a at each processor which is initialized at 0 everywhere. For the BSML12b library each processor has this value in its memory. Second, a boolean parallel vector d a n g e r is created which is trivially t r u e if the processor number is even or f a l s e otherwise. Thus, from the BSMLZib point of view, the location a has now a different value at each processor. After the ifat construct, some processor would execute E1 and some other E2. But, the ifat is a global synchronous operation and all the processors need to execute the same branch of the conditional. If this expression had been evaluated with the BSMLZ 2b library, we would have obtained an incoherent result and a crash of the BSP machine. The goal of our new semantics is to dynamically reject this kind of problems (and to have an exception raised in the implementation).
3. D Y N A M I C S E M A N T I C S OF BSML W I T H I M P E R A T I V E F E A T U R E S This section introduces the syntax and dynamic semantics of a core langage, together with some conventions, definitions and notations that are used in the paper. 3.1. Syntax The expressions of mini-BSML, written e, have the following abstract syntax: e ::= x I ( e e ) [ f u n x - ~ e c oplletx=eine I (e,e) [ e ifetheneelsee [ ifeatetheneelsee In this grammar, x ranges over a countable set of identifiers. The form (e e') stands for the application of a function or an operator e, to an argument e'. The form fun x - . e is the lambda-abstraction that defines the function whose parameter is x and whose result is the value of e. Constants c are the integers, the booleans and we assume having a unique value: 0 that have the type unit. This is the result type of assignment (like in Objective Caml). The set of primitive operations ol) contains arithmetic operations, fix-point operator fix, test function isnc of nc (which plays the role of Objective Caml's Hone), our parallel operations (mkpar, apply, put) and our store operation ref, ! and :=. We note el :=e2 for :=(el, e2). Locations are written g, pairs (e, e). We also have two conditional constructs: usual conditional if then else and global conditional if at then else. We note 3r(e), the set of free variables of an expression e. let and fun are the binding operators and the flee variables of a location is the empty set. It is defined by trivial structural induction on e. Before presenting the dynamic semantics of the language, i.e., how the expressions of mini-BSML are computed to values, we present the values themselves. There is one semantics per value of p, the number of processes of the parallel machine. In the following, Vi means Vi E { 0 , . . . , p - 1} and the expressions are extended with enumerated parallel vectors: ( e , . . . , e) (nesting of parallel vectors is prohibited; our type system enforces this restriction [4]). The values of mini-BSML are defined by the following grammar: v ::= fun z --, e functional value c constant [ oi3 primitive ] ( v , . . . , v) p-wide parallel vector value (v, v) pair value I g location
99
3.2. Rules The dynamic semantics is defined by an evaluation mechanism that relates expressions to values 9 To express this relation, we used a small-step semantics. It consists of a predicate between an expression and another expression defined by a set of axioms and rules called steps. The small-step semantics describes all the steps of the calculus from an expression to a value. We suppose that we evaluate only expressions that have been type-checked [4]. Unlike in a sequential computer and a sequential language, an unique store is not sufficient. We need to express the store of all our processors. We assume a finite set A/" = { 0 , . . . , p - 1} which represents the set of processors names and we write i for these names and N for all the network. Now, we can formalize the location and the store for each processor and for the network. We write s~ for the store of processor i with i E N'. We assume that each processor has a store and a infinite set of addresses which are different at each processor (we could distinguish them by the name ofthe processor). We write S = [So,..., sp-1] for the sequence ofall the stores of our parallel machine. The imperative version of the small-steps semantics has the following form: e / S ~ e'/S'. We will also write e / s ~ e'/s' when only one store of the parallel machine can be modified. We note - - , for the transitive closure of ~ and note e0/S0 - - v / S for e0/So ~ el/S1 ____x e2/$2 ~ ... ~ v / S . We begin the reduction with a set of empty stores { 0 0 , . . . , Op-1} noted ON. To define the relation ~ , we begin with some axioms for two kinds of reductions:
1. e/si i e,/s~i which could be read as "in the initial store si, at processor i, the expression e is reduced to e' in the store s~IV! . 2. e / S ~ e ' / S ' which could be read as "in the initial network store S, the expression e is reduced to e' in the network store S'". We write s + {g ~ v} for the extension of s to the mapping of g to v. If, before this operation, we have g E Dora(s), we can replace the range by the new value for the location g. To define these relations, we begin with some axioms for the relation of head reduction. We write e~ [x +-- e~] the expression obtained by substituting all the free occurrences of x in el by e2.
For a single process 9 (fun x -+ e) v / si For the whole parallel machine:
~-~ e[x +-- v] / si i
(fun x --~ e) v / S
( /~i I~n)
~-~ e[x +-- v] / S M
9 (flZ~n)N .
Rules (/3~t) and (fl~t) are the same but having let x = v in e instead of fun. For primitive operators and constructs we have some axioms, the 6-rules. For each classical 6-rule, we have two new reduction rules: e / si ~ e' / s i' a n d e / S i e ' / S'. Indeed, these reductions do not 6i
(SN
change the stores and do not depend on the stores (we omit these rules for lack of space and we refer to [3]). Naturally, for the parallel operators, we also have some ~-rules but we do not have those ~-rules on a single processor but only for the network (figure 1). A problem appears with the p u t operator. The p u t operator is used for the exchange of values, in particular, locations. But a location could be seen as a pointer to the memory (a location is a memory's addresses). If we send a local allocation to a processor that does not have the location in its store, there is no reduction rule to apply and the program stops with an error (the famous segmentation fault of the C language) if it deferences this location (if it reads "out" of the memory). A dynamic solution is to communicate the value contained by the location and to create a new location for this value (as in the M a r a h a q module of Objective Caml). This solution implies the renaming of locations that are communicated to other processes. For this,
100 we define Loc the set of location of a value. It is defined by trivial structural induction on the value. We define how to add a sequence of pair of location and value to a store with: s + O = s and s + [go ~ V o , . . . , gn ~ Vn] = (S + {gO ~ VO}) + [el b--+ V l , . . . , en ~+ Vn]. We note q; = {go ~ g'o,.., gn ~ g.'} for the substitution, i.e, a finite application from location gi to another location g{ with {go,... gn} is the domain of q;.
(mkparv)/S
apply((vo,..., Vp_l) (v; ..., V~_l)) / S
~" 6M
((vO),...,(v(p-1)))/S
~
((v 0 v;) ... (Up-1 Up_l)) / S
~" 6N
el
M M
n
if ( . . . , t r u e , . . . ) n
at v t h e n ei else e2 / S
if (. . . , false, . . .) at v t h e n el else e2 / S
put((fun d s t
ep_l) ) / ~
~ eo,... ,fun dst ~
~"
/ S
if v = n
((~f atT)
e2/Sifv=n
6M ~r (r0,..., Tp_l) / S t 6N
((Sp%t)
where S = [So,.. ., 8p--1]and S' = [s~,.. ., sp__x] ' ' 1 where Vj . sj' = sj + h 'o + . . . + hp__ ! where hj = [e'o ~ Vo, . . . , g" H Vn] and hj = { (go, Vo), . . . , (gn, Vn) } where gk E Loc(ey) and {lk H vk} E sj and qpj = {go H g ~ , . . . , gn ~ g'~} and e} - ~ y ( e j ) and Vi ri -- (let v~ = eo[dst' +-- i] i n . . . V~_li _ ep_lt [dst +-- i] infi) where fi -- f u n x - + if x -- 0 t h e n v~ e l s e . . , if x = (p - 1) t h e n v~_ii else n c O Figure 1. Parallel 5-rules Now we complete our semantics by giving the &rules of the operators on the stores and the references. We need two kinds of reductions. First for a single processor, 5-rules are (5~f), (5~) and (5~=) (given in figure 2). Those operations work on the store of the processor where this operation is executed. The r e f operation creates a new allocation in the store of the processor, the ! operation gives the value contained in the location of the store and the := operation changes this value by another one.
ref(v)
/ s~
!(g) / si e:-v / ~
e
5~ ~ 5~ ~, &
g / si + {g H v}
ifg r
Dom(si)
(Sref
si(g) / si
ifg E Dom(si)
(5~)
() / ~ + {e H v}
if e ~ Dom(,~)
(at__)
S t=
[80 -[- {e~ ~+ ~')0(V)},..., 8p--1 qt- {~:~-1 ~+ ~DP--1(V)}]
ref(v) / S ~',~ e~ / S' where gi.g~ q~ Dom(si) !(g~a) / S W:=v / S
e
v/ S
where 3v Vi s~(g~) = v and g~ ~ Dom(s~) or 3v v = P-l(s~(g~))
~ a~
() / S'
where
5N
&
S t--. [8 0 --[- {e~ ~-+ ~Oo(V)},...
Vi.g~ ~ Dom(si)
8p--1 + {e~__1 w-+ ~Op__1 (v)}]
(a~) (a~) (a~) ((~proj)
Figure 2. "Store" 5-rules For the whole network, we have to distinguish between the name of a location created outside
101 a m k p a r which is used in expressions and its "projections" in the stores of each process. We note 1~ in the first case and l~ for its projection in the store of process i. When an expression outside a m k p a r creates a new location, each process creates a new location (an address) on its store (rule (~ef), figure 2) where 7~i is a trivial function of projection for the parallel vectors of a value. With this function, we assure that there is no data from another processes in a store. The assignment of location 1N (rule (6.~), figure 2) modifies the values of locations l~ using also the function of projection (which have no effect if the value do no contain any parallel vectors). This rule is only valid outside a mkpar. But a reference created outside a m k p a r can be affected and deferenced inside an mkpar. For assignment, the value can be different on each process. To allow this, we need to introduce a rule (6~roj) (figure 2) which transforms (only inside an m k p a r and at process i) the common name l N into its projection l~. Notice that the affection or the deferencing of a location l N cannot be done inside a m k p a r with rules (~i=) and (6() since the condition l c Dom(si) does not hold. The use of the rule ((~pNroj)is first needed. The deferencing of 1N outside a m k p a r can only occur if the value held by its projections at each process is the same or if this value is the projection of a value which contain a parallel vector (a value which cannot be modify by any process since nested of parallelism is forbidden [4]). This verification is done by rule ( ~ ) where 7~-1 is a trivial function of de-projection for the values stores one each processes which contains the projection values. This de-projection do no need any communication. The complete definitions of our reductions are:
-
U i
and - - ' : - ~ U ~-~. It is easy to see 6i
N
~N
that we cannot always make a head reduction. We have to reduce in depth in the sub-expression. To define this deep reduction, we need to define two kinds of contexts. We refer to [3] for such definitions.
3.3. Cost model preserving semantics In order to avoid the comparison of the values held by projections of a 1N location in rule ( ~ ) we can forbid the assignment of such a location inside a mkpar. This can be done by suppressing rule (6~roj)" But in this case, deferencing inside a m k p a r is no longer allowed. Thus, we need to add a new rule:
!(~N) / si
~-~ 6~
si(g~) / si
(6~ ) to suppress the comparison: !(gN) / S
~~N
if ~ C
Dom(s~) (6~N)
7~-1(s~(/~)) / S
and modify the rule
ifg~ c
Dom(s~) (3,N,) " "
This rule is not deterministic but since assignment of a 1N location is not allowed inside a
mkpar the projections of a 1N location always contain the same value. The cost model is now compositional since the new (~'~) does not need communications and synchronization. 4. CONCLUSIONS AND FUTURE W O R K The Bulk Synchronous Parallel ML allows direct mode Bulk Synchronous Parallel programming. The semantics of BSML were pure functional semantics. Nevertheless, the current implementation of BSML is the BSMT.1• library for Objective Caml which offers imperative features. We presented in this paper semantics of the interaction of our bulk synchronous operations with imperative features. The safe communication of references has been investigated, and for this particular point, the presented semantics conforms to the implementation. To ensure safety, communications may be needed in case of assignment (but in this case the cost model is no longer compositional) or references may contain additional information used dynamically
102 to ensure that dereferencing of references pointing to local values will give the same value on all processes. We are currently working on the typing of effects [1 O] to avoid this problem statically. REFERENCES
M. Cole. Algorithmic Skeletons: Structured Management of Parallel Computation. MIT Press, 1989. [2] F. Gava. Formal Proofs of Functional BSP Programs. Parallel Processing Letters, 2003. to appear. [3] F. Gava and F. Loulergue. Semantics of a Functional BSP Language with Imperative Features. Technical Report 2002-14, University of Paris Val-de-Marne, LACL, october 2002. [4] F. Gava and F. Loulergue. A Polymorphic Type System for Bulk Synchronous Parallel ML. In Parallel Computing Technologies (PACT 2003), LNCS. Springer Verlag, 2003. [5] F. Loulergue. Implementation of a Functional Bulk Synchronous Parallel Programming Library. In 14th lASTED PDCS Conference, pages 452-457. ACTA Press, 2002. [6] F. Loulergue, G. Hains, and C. Foisy. A Calculus of Functional BSP Programs. Science of Computer Programming, 37(1-3):253-277, 2000. [7] F.A. Rabhi and S. Gorlatch, editors. Patterns and Skeletons for Parallel and Distributed Computing. Springer, 2002. [8] D. R6my. Using, Understanding, and Unravellling the OCaml Language. In G. Barthe and al., editors, Applied Semantics, number 2395 in LNCS, pages 413-536. Springer, 2002. [9] D.B. Skillicorn, J. M. D. Hill, and W. F. McColl. Questions and Answers about BSP. Scientific Programming, 6(3):249-274, 1997. [ 10] Jean-Pierre Talpin and Pierre Jouvelot. The Type and Effect Discipline. Information and Computation, 111(2):245-296, June 1994. [11 ] Leslie G Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103, August 1990. [1 ]
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
103
The Use of Parallel Genetic Algorithms for Optimization in the Early Design Phases E. Slaby a and W. Funk b aWaldorfer Str. 8, D 50969 K61n, Germany slaby @unibw-hamburg. de bInstitute of Machine Design and Production Technology University of the Federal Armed Forces, D 22039 Hamburg, Germany This paper deals with the use of genetic algorithms, which are integrated into a commercial CAE-system in such a way that all the CAE-systems functions can be used for evaluation and data processing. To reduce runtime, a parallelization of the fitness evaluation routine based on the CORBA technology is presented. 1. I N T R O D U C T I O N In these days, the use of CAE-systems (Computer Aided Engineering) for the design process is quite usual in many development departments. Presently a great number of different computerized support systems for the engineering design process are available. Particularly in the last years even more computation and dimensioning tools were integrated into the systems. But most of these tools are applicable only in the later design phases. The early phases like "planning" and "conceptual design" are still relatively unexplored issues concerning computer aided systems. Optimization algorithms and technologies in the field of Computational Intelligence can hardly be found as an integrated component of the systems [3]. Therefore the use of these algorithms is quite rare in practice. Among others the reason for this are long training periods for the user, difficult handling and the high programming efforts which are necessary for each new task. On the other hand some projects ([5], [7], [2]) show the large potential of these technologies for the design process, even to support the engineer in those areas which are considered as a domain of human intuition and creativity. Bentley points this out in [ 1], where particularly the evolutionary algorithms are used not only for the optimization of already existing construction units, but also for the generation of new concept variants. With a closer view on such projects it is noticeable that besides the engineer as the user a computer scientist is often needed for the software-technical realization. Such a time-consuming effort with additional personnel expenditures cannot be operated in practice during the already temporally limited design process. For this reason a software has been developed that combines an evolutionary algorithm and a commercial 3D-CAE-system (I-DEAS Master Series) in such a way that no programming has to be done by the engineer.
104 At first this paper gives a short introduction into evolutionary algorithms (EA), considers the disadvantages of existing applications which use EA's in section three. Section 4 points out the intention of the developed software-system for the integration of EA's into an existing CAE-system (section 5), section 6 deals with the parallelization followed by the use of a neural network for the EA fitness evaluation routine in section 7. 2. EVOLUTIONARY ALGORITHMS Evolutionary strategies, especially genetic algorithms, represent promising tools for the engineering design process. Evolutionary strategies imitate the principles of natural evolution: reproduction, mutation and selection. Starting with an initial random population, the fitness of each individual is evaluated under consideration of the given constraints. The strings representing the individuals are crossed-over and mutated. The individuals of such a generation are tested and given a probability to survive according to their fitness. After a sufficient number of generations, optimal or nearly optimal solutions can be found. By randomly mutating some strings, areas of the solution space are examined which may not have been considered with the initial population [4]. Some fundamental differences to conventional deterministic and stochastic search methods are [8]: 9 Evolutionary algorithms select in a parallel way in a population of points, not only from one individual point. 9 No derivatives of the objective function or other auxiliary information are needed by the evolutionary algorithms. Only the objective function value is used as a basis for the search. 9 Evolutionary algorithms can offer a number of possible solutions for a problem. 9 Evolutionary algorithms can be applied to problems with different representations of the variables (continuous and discrete variables). 9 Evolutionary algorithms are simple and flexible in use and there are no restrictions for the definition of the objective function. Evolutionary algorithms represent thus versatile, durable and efficient optimization procedures, which can be used with strongly nonlinear, non-continuous objective functions and for problems with variables of different representation (binary, integral, real). It is recommended not to use evolutionary algorithms if the computation of the objective function is very complex or timeconsuming, because a large number of objective function computations is necessary [8]. Basis of the implemented optimization module is the software package ECJ (Evolutionary Computing system written in Java [6]). A modified genetic algorithm is used that fits to a broad band of design problems and leads to good results without beeing adapted in an individual way to each problem by the user. Because of the parallel way of stepping through the solution space, EA's suit for a parallel fitness evaluation very well. Thus will be discussed in section 5.
105 3. EXISTING APPLICATIONS USING EA'S IN THE DESIGN PROCESS Existing examples which use genetic algorithms in the early design phases show the great potential that lies among those algorithms for the optimization of design-concepts and the possibility to create new designs from scratch. On the other hand, a lot of programming work is necessary to develop the algorithm and the necessary input/output interfaces and the fitness evaluation routine. In many cases, a software developer has to support the engineer. The resulting software often fits only to a small kind of problems and has to be adapted with a lot of work to a different task. The input of the optimization objectives as well as the interpretation of the results of an optimization run is often very difficult because of a lacking linkage to a CAE-system. The EA's output data must usually be converted with an additional step or entered manually. Some applications contain a special geometry model for the computation and representation. Such a development is connected with a great programming effort and does not reach the functionality of a modem CAE-systems (e.g. free forming surface modelling). For many of the examined applications computation routines are especially programmed for the evaluation of the solution variants, instead of using already existing computation tools of a CAE-system. 4. INTENTION OF "EasyEvoOpt-3D" The intention of the developed system "EasyEvoOpt-3D" is to offer the engineer a tool which gives him the possibility to use the above mentioned technologies during the product design process via the 3D-CAD-systems graphical user interface without the need of deeper knowledge of the used algorithms. In detail the following points have been considered for the development: 9 Optimization algorithms have to be integrated into a CAE-system to offer the possibility to input all relevant information with the help of the CAE-systems input functions. 9 The computation routines of the CAE-system should be usable by optimization algorithms, in order to be able to evaluate individual solutions. 9 The results of the optimization run have to be available in the CAE-system without further action or data conversion. 9 The use of the optimization procedures should be possible for a wide range of different tasks without further adjustment of the optimization algorithms. 9 If necessary, there should be an option to particularly adapt the optimization algorithm to the problem. 9 A modular expendability has to be planned, so that the integration of further external applications for the evaluation of solution variants is possible. A main point for the development of the system is that an engineer should have the possibility to let the system generate solution variants at an early phase of the design process without programming efforts and without a deeper knowledge of the optimization algorithms.
106 5. INTEGRATION INTO THE CAE-SYSTEM I-DEAS The geometry model of the CAE-system is regarded as the basic element of the optimization. Thus the computation tools of the CAE-system are available apart from the possibilities for visualization. The definition of the restrictions should be made not by mathematical descriptions, but by graphic elements (e.g. restricted areas).
........~: ...................................................................................................................... .: ..............................,+r.......................
IV Object-data, ~ Restrictions. optimization-
. . . . . . . . . . . .
"............................................
Record-Mode
Figure 1. User inputs at the CAE-system recorded by the optimization module
All above mentioned points - even the input of the restrictions and the objective function have to be available without programming by the user. With this demand, the communication of the optimization module with the CAE-system gets a central meaning for the further procedure. Here two different cases of communication between the CAE-system and the optimization module can be pointed out. The optimization module has to record the inputs such as variables, optimization objective and restrictions done by the user via the CAE-system and convert them in such a way, that the optimization module can replay them without any user interaction. This means that the variables have to be set according to the outputs of the evolutionary algorithm, and that the fitness evaluation routine is able to use the CAD-systems tools. The possibilities to enter the problem, the restrictions and the fitness evaluation with userfriendly graphical tools of the CAD-system are shown in figure 1. The inputs are recorded and processed in such a way that the tools of the CAD-system can be used for fitness evaluation without any user interaction as shown in figure 2. The communication module for communication with the CAE-system I-DEAS is represented by a Java class named o:r_com (Open I-DEAS Communication). Task of this class is to execute the entire internal CORBA communication and to perform the necessary conversions for communication with the CAE-system I-DEAS. Low level communication with I-DEAS has been developed with the Open I-DEAS programming interface. These CAE-systems library is based on the CORBA technology. It offers a number of methods in order to access internal data of I-DEAS. In the first phase of the optimization application the user has to define which parameters of
107
EasyEv0Opt-3D
, .................. .
OptimizationRun Figure 2. Fitness evaluation without any user interaction a construction unit are variable. For users comfort, this can be done by interactively picking the dimensions or other defining parameters in the graphics region of the CAE-system. The Open-I-DEAS API includes a method to get the 3D-coordinates of any picked point which is not useable for the later use because by changing some parameters of the object, their position in space will change and the system could not pick the element afterwards. For this reason the OI_Com-class contains different methods, which offer the possibility to convert the selection of an element in the graphics region from I-DEAS into the system internal labels. The OI_Com-class also includes methods to access all computation results in the CAEsystem, even if these routines are not supported by the Open-I-DEAS API. This is possible with a specially developed parser which can access and filter all internal result lists and pass them to the optimization module. 6. PARALLEL FITNESS EVALUATION The disadvantage of the use of these CAE-system tools lies in an unsatisfying runtime. Additionally, the convergence of genetic algorithms is generally slow because of the huge number of fitness evaluations. Each individual in a generation encodes a possible solution that is independent from the information of other individuals. Therefore the fitness of the individuals can be evaluated in parallel. As fitness evaluation with the tools of the CAE-system is the most time consuming part of the process, a significant reduction of execution time can be realized through the parallelization. The implemented parallelization scheme had to be applicable on distributed memory systems with different operation systems (Windows, Unix) connected via a LAN and on multiprocessor supercomputers. Therefore the main focus of this section lies on developing a parallel fitness evaluation routine which is able to use many processes of the CAD-system on a heterogeneous cluster of workstations as it can be found in the development department of a engineering company. The parallel fitness evaluation routine is able to integrate workstations into the cluster with different operation systems (Windows, Unix) and also multiprocessor-systems, on which a lot of CAD-system processes can be executed. The parallelization-routine uses Java-threads and the Common Object Request Broker Archi-
108 tecture (CORBA). This architecture offers a possibility to communicate with different processes of the CAE-system, even on different operating systems. It is not necessary to change the CAEsystems source code. An advantage of this approach is the fact that the developed system offers the possibility to use all available CAE-workstation for the parallelization without having to modify the system or even boot a different operating system. The parallelization takes place in a master thread, which is responsible for the distribution of the individual computations as well as the error handling, and slave threads, which can be instantiated several times and perform the direct communication with the CAE-system. The master thread communicates via messages with the slave threads. Each slave thread has a separate message register. Via this register defined messages conceming the status (READY, C A L C U L A T I O N FINISHED, RESULT RECEIVED, ERROR) and for calling subroutines ( I N I T , CALCULATE FITNESS, SLEEP, DONE) can be passed. To avoid slower computers to slow down the whole process, a load balancing is included. If all individual of one EA's generation have been sent to a slave for evaluation and some slaves are already waiting for new individuals, some of the not yet evaluated individuals are sent again to another slave. If a result has been retumed, the other slaves evaluating the same individual are able to stop their run.
7. EVALUATION OF FITNESS VALUES W I T H A NEURAL N E T W O R K A second approach for a reduction of run-time of an optimization run consists in the use of a neural network for the evaluation of the individuals' fitness values. Basis of this concept is to train a neural network with the data, which describe the solution variants, and which associated fitness values computed by the CAE-system. If the neural network achieved a given quality by many learning procedures, a determination of the fitness values can be accomplished by the neural network, which is very much faster than the complex computation of the CAE-system. A neural network must be trained at first with example data, in order to obtain a desired behavior. Therefore the neural net is trained with the individuals, for which the CAE-system did already compute a fitness value. If for a given number of new data sets a demanded accuracy is reached by the neural network, it will be changed over to the second phase, the evaluation of the fitness values by the neural network. In this second phase there is a mixture of fitness values evaluated by the neural network and the computation of selected individuals made by the CAE-system. A closer view of the genetic algorithm is necessary at this point. The best individuals of a generation are preferred for reproduction. For the remaining individuals only the order concerning their quality in the population is important. The exact fitness value must be available only for the best individuals. Therefore the fitness values of all individuals of one generation are evaluated by the neural network and additionally the fitness values of the best individuals as well as some randomly selected are computed by the CAE-system. The neural network module was developed with MatLAB and converted with the MatLAB/C++ compiler to C++ source code. The CORBA-methods were implemented so that the neural network module can be integrated in the optimization system via a network. Figure 3 gives an overview of all components of the optimization system an the integration of the CAE-systems for input/output and fitness evaluation.
109
i
M
Multiprocessor CORBACORBA-
coRPBCA.
fs ....
CAD/CAE
CORBA-Communication-Modul CAD/CAE-System i:i....... i:i~:!~I
Ii!i:
Parallelization-Modul (multithreaded) I
Graphical User Interface
(GU~)
lodul [" I
Evolutionary Algorithm Datarecording and Conversion
Datenbasis
Figure 3. Overview of all components
8. R E S U L T S A N D C O N C L U S I O N
The optimization system "EasyEvoOpt-3D" has been successfully applied for the search for a non-uniform transmission for a wicket and a ergonomics-study for a bench [9]. The flexibility to user changes and the possibility to follow the optimization steps generation by generation on screen has shown the advantages of the use of the CAE-system. The disadvantages of the quite time-consuming CAE-tools for the fitness evaluation could be compensated by the parallelization and the use of a neural network for the fitness evaluation. The system shows that it is possible to use optimization methods at an early stage of the design process without having to consult a optimization and software specialist with just using the already available hardware. REFERENCES
[1] Bentley, Peter John: Generic Evolutionary Design of Solid Objects using a Genetic Algorithm; Dissertation University of Huddersfield 1996 [2]
Dasgupta, D.; Michalewicz, Z (Hrsg.): Evolutionary Algorithms in Engineering Applica-
110 tions; Berlin, Heidelberg, New York, u.a.: Springer 1997 Figel, Klaus: Optimieren beim Konstruieren: Verfahren, Operatoren und Hinweise ffir die Praxis; Mfinchen, Wien: Hanser 1988 [4] Goldberg, David E.: Genetic Algorithms in Search, Optimization & Machine Learning; Massachusetts, Harlow, Menlo Park, u.a.: Addison Wesley Longmann, 1989 [5] Hafner, S.;Kiendl, H.; Kruse, R.; Schwefel, H.-P.: Computational Intelligence im industriellen Einsatz: Tagung Baden-Baden, Mai 2000 VDINDE-Gesellschaft Mess- und Automatisierungstechnik; Diisseldorf: VDI-Verlag 2000 [6] Luke, Sean: Evolutionary Computing system written in Java. http://www.cs.umd.edu/users/seanl/, University of Maryland [7] Miettinen, K.; Neittaanmdki, P.; Mdkeld, M. M.; Periaux, J. (Hrsg.): Evolutionary Algorithms in Engineering and Computer Science; Chichester, Weinheim, New York u.a.: Wiley 1999 [s] Pohlheim, Hartmut: Evolution~ire Algorithmen: Einsatz von Optimierungsverfahren, CAD und Expertensystemen; Berlin, Heidelberg, New York, u.a.: Springer 2000 [9] Slaby, Emanuel: Einsatz Evolution~irer Algorithmen zur Optimierung im frfihen Konstruktionsprozess. Fortschr.-Ber. VDI Reihe 20 Nr. 361. Dfisseldorf: VDI Verlag 2003
[3]
Parallel Computing: SoftwareTechnology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
111
A n Integrated A n n o t a t i o n and C o m p i l a t i o n F r a m e w o r k for Task and D a t a P a r a l l e l P r o g r a m m i n g in Java* H.J. Sips a and K. van Reeuwijk aDelft University of Technology (sips,reeuwij k)@its.tudelft.nl
1. I N T R O D U C T I O N For most applications Fortran has been replaced by languages such as ANSI C, C++. and Java. Yet Fortran has a number of features that still make it particularly useful for scientific programs, namely multi-dimensional arrays, complex numbers, and, in later versions, array expressions. Moreover, Fortran compilers tend to produce highly efficient code compared to compilers for other languages. We have developed a set of language constructs, called Spar, that augment Java with a set of language constructs for scientific programs. The set consists of multi-dimensional arrays; complex numbers; a 'toolkit' to build specialized array representations such as block, symmetric, or sparse arrays; annotations; and parallelization. In this paper we concentrate on the constructs for parallel programming; the other extensions have been described in [9]. In this paper, we present a unified model to describe data and task parallelism in an object oriented language. The approach allows compile-time and run-time techniques to be combined. 2. THE F O R E A C H C O N S T R U C T In a sequential program the execution order of all statements is specified exactly. This is often an over-specification: the programmer may not care about the execution order of statements, even if the observable results differ. For example, the code for(
int i=O;
i
i++
) a[i]
= Math.sin(
2*Math. PI/i
);
fills array a in the exact order that is specified in the f o r loop. This is an over-specification, since order in which the array is filled is irrelevant for the final result. Since this form of overspecification often prevents parallelization of the program, Spar provides the e a c h and f o r e a c h statements. Given an e a c h statement such as: each
{ sl;
s2;
}
the compiler may choose one ofthe execution orders s l ; s2 ; or s2 ; s l ;. Once a statement has been started, it must complete before another statement is started. *This research was supported by NWO (project 'Automap'), Esprit (LTR Project 'Joses' (#28198)), and Delft University of Technology (DIOC project 'Ubicom').
112 The f o r e a c h statement is a parameterized version of the e a c h statement. For example, we can now write the loop above as: foreach(
i :- O : a . l e n g t h
) a[i]
= Math.sin(
2*Math. PI/i
);
Similar to the e a c h statement, the iterations may be executed in any order, but an iteration must complete before the next one is started. 3. T H E A N N O T A T I O N L A N G U A G E Statements, expressions, types, declarations, formal parameters, and the entire program can be annotated with pragmas. For example: <$ reduction, i t e r a t i o n s = @ n
$> foreach(
i :- O:n
){ sum += i;
}
annotates a f o r e a c h statement with two pragmas: a r e d u c t i o n pragma, and an i t e r a t i o n s pragma. Spar provides a general annotation mechanism. An annotation consists of a list ofpragmas. These pragmas allow the user to give the compiler information about the program, and give hints for efficient compilation. By convention, a pragma does not influence the behavior of a program2; it only improves the efficiency of the program in terms of execution time, memory use, or any other measure. For example, the annotation: <$ i n d e p e n d e n t , b o u n d s c h e c k = f a l s e ,
iterations=42
$>
consists of three pragmas. The first one, i n d e p e n d e n t , does not have a value, and is called a flagpragma. The other two, b o u n d s c h e c k and i t e r a t i o n s , have a value, and are called value pragmas. Identifiers in the pragma name and pragma value are completely independent of those in Spar/Java; they have a different name space. It is possible, however, to refer to variables in the host language by prefixing a name with a '@'. For example, in the code fragment foreach(
i :- O:n
) <$ i t e r a t i o n s = @ n
$>
{ sum += i;
}
the value of the i t e r a t i o n s pragma contains a reference to the variable n. Instead of a single value, a pragma may have a list of values. Such a list is written as a sequence of expressions surrounded by brackets. Since the lists can be nested to an arbitrary depth, this allows expressions of arbitrary complexity. For example: <$cost=(lambda
(i j)
(sum
(prod 5 i i j j)
(prod 3 i j) 42)$>
As a convenience, Spar allows a number of binary operators to be used in pragma expressions. The traditional precedence rules on these operators are obeyed. These expressions are immediately translated to expression lists. For example, the pragma expression 3 * n * n + 5 * n + 1 is translated t o ' ( s u m ( p r o d 3 n n) ( p r o d 5 n) 1 ) ' Furthermore, Spar allows subscript-like expressions. They are immediately translated to a list starting with the identifier' a t ' . For example, the pragma expression 'p [ 1, a ] ' is translated to '(at
p
1 a)'.
2Since the each and f o r e a c h statements are non-deterministic, we consider a program to have the same behavior for all possible execution orders of these statements. However, an annotation may cause another execution order of such a statement.
113
4. P R A G M A S F O R P A R A L L E L I Z A T I O N
Both data declarations and code fragments can be annotated with the on pragma, which specifies the placement of that data, or code. For example" int[*,*] <$ on:(lambda b = new int [50,50] ;
(i j) dsp2D[(block
j 5), all])
$>
annotates an array with a placement pragma in data-parallel fashion, and foreach( i -- 0:I00 ) <$ on=dsplD[ (block @i 25) ] $> { a[i] = a[i] + i; }
annotates the iterations of a foreach with a placement pragma in task-parallel fashion. To help the compiler with the parallelization of a program, the user may annotate a program with pragmas to specify the distribution of data, or the place where a block of code is executed. For this purpose the parallelization engines support the following annotations: The ProcessorType pragma is used to declare names of processor types and to associate them with processors characteristics (e.g. alignment of data structures and endianness of primitive types etc.) and capabilities (e.g. whether it contains a FPU etc.). P r o c e s s o r T y p e pragmas must be global. For example: <$ ProcessorType:( (Gpp "Pentium2")
(Dsp "Trimedia"))
$>
The strings " P e n t i u m 2 " and " T r i m e d i a " refer to processor descriptions that are known to the compiler. A P r o c e s s o r s pragma is used to name the processors in a system, and describe their arrangement to the compiler. The P r o c e s s o r s pragma must be a global pragma. Either a single processor can be declared, or an array of processors. The processor array can have any number of dimensions. For example: <$ Processors:( (Gpp gppl)
(Dsp dsplD[4] ) (Dsp dsp2D[2,3] )) $>
This system consists of a single processor g p p l of type Gpp, a one-dimensional processor array d s p l D , and a two-dimensional array dsp2D, both of type Dsp. Each modeled processor corresponds to a single physical processor. A Spar/Java program can be annotated with on pragmas to place data and work on specific processors. Pragmas can annotate expressions, statements, and member functions. <$ on=Gpp $> <$ on:Dsp[0] $>
Pragmas may be nested; an on pragrna on an inner block or expression overrules the enclosing specification. The special on value _ a l l is used to denote all processors; the value '_' (a single underscore) is used to denote an unspecified distribution ('don't care'). It is also allowed to use three special functions as on expressions: The (b 1 o c k a* i +b m) distribution function places index i onto processor p - (a. i + b ) / m . The value ofp is bounded by the index range in the corresponding dimension of the processor type. If no m is specified, the value is derived from the context, if possible" if there are N elements in the corresponding
114 array dimension, m = N/Pext is assumed. The ( c y c 1 i c a* i +b) distribution function places index i onto processor p = ( a . ~ + b) rood Pext. The ( b l o c k c y c l i c a * i + b m) distribution function places index i onto processor p = ((a. i + b)/m) rood P~xt. For data that is distributed with the b l o c k and c y c l i c functions, the compiler is able to generate highly efficient code to enumeration local elements and to translate global to local array indices; see [8] for details. With these functions, all the data mappings of High-Performance Fortran (HPF) 2.0 [5] can be specified. Annotating declarations By annotating a member function, the user can specify the group of processors allowed to execute the member function. For example: <$ on d s p l D [ _ a l l ] $> int m y f u n c ( int i, S t r i n g
s ) { return
i+l;
Annotating statements A statement annotated with an
}
on
pragma will be executed only on the specified processor(s). In principle arbitrary statements may be annotated, but in practice the annotation is mainly interesting for code blocks. Annotating expressions In principle, any expression can be annotated with an o n pragma. This specifies the place where the expression must be evaluated. The new expression is a special case, since this not only specifies the distribution of the constructor execution (if not overridden by an annotation on the constructor), but also the distribution of the newly constructed class or array instance. For example, the pragma: String
a = <$ o n = g p p l
$> n e w
String();
specifies that a new S t r i n g must be constructed on g p p l . In the case of array new expressions, a slightly extended version of the on pragma is allowed. For example, the pragma: int [*,*] <$ o n = ( l a m b d a (i j) d s p 2 D [ ( b l o c k b = n e w int [50,50] ;
j 5),_all])
$>
specifies that every array element b [ i , j ] is constructed on processors d s p 2 D [ ( b l o c k j 5) , _ a 1 1 ] . The _ a l l expression in the second dimension means that the elements are replicated in that dimension. Since the formal parameter i is not used in the distribution expression, the first dimension of the array does not influence the distribution of elements. Other annotations The Spar/Java programs that we discuss in Sec. 5 use two other annotations. The f o r e a c h statements allows reduction operations such as int s u m = 0; <$ r e d u c t i o n
$> f o r e a c h (
i -- 0 : a . l e n g t h
) sum
+= a[i] ;
If the array that is being reduced is b l o c k or c y c i i c distributed, and if the loop is annotated as a reduction, the parallelization engines are able to generate efficient parallel code for such a reduction. As mentioned in Sec. 2, the e a c h and f o r e a c h statements can only be executed in parallel when there is no observable interference between statements or iterations. Such loops are annotated with the i n d e p e n d e n t annotation.
115 1000
. . . . . . . . . .
loo L
spa; ' ; ' "
,Spar
. . . . . . . . . .
~,
-~-'
'
Spar (unchecked) ---• H P F --- ~---
~..:.::. ...-..
G)
100
.g
E
._
c .o_
.~
u.I
w
10
p,
"rq
1 . . . . . . .
,
,
,
, ,,,,I
,
1'0
10
N u m b e r of p r o c e s s o r s
N u m b e r of p r o c e s s o r s
. . . .
,,, 100
Figure 1. Results for the NAS EP 'A' dataset (228 samples) and for the NAS FT 'W' dataset (128 x 128 x 32 elements), left and right figure, respectively 5. RESULTS To evaluate the practicality of our language extensions, we have implemented a compiler for Spar/Java. This compiler is called 'Timber'. It is available for downloading at [7]. To measure the performance of the experimental compiler, we have implemented three benchmarks from the NASA Numerical Aerospace Simulation group (NAS) parallel benchmark suite Version 2.3 [6, 4]. Guided by the HPF and Fortran 77 versions we have available, the algorithms were reimplemented in Spar/Java. For the benchmarks, the original NAS implementations for Fortran 77/MPI were used as reference. For the FT and MG benchmarks the HPF version was also used as reference. For the EP benchmark no HPF version was available. All measurements were done on the DAS [3] distributed supercomputer. The Fortran/MPI implementation was compiled with the PGI Fortran 90 Version 3.1 compiler. The HPF version was compiled with the PGI HPF Version 3.1 compiler. All versions, including the Spar/Java version, used the communication library Panda [ 10]. The Fortran/MPI and HPF version used the MPI library implemented on top of Panda, the Spar/Java version used Panda directly. Due to their method of implementation, the Fortran/MPI version of FT and MG can only be run for a number of processors that is a power of 2. The HPF and Spar/Java versions do not have this restriction. NAS EP benchmark. The NAS EP benchmark approximates 7c by choosing random points in a square, and counting the percentage that falls in the inscribed circle of the square. To parallelize the code, the calculations are divided in batches. The result for each batch (the percentage of 'hits') is stored in an element of an array; the global result can then be calculated with a reduction operation on this array. The Spar/Java version contains the following annotations: a distribution annotation for the results array; a distribution annotation for the loop that iterates over the batches; and an annotation to label the result reduction loops as reductions. From these annotations only, the compiler is able to generate a parallel program, with the results shown in Fig. 1. Note that both data and task distribution is used. The Spar/Java program scales just as well as the Fortran/MPI version, and is slightly faster. NAS FT benchmark. The NAS FT benchmark performs a 3D Fast Fourier Transform (FFT). The Spar/Java version distributes the FFT array in cyclic fashion in one dimension. Thus, if a 64 x 64 x 64 array is distributed over 2 processors, each processor gets a 64 x 64 x 32 elements
116 1000 .
.
.
.
.
.
.
.
HpF ' ; ' "
.
Spar Spar
100
._~
---•
unchecked F77
---~---
......~a......
!. . . . . " ' - - x .
..........
"--..'-x.
~5 iii ..... N .... ....... [3. .....
.... [] ,
,
i
,
, ,,,1
, 10
Number
,
i
. . . . . 100
of processors
Figure 2. Results for the NAS MG 'W' dataset (64 x 64 • 64 elements).
slice of the array. For 4 processors the slice is 64 x 64 • 16, and so on. In the original NAS benchmark program, The 3D FFT is implemented by doing a 1D FFT on all one-dimensional array sections parallel to the x axis, then parallel to y axis and then the z axis. The Spar/Java version always performs 1D FFT transforms on array sections parallel to the x axis. For the last two phases, it copies and permutes the elements in the original array to a suitably distributed temporary array, performs the 1D transform, and copies the elements back again. Since we distribute the temporary array in a different dimension, all elements for the 1D FFT transform on one array section are always locally available, which allows it to be implemented very efficiently. The Spar/Java version contains the following annotations: distribution annotations for the distributed arrays; a distribution annotation for the iteration over the 1D FFT transform; a r e d u c t i o n annotation on the loop that calculates the verification checksum, and a number of i n d e p e n d e n t annotations. Using these annotations only, the compiler generates a parallel program, with the results shown in Fig. 1. Again, note that both task and data distributions were used. In most cases the Spar/Java results compare favorably with those for Fortran/MPI and HPF. For larger number of processors secondary effects become significant, such as initialization and setup times, and load imbalance due to the distribution 3. NAS MG benchmark. The NAS MG benchmark is a multigrid solver using grids of different coarseness. The Spar/Java version only uses distribution annotations for the different arrays; an annotation to label the loop that calculates the verification checksum as a reduction, and a number of annotations to label loops as 'independent'. Using these annotations only, the compiler generates a parallel program, with the results shown in Fig. 2. As these measurements indicate, the overhead of null pointer checking and bounds checking in the Spar/Java version is somewhat higher than for the other benchmarks. The performance difference with F77/MPI is caused by the larger number of copy operations on the data, and an inefficient use of the available cache hardware.
3For example, if an array of 64 elements is distributed over 24 processors, some processors get 2 elements, and some processors get 3 elements.
117 6. RELATED W O R K
Spar supports the same class of distributions as HPF 2.0, but uses a significantly different notation. In HPF a distribution annotation is used that is more limited than the Spar version. Whereas in Spar the annotation ( c y c 1 i c a* i +b) is allowed, in HPF only the annotation (cyclic i ) is allowed. However, HPF provides alignment statements to specify a distribution relative to another distribution. Through this mechanism the equivalent of distribution ( c y c l i c a* i +b) can be specified. The absence of alignment makes distributions somewhat less convenient to express, but we think that this is a small inconvenience. On the other hand, distributions like ( c y c l i c a* i +b) can be expressed more conveniently than in HPF, since no alignment construct is necessary. The b l o c k , c y c l i c , and b l o c k c y c l i c distributions can also be used for the efficient distribution of loops. Spar is not the only proposal to extend Java for scientific computing. HPJava [2, 12] adds multi-dimensional arrays and data-parallel programming to Java. In contrast to Spar, the programs are explicitly data-parallel. That is, any data transfer between processors has to be explicitly done in the program by a call to the HPJava communication library. Blount, Chatterjee, and Philippsen [1] describe a compiler that extends Java with a f o r a l l statement. To execute the f o r a l l statement, the compiler spawns a Java thread on each processor, and the iterations are evenly distributed over these threads. Synchronization between iterations is done by the user using the standard Java synchronization mechanism. No explicit communication is performed; a shared-memory system is assumed. Due to the dynamic nature of the implementation, they can easily handle irregular data and nested parallelism. Titanium [11] provides vectors, multidimensional arrays, iteration ranges, and a f o r e a c h statement comparable to those in Spar. In most cases the Spar version of these constructs is more general. They explicitly state that their f o r e a c h is not intended for parallelization. Titanium supports iterations over arbitrary sets; moreover these iteration ranges are 'first-class citizens'; they can be handled and modified independent of any loop statements. Unsurprisingly, support for arbitrary iteration sets leads to inefficiencies. Therefore, Titanium provides a separate representation for rectangular iteration. In contrast, Spar supports 'classical' iteration sets that can be expressed as nested iteration ranges with strides. Spar's iteration ranges are more general than the rectangular iteration sets of Titanium, and can be implemented just as efficiently.
REFERENCES
[1] B.Blount, S. Chatterjee, and M. Philippsen. Irregular parallel algorithms in Java. In Irregular'99, Apr. 1999. [2]
[3] [4] [5] [61
B. Carpenter, Y. Chang, G. Fox, D. Leskiw, and X. Li. Experiments with "HPJava". Concurrency, Practice and Experience, 9(6):633-648, June 1997. DAS website, www.asci.tudelft.nl/das/das.shtml. M. Frumkin, H. Jin, and J. Yan. Implementation of NAS parallel benchmarks in high performance fortran. In IPPS, 1999. High Performance Fortran Forum. High Performance Fortran Language Specification, 2.0 edition, Feb. 1997. NAS parallel benchmarks website, www.nas.nasa.gov/Software/NPB.
ll8 [7] [8]
[9] [ 10]
[ 11]
[12]
C. v. Reeuwijk. Timber download page. www.pds.twi.tudelfl.nl/timber/downloading.html. C.v. Reeuwijk, W. Denissen, H. Sips, and E. Paalvast. An implementation framework for HPF distributed arrays on message-passing parallel computer systems. IEEE Transactions on Parallel and Distributed Systems, 7(9):897-914, Sept. 1996. C.v. Reeuwijk, F. Kuijlman, and H. Sips. Spar: an extension of Java for scientific computation. In ACMJava Grande- ISCOPE Conference, pages 58-67, June 2001. T. Riihl, H. Bal, R. Bhoudjang, K. Langendoen, and G. Benson. Experience with a portability layer for implementing parallel programming systems. In Int. Conf. on Parallel and Distributed Processing Techniques and Applications, pages 1477-1488, Aug. 1996. K. Yelick, G. Pike, C. Miyamoto, B. Liblit, A. Krishnamurthy, P. Hilfinger, S. Graham, D. Gay, P. C., and A. Aiken. Titanium: a high-performance Java dialect. In ACM Workshop on Java for High-Performance Network Computing, pages 1-13, Feb. 1998. G. Zhang, B. Carpenter, G. Fox, X. Li, and Y. Wen. Considerations in HPJava language design and implementation. In Proc. of the l lth International Workshop on Languages and Compilers for Parallel Computing, pages 18-33. Springer, Aug. 1998.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
119
O n T h e U s e o f J a v a A r r a y s for S p a r s e M a t r i x C o m p u t a t i o n s G. Gundersen a and T. Steihaug ~ aDepartment of Informatics, University of Bergen, Norway [email protected], [email protected] Each element in the outermost array of a two dimensional native Java array is an object reference while the inner is an array of primitive elements. Each inner array can have its own size. In this paper it is shown how to utilize this flexibility of native Java arrays for sparse matrix computations. We discuss the use of Java arrays for storing matrices and particularly two different storage schemes for sparse matrices: the Compressed Row Storage and a profile or skyline storage structure and the implementations in Java. The storage schemes for sparse matrices are discussed on the basis of performance, memory and flexibility on several basic algorithms. Numerical results show that the efficiency is not lost using the more flexible data structures compared to classical data structure for sparse matrices. This flexibility can be utilized for high performance computing (HPC). 1. I N T R O D U C T I O N Java's considerable impact implies that it will be used for high performance computing and Java is already introduced as programming languages in introductory courses in scientific computation [ 1]. Matrix computation is a large and important area in scientific computation. Developing efficient algorithms for working with matrices are of considerable practical interest. Java's native arrays can be seen as a hybrid between an object and a primitive handled with references. When creating an array of primitive elements, the array holds the actual values for those elements. An array of objects stores references to the actual objects. Since arrays are handled through references, an array element may refer to another array thus creating a multidimensional array. A rectangular array of numbers (matrix) as shown in Figure 1 is implemented as Figure 2. Since each element in the outermost array of a multidimensional array is an object reference, arrays need not be rectangular and each inner array can have its own size as in Figure 3 creating a jagged form. When an object is created and gets heap allocated, the object can be placed anywhere in the memory. This implies that the elements d o u b l e [] of a d o u b l e [] [] may be scattered throughout the memory space. Non-contiguous memory access is slower than contiguous ones in any language on a a cache based memory due to reduced spatial locality. Java's native arrays have not been regarded efficient enough for high performance computing. Replacing the native arrays with multidimensional array [2], and extensions like Spar [3] have been suggested. However, packages like JAMA [4], JAMPACK [5] and Colt [6] use the native arrays as the underlying storage format. Investigating how to utilize the flexibility of Java arrays for creating elegant and efficient algorithms is of great practical interest.
120
II[ll Illll IIIII IIIII IIIII Figure 1. A t r u e 2D array.
II I II
Figure 2. A 2D Java array.
Figure 3. General Java array.
We discuss the use of native Java arrays for storing matrices and particularly two different storage schemes for sparse matrices. The implementation of Compressed Row Storage as Java Sparse Arrays [ 15], and a profile or skyline structure introduced in this paper. The compressed row format and profile format have a static implementation frequently used in languages like C and Fortran, and a dynamic implementation as a jagged array that can be used in languages like Java and c#. For operations like matrix vector and vector matrix multiplications we show that there is no loss in efficiency using the more dynamic and flexible data structure created by jagged arrays for storing sparse matrices compared to the more static format. For operations like matrix multiplication, matrix updates, and matrix factorization where back-fill or fill-ins are created we demonstrate that there is a gain in efficiency using the more dynamic and flexible data structure. The timings for the matrix operations where done on Red Hat Linux with Sun's Java Development Kit (JDK) 1.4.2, processor is Intel Pentium 4, processor speed is 2.26GHz and memory is 500MB. The time is measured in milliseconds (mS) and is an average of three trials.
2. SPARSE MATRICES A sparse matrix is usually described as a matrix where "many" of its elements are equal to zero and we benefit both in time and space by working only on the nonzero elements [7]. The difficulty is that sparse data structures include more overhead since indexes as well as numerical values of nonzero matrix entries are stored. There are several different storage schemes for large and unstructured sparse matrices that are used in languages like Fortran, C and C++. These storage schemes have enjoyed several decades of research and the most commonly used storage schemes for large sparse matrices are the compressed row storage [7, 8, 9, 10]. There are few released packages or separate algorithms implemented in Java for numerical computation on sparse matrices [6, 11, 12, 13, 15].
2.1. Compressed Row Storage scheme The Compressed Row Storage (CRS) format puts the subsequent nonzeros of the matrix rows in consecutive locations. For a sparse matrix we create three vectors" one for the double type ( v a l u e ) and the other two for integers ( c o l u m n i n d e x , r o w p o i n t e r ) . The v a l u e vector stores the values of the nonzero elements of the matrix, as they are traversed in a row-wise fashion. The c o l u m n i n d e x vector stores the column indexes of the elements in the v a l u e vector. The r o w p o i n t e r vector stores the locations in the v a l u e vector that start a row. If v a l u e
[k] - A # then c o l u m n i n d e x
[k] - j and r o w p o i n t e r [ / ]
__ k <
121 rowpointer [i + 1]. By convention r o w p o i n t e r [m] = n n z , where n n z is the number of nonzero elements in the matrix and m is the number of rows. Consider the sparse 6 x 6 matrix A which is a permutation of a matrix in [ 14]. To be consistent with Java first row and column index is 0. -1 13 0-2 3 0 0
A-
2 9 0 5 0
0 4 0 0 8 9 10 0 0 3 9 0 0 3 0 7 8 0 7 7
0 0 0 "
(1)
8
The nonzero structure of the matrix (1) stored in the CRS scheme:
double[] v a l u e = { - I , 2 , 4 , 1 3 , 9 , 8 , 9 , - 2 , 1 0 , 3 , 3 , 9 , 5 , 3 , 7 , 8 , 7 , 7 , 8 } ; int[] c o l u m n i n d e x : {0,1,3,0,1,3,4,1,2,0,2,3,1,2,4,5,3,4,5}; int [] r o w p o i n t e r = { 0 , 3 , 7 , 9 , 1 2 , 1 6 , 1 9 } ;
2.2. Java Sparse Array The Java Sparse Array format (JSA) introduced in [15] is a unique implementation in languages that supports jagged arrays. JSA takes advantage of the feature that Java's native arrays are objects. This concept is illustrated in Figure 4 and uses an array of arrays. There are two arrays, one for storing the references to the value arrays and one for storing the references to the index arrays (one for each row). With the JSA format it is possible to manipulate the rows independently without updating the rest of the structure as would have been necessary with CRS. Each row consist of a value and an index array each with its own unique reference. To store an m - n matrix, JSA use 2nr~z + 2 m storage locations compared to 2 n n z + m + 1 for the CRS format. A J a v a S p a r s e A r r a y "skeleton" class can look like this.
public class JavaSparseArray{ p r i v a t e double[] [] A v a l u e ; p r i v a t e int [] [] A i n d e x ; public JavaSparseArray(double[] this.Avalue = Avalue; this.Aindex = Aindex;
}
public JavaSparseArray
[] A v a l u e ,
int [] [] Aindex) {
times(JavaSparseArray
B) {...}
In this JavaSparseArray class, there are two instance variables, double [] [] and i n t [] []. For each row these arrays are used to store the actual values and the column indexes. The nonzero structure of matrix (1) is stored as follows in JSA.
double[] [] Avalue : {{-1,2,4},{13,9,8,9},{-2,10},{3,3,9},{5,3,7,8},{7,7,8}}; int[] [] Aindex = { {0,1,3}, {0,1,3,4}, { i, 2}, {0,2,3}, {1,2,4,5}, {3,4,5}};
122 int [ ] [ ] Aindex
JavaSparseArray
"-----~
liui
Value Array
Figure 4. Java Sparse Array format.
Figure 5. Jagged Variable Band Storage and Non Symmetric Skyline format.
2.3. Variable Band Storage The Variable Band Storage (VBS) is a row oriented storage scheme defined by the semibandwidths of the matrix. For each row i define upper ui and lower li semibandwidth
li = maxj i - j for nonzero Aij, ui = metxj j - i for nonzero Aij. In the VBS data structure, all matrix elements from the first nonzero in each row to the last nonzero in the row are explicitly stored. >From the semibandwidths ui and li the column indexes are i - li <_ j < i + ui. The number of nonzero elements of an m x n matrix is m--1 Y~'~i=o (li + ui) + m. For rows with only elements in the upper triangular part the corresponding l~ will be negative. The matrix (1) can be stored as double[] v a l u e = { - 1 , 2 , 0 , 4 , 1 3 , 9 , 0 , 8 , 9 , - 2 , 1 0 , 3 , 0 , 3 , 9 , 5 , 3 , 0 , 7 , 8 , 7 , 7 , 8 } ; int[] r o w p o i n t e r = { 0 , 4 , 9 , 1 1 , 1 5 , 2 0 , 2 3 } ; int[] f i r s t c o l u m n i n d e x = {0,0,i,0,i,3};
in a compressed format and in a jagged VBS format as double[] [] Avalue = {{-1,2,0,4},{13,9,0,8,9},{-2,10},{3,0,3,9},{5,3,0,7,8}, int[]
firstcolumnindex
{v,7,8}};
= {0,0,I,0,i,3};
In the VBS in a compressed format the columnindexes of A in row i are firstcolumnindex[i] _<j < firstcolumnindex[i] +rowpointer[i+l] -rowpointer[i],
and in the jagged VBS format firstcolumnindex
[i] _< j < f i r s t c o l u m n i n d e x
[i] + A v a l u e
[i] . l e n g t h .
123 Table 1 The algorithms for c = Ab and c T - bTA. The time is in milliseconds and the measuring is done on Red Hat Linux with Sun's JDK 1.4.2. Sparse Matrix Vector and Vector Matrix Multiplication Type GRE 216A GRE 216B GRE 343 GRE 512 SHERMAN SHERMAN SHERMAN SHERMAN SHERMAN WATT 1 WATT 2
1 2 3 4 5
m - n
nnz(A)
216 216 343 512 1000 1080 5005 1104 3312 1856 1856
876 876 1435 2192 3750 23094 20033 3786 20793 11360 11550
CRS Ab bTA 0 0 0
0 0 0 1 1 5 5 1 5 3 3
JSA Ab [ bTA 0 0 0 0 1 6 5 1 6 3 3
0 0 0 1 1 5 5 0 5 2 2
VBS Ab bTA
NSK Ab bTA
1 1 4 5 4 8 29 7 31 6 7
1 1 3 6 9 12 29 10 32 10 10
1 1 4 4 4 6 19 6 19 6 6
1 2 4 6 9 11 29 10 29 10 10
2.4. The Non Symmetric Skyline format The Non Symmetric Skyline format (NSK) is a standard way of storing variable band matrices [ 14, 16] for direct methods. A nonsymmetric skyline matrix or profile matrix has nonzero elements clustered around the main diagonal. Let li = maxj<_i i - j for nonzero Aij, ui = maxj
The column indexes for the lower part in row i are f i r s t c o l u m n i n d e x [ { ] the row indexes in column j are 2j - f i r s t c o l u m n i n d e x [j] - A v a l u e i<j.
i j i i while [j] . l e n g t h _<
2.5. Numerical results We have investigated VBS and NSK for storing banded matrices in Java. These two formats are compared on the basis of memory, efficiency and flexibility. We have only used square matrices but there is no limitations in concept or implementation using matrices where m --fin for JSA and VBS. We tested the formats on matrix vector multiplication (fight and left multiplication), and matrix multiplication. The sparse matrices used in Table 1 are from Matrix Market
124 [ 17] with banded features and for matrix-vector and vector-matrix operations a vector was full. The sparse pentadiagonal matrices in Table 2 are generated by the authors. There were only minor time differences between VBS and NSK on these operations, as shown in Table 1. VBS was slightly more efficient. The memory consumption between these two formats are nnz + 2n for VBS and nnz + 3n for NSK. So there is no loss in introducing VBS as a storage format for banded matrices in Java. We also compared VBS to CRS and JSA on the same operations, as shown in Table 1 and Table 2. Both JSA and CRS was significantly more efficient than VBS and NSK this is due to the fact that CRS and JSA does not store any nonzero elements as VBS and NSK does.
Table 2 The Matrix Multiplication algorithms for C = A 2. The time is in milliseconds and the measuring is done on Red Hat Linux with Sun's JDK 1.4.2. Sparse Matrix Multiplication I ?TZ=n
100 200 300 500 1000 5000 10000 15000 17000 20000
nnz(A) 493 993 1493 2493 4993 24993 44993 74993 84993 99993
nnz(C) 876 1776 2676 4476 8976 44976 89976 134976 179976 359976
JSA 1 2 3 9 10 19 47 57 77 97
VBS 0 1 1 7 7 16 31 43 48 87
In Table 2 used pentadiagonal matrices, that is three elements below the diagonal and one element above the diagonal. The diagonal elements were all non zero. Then we performed C = A 2, it is shown that VBS is competitive with JSA on Matrix Multiplication with matrices that has structure were we traverse only nonzero elements in the matrices involved. VBS was an average of 1.33 more efficient than JSA. This difference is partially explained by the fact that JSA gets indexes from a heap allocated arrays and uses extensive copying from temporary arrays to resulting arrays. When m = n increases we see that the difference between JSA and VBS is getting smaller, this can be explained by the cost of accessing i n t [ ] [] when the arrays gets large. This effect can maybe be removed by introducing two i n t [] arrays, one for li and one for ui. The performance improves if one-dimensional arrays are accessed instead of two-dimensional arrays. This flexibility (independently row updating) is especially useful for operations that calls for partial updating of the data structure where we with CRS and NSK (lower or upper) have to traverse the whole nonzero structure with extensive copying as shown in [15].
125 3. C O N C L U D I N G R E M A R K S
The VBS format has greater flexibility than NSK since the rows can be manipulated independently of the rest of the structure as we can with the classical data structure JSA. They also have the same memory consumption as NSK and where slightly more efficient on our test problems. But if the bands are mostly nonzero (dense) VBS may even compete with JSA and CRS for the test problems in efficiency, this since we do not have to get index elements from the memory. JSA can be extended to dense blocks in a similar way as CRS is extended to blocks[ 14]. Dense bands in VBS might be extended to a block matrix implementation and operations.
REFERENCES
[1 ]
[2]
[3]
[4]
[5] [6] [7] [8] [9] [ 10]
[ 11 ] [12]
[ 13] [ 14]
C.H. Bischof, H. M. Bticker, J. Henrichs, and B. Lang. Hands-On Training for Undergraduates in High-Performance Computing Using Java. In Applied Parallel Computing: New Paradigms for HPC in Industry and Academia, T. Sorevik, F. Manne, R. Moe, and A. H. Gebremedhin (eds), Proceedings of the 5th International Workshop, PARA 2000, Bergen, Norway, June 2000, pages 306-315, Lecture Notes in Computer Science, Volume 1947, Springer Verlag, 2001. Jos6 E. Moreira, Samuel P. Midkiff, and Manish Gupta. Supporting Multidimensional Arrays in Java. Concurrency and Computation: Practice and Experience, Volume 15, Issue 3-5, pages 317-340, 2003. Kees van Reeuwijk, Frits Kuijlman, and Henk J. Sips: Spar: a set of extensions Java for scientific computation. Concurrency and Computation: Practice and Experience, Volume 15, Issue 3-5, pages 277-299, 2003. Joe Hicklin, Cleve Moler, Peter Webb, Ronald F. Boisvert, Bruce Miller, Roldan Pozo, and Karin Remington. JAMA: A Java Matrix Package. June 1999. http://math.nist.gov/javanumerics/jama. G.W. Stewart. JAMPACK: A Package for Matrix Computations. February 1999. ftp ://math.nist. gov/pub/Jampack/Jampack/AboutJampack.html. The Colt Distribution. Open Source Libraries for High Performance Scientific and Technical Computing in Java. http://hoschek.home.cern.ch/hoschek/colt/. Sergio Pissanetzky. Sparse Matrix Technology. Academic Press, Massachusetts, 1984. Yousef Saad. Iterative Methods for Sparse linear Systems. PWS Publishing Company, Boston, 1996. I.S.Duff, A.M.Erisman, and J.K.Reid. Direct Methods for Sparse Matrices. Oxford University Press, 1986. J. Dongarra, A. Lumsdaine, R. Pozo, K. Remington. A Sparse Matrix Library in C++ for High Performance Architectures. Proceedings of the Second Object Oriented Numerics Conference, pp. 214-218, 1994. Java Numerical Toolkit. http://math.nist.gov/jnt. Rong-Guey Chang, Cheng-Wei Chen, Tyng-Ruey Chuang, and Jenq Kuen Lee. Towards Automatic Support of Parallel Sparse Computation in Java with Continuous Compilation. Concurrency: Practice and Experience, 9(1997) 1101-1111. SciMark 2.0. http://math.nist.gov/scimark2/about.html. R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo,
126 C. Romine, and H. van der Vorst. Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods. SIAM Publications, Philadelphia, 1994. [ 15] Geir Gundersen and Trond Steihaug. Data Structures in Java for Matrix Computation. To appear in Concurrency and Computation: Practice and Experience, 2003. [ 16] Yousef Saad. SPARSKIT: A basic tool kit for sparse matrix computations, Version 2. Technical Report, Computer Science Department, University of Minnesota, June 1994. [ 17] Matrix Market. http://math.nist.gov/MatrixMarket.
Parallel Computing: SoftwareTechnology,Algorithms,Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
127
A Calculus of Functional BSP Programs with Explicit Substitution F. Loulergue ~ ~Laboratory of Algorithms, Complexity and Logic, 61, avenue du G6n6ral de Gaulle, 94010 Cr6teil cedex, France The BSA~-calculus is a calculus of functional Bulk Synchronous Parallel (BSP) programs with enumerated parallel vectors and explicit substitution. This calculus is defined and proved confluent. It constitutes the core of a formal design for a Bulk Synchronous Parallel dialect of ML (BSML) as well as a framework for proving parallel abstract machines which can evaluate BSML programs. 1. INTRODUCTION Bulk Synchronous Parallel ML or BSML is an extension of ML for programming direct-mode parallel Bulk Synchronous Parallel algorithms as functional programs. Bulk-Synchronous Parallel (BSP) computing is a parallel programming model introduced by Valiant [24] to offer a high degree of abstraction like PRAM models and yet allow portable and predictable performance on a wide variety of architectures. A BSP algorithm is said to be in direct mode [10] when its physical process structure is made explicit. Such algorithms offer predictable and scalable performances and BSML expresses them with a small set of primitives taken from the confluent BSA calculus [17]: a constructor of parallel vectors, asynchronous parallel function application, synchronous global communications and a synchronous global conditional. Our B S M L I ib library implements the BSML primitives using Objective Caml [ 15] and MPI [23]. It is efficient [ 16] and its performance follows curves predicted by the BSP cost model (the cost model estimates parallel execution times). This library is used as the basis for the C ARAML project, which aims to use Objective Carol for Grid computing with, for example, applications to parallel databases and molecular simulation. In such a context, security is an important issue, but in order to obtain security, safety must be first achieved. An abstract machine was used for the implementation of Caml and is particular easy to prove correct w.r.t, the dynamic semantics [11 ] and the proof can be done using the Coq Proof Assistant [2]. In order to have both simple implementation and cost model that follows the BSP model, nesting of parallel vectors is not allowed. BSML1 i b being a library, the programmer is responsible for this absence of nesting. This breaks the safety of our environment. A polymorphic type system and a type inference has been designed and proved correct w.r.t, a small-steps semantics. It will be included in a full BSML compiler. A parallel abstract machine [19] (and a compilation scheme) for the execution of BSML programs has been designed and proved correct w.r.t, the BSA-calculus [ 17], using an intermediate semantics. Another abstract machine [ 18] has been designed but those machines are not adapted for grid computing and security because the compilation schemes need the static number of pro-
128 cesses (this is not possible for Grid computing) and some instructions are not realistic for real code and a real implementation. [8] is based on the Zinc Abstract Machine (or ZAM) and has not the drawbacks of the previous machines. [ 11 ] used the A,,-calculus - a A-calculus with explicit substitution- to prove several abstract machines for the evaluation of functional languages: the Krivine machine, the SECD machine [13], the Categorical Abstract Machine (CAM) [4], the Functional Abstract Machine (FAM) [3] and it has also been used to prove the correction of the Zinc Abstract Machine (ZAM) [ 14]. For each machine, the corresponding reduction strategy of the Ao-calculus is given. We are particularly interested by the CAM and the ZAM because they are respectively the abstract machine of implementations of Caml and Objective-Caml [ 15, 21 ] (and also Caml-light). In order to ease the proof of correctness of the parallel abstract machines we designed, we need a version of the BSA-calculus with explicit substitution. This paper is about such a calculus. We first presents the BSP model and the A~-calculus (section 2). We then define the BSA~calculus (section 3) which is confluent. We end with conclusions and future work (section 4). 2. P R E L I M I N A R I E S 2.1. The Bulk Synchronous Parallel model Bulk-Synchronous Parallel (BSP) computing is a parallel programming model introduced by Valiant [24, 22] to offer a high degree of abstraction like PRAM models and yet allow portable and predictable performance on a wide variety of architectures. A BSP computer contains a set of processor-memory pairs, a communication network allowing inter-processor delivery of messages and a global synchronization unit which executes collective requests for a synchronization barrier. Its performance is characterized by 3 parameters expressed as multiples of the local processing speed: the number of processor-memory pairs p, the time I required for a global synchronization the time 9 for collectively delivering a 1-relation (communication phase where every processor receives/sends at most one word). The network can deliver an h-relation in time 9h for any arity h. A BSP program is executed as a sequence of super-steps, each one divided into (at most) three successive and logically disjoint phases. In the first phase each processor uses its local data (only) to perform sequential computations and to request data transfers to/from other nodes. In the second phase the network delivers the requested data transfers and in the third phase a global synchronization barrier occurs, making the transferred data available for the next superstep. The execution time of a super-step s is thus the sum of the maximal local processing time, of the data delivery time and of the global synchronization time:
Time(s) where ~69U i(s)
-
max
i:processor
(s)
,wi
+
max
i:processor
(s)
hi
9g + 1
local processing time on processor i during super-step s and hl s)
-
where 'h~ i(s) (resp hl s)) is the number of words transmitted (resp. received) by processor i + during super-step s. The execution time ~ s Time(s) of a BSP program composed of S superA i(s) and H = steps is therefore a sum of 3 terms:W + H 99 + S 9 l where W - ~ s maxi w ~--~smaxi hl s). In general W, H and S are functions of p and of the size of data n, or of more complex parameters like data skew and histogram sizes. To minimize execution time the BSP
129 algorithm design must jointly minimize the number S of super-steps and the total volume h (resp. W) and imbalance h (s) (resp. W (s)) of communication (resp. local computation). 2.2. The A~#-calculus The reader is assumed to have basic knowledge of the implementation of functional languages [5] and of A-calculus [ 1]. The A~,-calculus is a A calculus of explicit substitution which is confluent [6] (for every term and not only closed ones). The sort of terms is given by the following grammar : ct
::=
I
Xt
Aa
term meta-variable Abstraction
n
a[s]
De Bruijn indice Closure
I aa
Application
xt can be considered as a "hole" that can be filled by a term, but this kind a variable cannot be bound. Terms containing such variables correspond much to contexts in usual A-calculus. De Bruijn indices [7] are used instead of variable names. This method was used to avoid the burden of a-conversion (renaming of bound variables in order to avoid the capture of free variables during/3-reduction). They replace a given occurrence of a variable name by the number of A-abstraction which are included between the occurrence and the A-abstraction that binds it. Application and abstraction are the same than application and abstraction in A-calculus. The last kind of term corresponds to the notion of closure found in the implementations of functional languages: a is the "body" of the function and s is an environment collecting the values of free variables. In the formalism presented here, s is called a substitution. The substitutions are given by the following grammar: s ::= xs [ id 1 T I a . s ] s o s ] zs is a substitution meta-variable. There are two special substitutions: id and T. A substitution is either of list of terms a . s or a composition of two substitutions s o s or a "lifted" substitution that is a substitution which was propagated under an abstraction. The rules of the A~ are given in figure 1. About half of the rules have no intuitive meaning but are part of the calculus in order to make it confluent. (Beta) corresponds to the creation of a closure from the application of an abstraction to another term. Rule (App) is for the propagation of the substitutions through application. Rules (FVarCons) and (RVarCons) show how the value of a variable (here a De Bruijn index) is retrieved from the a substitution. The application of a substitution s' to a term Aa[s] will result in applying s' to s and leading to a new substitution s' o s, by rule (Clos). Other rules are used to compose substitution: (AssEnv), (MapEnv), (ShiftCons) and (IdL). Rule (Lambda) is for the propagation through an abstraction and uses the "lifted" substitution construct. Rules (IdR) and (Id) define id as the identity substitution. The other rules make the adjustments of De Bruijn indices explicit.
3. THE BSA~,-CALCULUS The BSA~-calculus is a calculus for functional Bulk Synchronous Parallel programs. It is composed by the A~-calculus and a term rewriting system [12] (TRS) to express bulk synchronous parallelism.
130 (A a ) b - - , a[b. id]
(Beta)
(ab)[s] --~ (a[s])(b[s])
(App) (Lambda)
A a [ s ] - ~ A(a[~ (s)]) (a[s])[t] ~
(Clos)
a[s o t]
n T --~ n+l n[T os] ~ n+l[s] l[a.s] --~ a
(VarShiftl) (VarShift2)
n+l[~ (s)] -+ n[so T]
(RVarliftl)
n+l[~ (s) o t] -+ n[s o (T ot)] (RVarlift2) T o(a. s) ~ s (ShiflCons) T o ~ (s) -+ so T
(ShiflLifll)
T o(~ (s) o t) --+ s o (T or) ~ (s)o ~ (t) - ~ (s o t)
(ShiflLift2) (Liftl)
(FVarCons)~ (s) o (~ (t) o u) -+~ (s o t) o u
(Lift2)
1[~ (s)] --~ 1 1[~ (s) o t] --~ l[t]
(FVarliftl) (FVarlift2)
~ (s) o (a. t) -+ a - ( s o t) id o s -+ s
(LiflEnv) (IdL)
n + l [ a . s ] - + n[s]
(RVarCons)
s o id -+ s
(IdR)
(s o t) o u --+ s o (t o u) ( a . s) o t --~ a[t] . (s o t)
(AssEnv) (MapEnv)
~ (id) -+ id a[id] --~ a
(Liftld) (Id)
Figure 1. The Ao~-calculus
3.1. The term rewriting system BS BS is a term rewriting system which will be used to extend the A~-calculus. This system introduces operations for data-parallel programming but with explicit processes in the spirit of BSP. We now describe its syntax and its reduction with operational motivations. The terms are defined by the following grammar: a "'= c [ | a . . . a I (a --~ a, a) usual operations [ a # a pointwise parallel application [ (a,..., a,..., a) parallel vector [ a ? a get operation [ (a --% a, a) global conditional c is an integer or boolean constant, | C { + , - , • or, and, <, > , = - , n o t } . The integer constants n must not be mixed up with the De Bruijn indices n which are used instead of usual variables. The principal terms represent parallel vectors i.e. tuples of p local values where the i th value is located at the processor with rank i. The width p is fixed, so there is one set of rule for each value ofp. The notation is: ( ao , . . . , ap-1 ) where a o , . . . , ap_l are terms. The forms al # a2 and al ? a2 are called point-wise parallel application and get respectively. Point-wise parallel application represents point-wise application of a vector of functions to a vector of values, i.e. the pure computation phase of a BSP super-step. Get represents the communication phase of a BSP super-step: a collective data exchange with a barrier synchronization. In a l ? a2, the resulting vector field contains values from al taken at processor names defined in a2. The exact meanings of point-wise parallel application and get are defined by the BS rules. The last form of global terms define synchronous conditional expressions. The meaning of (at --~ a2, a3) (not to be confused with (al --~ a2, a3)) is that of a2 (resp. a3) if the vector denoted by a l has value t r u e (resp. false) at the processor name denoted by n.
131 true
a,2)
> a I
(false ~ al, a2)
> a2
---+ a l ,
(ItTrue) (IfFalse)
(ao bo,..., ap-1 bp-1) ' { ano , . . . , anp_, )
(< a , . . . , ap, Z ' " "
(ParApp) (Get)
a>
>
al
(IfAtTrue)
a ) & a l , a2)
>
a2
(IfAtFalse)
i
({ a , . . . ,
fa~,..., i
(Get) is not a rule but a set of rules such that for all i in { 0 , . . . , p - 1}, ni is integer constant between 0 and p - 1.
Figure 2. Rules of the term rewriting system BS
The rules for usual operators are defined by rules like : 3 + 2 > 5 or (true or false) true. The equality is only defined on integer and boolean constants. The value of al ? a2 at processor name i is the value of el at processor name given by the value of e2 at i. Notice that, in practical terms, this represents an operation whereby every processor receives one and only one value from one and only one other processor. This restriction can be lifted by defining a "put" operation (which is also a pure functional operation) but which is not given here for the sake of conciseness. Next, the global conditional is defined by two rules. The above two cases generate the following bulk-synchronous computation: first a pure computation phase where all processors evaluate the local term yielding to n; then processor n evaluates the parallel vector ofbooleans; if at processor n the value is t r u e (resp. false) then processor n broadcasts the order for global evaluation of a 1 (resp. a2); otherwise the computation fails. Those two rules are necessary to express algorithms of the form: Repeat Parallel
Iteration
UntilMax
of
local
errors
< e
because without them, the global control can not take into account data computed locally, ie global control can not depend on data. Additional rules bare also needed to propagate substitutions through the symbols of BS. For every symbol f of BS with arity n we have the rule : f(al,...,an)[s]
, f(al[s],...,an[s])
If n = 0 this rule means
f[s]
,
f.
3.2. Confluence of the calculus We will use the follow theorem due to Pagano [20] :
(f#)
132 Theorem 1. Let be (F, TO) a term rewriting system such that : 9F
and A~ (the set of terms of the Ao#) share at most the application symbol
9 T~ is
confluent
9 T~ is left
linear
9T~ does not contain variable-applicante rules
Then the AF~9-calculus is confluent R e m a r k 1. If F and A# do not share the application symbol then the last condition is not required. In our case (F, 7~) = BS. There are no critical pairs. Rules are left linear. The TRS BS is confluent. By theorem 1, the BSA~9-calculus is confluent. 3.3. Examples The first example shows that parallel vector can be also expressed by an intentional construction as in the BSA-calculus [17]. The 7r or parallel vector constructor of BSA can be defined as : =
A vector usually used is t h i s defined as: 7r(A1). It can be reduced to (using the rules indicated on the fight): this _= (A( 1 0 , , 1 ( p - I) ))AI > < 10,, l ( p - 1 ) ) ) [ A 1 id] >P ( ( 1 0 ) [ A a - i d ] , . . . , (1 ( p - 1))[A1. id] >
>P ( l [ A l . i d ] O [ A l . i d ] , . . . , a [ A l . i d ] ( p - 1 ) [ A l . i d ] ) >P < A1 (O[A1. i d ] ) , . . . , AI ( ( p - 1)[Al. id]) > >P ( .)kl 0 , . . . , A1 (p - 1) ) >P < 1[0. i d ] , . . . , i [ ( p - 1). id] ) >P < 0 , . . . , p - l >
(Beta)
(f~) (App) (FVarCons)
(f#) (Beta) (FVarCons)
The second one is the direct broadcast algorithm which broadcasts the value held at processor given as first argument: b c a s t : = AA1 ? ~-(A2). If applied to 1 (the root of the broadcast) and to an expression e which evaluates to a parallel vector ( a0 , . . . , ap_l ) it can be reduced as follows (we omit the steps similar to the previous example): bcast
1e
>* < a0 , . . . , ap_ 1 > .7 < 1 , . . . ,
1,...,
1 ) by rule (Get) < a 1 , - . - ,
a 1,-..,
al >
4. CONCLUSIONS AND FUTURE WORK
The BSAo~-calculus is a confluent calculus for functional bulk synchronous parallel programs. Being an extension of the A~-calculus it has its advantages such as to be closer to functional languages implementations than our BSA-calculus [ 17]. Another interesting feature of this calculus is the possibility to express weak reduction strategy (ie no reduction under a
133 A-abstraction) by removing rules of the calculus. This was not possible in the A-calculus: removing context rules which allow the reduction under a A-abstraction leads to a non confluent calculus. Thus the ease of expressing reduction strategies used in real functional languages allow to prove the correctness of abstract machines used in the implementations of those languages [ 11 ]. We have designed a Bulk Synchronous Parallel ZINC Abstract Machine [8] (BSP ZAM) which is an extension of the Zinc Abstract Machine used in the Objective Caml implementation The next phases of the project will be: the proof of correctness of this machine with respect to the BSA~-calculus, and the parallel implementation of this abstract machine. This BSP ZAM implementation will be the basis of a parallel programming environment developed from the Caml-light language and environment. It will include our type inference [9] and will thus provide a very safe parallel programming environment.
REFERENCES
[ 1] [2] [3]
[4] [5] [6] [7]
[8]
[9]
[ 10] [11 ] [12]
[13]
H. R Barendregt. Functional programming and lambda calculus. In J. Van Leeuwen, editor, Handbook of Theoretical Computer Science (vol. B), pages 321-364. Elsevier, 1990. S. Boutin. Proving correctness of the translation from mini-ml to the cam with the coq proof development system. Technical Report 2536, INRIA, 1995. L. Cardelli. Compiling a functional language. In Conference Record of the 1984 ACM Symposium on Lisp and Functional Programming, pages 208-217, Austin, Texas, August 1984. ACM. G. Cousineau, P.-L. Curien, and M. Mauny. The categorical abstract machine. Science of Computer Programming, 8:173-202, 1987. G. Cousineau and M. Mauny. The Functional Approach to Programming. Cambridge University Press, 1998. P.-L. Curien, T. Hardin, and J.-J. Ldvy. Confluence properties of weak and strong calculi of explicit substitutions. Journal of the ACM, 1996. N.G. De Bruijn. Lambda-calculus notation with nameless dummies, a tool for automatic formula manipulation, whith application to the Church-Rosser theorem. Indag. Math., 34:381-392, 1972. F. Gava and F. Loulergue. A Parallel Virtual Machine for Bulk Synchronous Parallel ML. In Peter M. A. Sloot and al., editors, International Conference on Computational Science (ICCS 2003), Part I, number 2657 in LNCS. Springer Verlag, june 2003. F. Gava and F. Loulergue. A Polymorphic Type System for Bulk Synchronous Parallel ML. In Seventh International Conference on Parallel Computing Technologies (PACT 2003), LNCS. Springer Verlag, 2003. A.V. Gerbessiotis and L. G. Valiant. Direct Bulk-Synchronous Parallel Algorithms. Journal of Parallel and Distributed Computing, 22:251-267, 1994. T. Hardin, L. Maranget, and L. Pagano. Functional runtime systems within the lambdasigma calculus. Journal of Functional Programming, 8(2): 131-176, 1998. Jan Willem Klop. Term rewriting systems. In S. Abramsky, D. M. Gabbay, and T. S. E. Maibaum, editors, Handbook of Logic in Computer Science, volume 2, chapter 1, pages 1-117. Oxford University Press, Oxford, 1992. P. J. Landin. The mechanical evaluation of expressions. The Computer Journal, 4(6):308-
134
[14] [ 15] [16]
[17] [18] [19]
[20] [21]
[22] [23] [24]
320, 1964. X. Leroy. The ZINC experiment: An economical implementation of the ML language. Rapport Technique 117, 1991. Xavier Leroy. The Objective Caml System 3.07, 2003. web pages at www.ocaml.org. F. Loulergue. Implementation of a Functional Bulk Synchronous Parallel Programming Library. In 14th IASTED International Conference on Parallel and Distributed Computing Systems, pages 452-457. ACTA Press, 2002. F. Loulergue, G. Hains, and C. Foisy. A Calculus of Functional BSP Programs. Science of Computer Programming, 37(1-3):253-277, 2000. A. Merlin and G. Hains. La Machine Abstraite Cat6gorique BSP. In Journdes Francophones des Langages Applicatifs. INRIA, 2002. A. Merlin, G. Hains, and F. Loulergue. A SPMD Environment Machine for Functional BSP Programs. In Proceedings of the Third Scottish Functional Programming Workshop, august 2001. Bruno Pagano. Des calculs de substitution explicite et de leur application g~la compilation des langagesfonctionnel. PhD thesis, universit6 Pierre et Marie Curie, 1997. D. R6my. Using, Understanding, and Unravellling the OCaml Language. In G. Barthe, E Dyjber, L. Pinto, and J. Saraiva, editors, Applied Semantics, number 2395 in LNCS, pages 413-536. Springer, 2002. D. B. Skillicorn, J. M. D. Hill, and W. F. McColl. Questions and Answers about BSP. Scientific Programming, 6(3):249-274, 1997. M. Snir and W. Gropp. MPI the Complete Reference. MIT Press, 1998. Leslie G Valiant. A bridging model for parallel computation. Communications of the ACM, 33(8):103, August 1990.
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 Elsevier B.V. All rights reserved.
135
JToe: a Java* API for Object Exchange S. Chaumette a, R Grange a, B. Mftrot a, and E Vignfras ~ a Laboratoire Bordelais de Recherche en Informatique, Universit6 Bordeaux 1, 351, cours de la Libfration, 33405 Talence Cedex, France. email: {Serge.Chaumette, Pascal.Grange, Benoit.Metrot, Pierre.Vigneras}@labri.fr This paper presents JToe, an API dedicated to the exchange of Java objects in the context of high performance computing. Even though Java RMI provides a good framework for distributed objects in general, it is known to be quite inefficient, mainly due to the Java serialization process. Many projects have already improved RMI either by redesigning and reimplementing it or by reimplementing the serialization process. We claim that both approaches are missing a clear and high level API for the exchange of objects. JToe proposes a new simple API that focuses on the exchange of objects. This API is flexible enough to allow, for instance, direct copy of byte streams representation of JVM objects over a specialized transport layer such as Myrinet or LAPI. Remote method invocation frameworks such as Java RMI can then be implemented over JToe with good performance enhancement perspectives. 1. INTRODUCTION Whereas remote procedure call and remote method invocation have given developers a good abstraction of the underlying transport layers for client/server communication, high performance computing has not adopted these programming models - message passing interface is still preferred- mainly because of the performance penalty they suffer from. It is acknowledged that the main drawback of RMI is its inefficiency for object serialization and de-serialization. A lot of developments exist that improve this serialization mechanism in the context of high performance computing. Some of them consist in a whole RMI reimplementation [ 1, 2, 4]. In these cases, the problem of the exchange of objects over the network has to be addressed simultaneously with the problems of distributed garbage collection, remote method invocation, threads management, registration management, and so on. Our contribution consists in defining the notion of exchange of objects as simply as possible through the JToe API. This allows one to specifically address this problem separately from the other RMI challenges. Other works (or parts of previously cited works) address only the serialization problem in the context of high performance computing [ 1, 3, 9]. From a technical point of view, they generally rely [I, 3] on the already defined Obj ectOutputStream and Obj ect InputStream *Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. The authors are independent of Sun Microsystems, Inc.
136 classes. We argue in section 2 that these APIs are low level ones. What we propose is a new high level API dedicated to the exchange of Java objects in the context of high performance computing. The rest of this paper is organized as follows. In section 2 we discuss why we introduce a new API. We describe this API in section 3. Existing implementations and performances are presented in section 4. 2. A N E W API
We need an API that allows one to get all the benefits of an efficient object exchange layer that can be used to implement a distributed application or more general frameworks such as RMI. For this purpose, a widely used API already exists in Java: Object (Output/Input)Stream [5]. However we believe that, in the context of high performance computing, a new distinct API for the exchange of objects is needed. First, it is acknowledged that standard Java serialization is not efficient. Faster implementations have been proposed [ 1,2, 3, 4]. Of course, none of them relies on, neither respects, the Java serialization specification: technically speaking, one can inherit Obj e c t O u t p u t S t r e a m and still deeply break the compatibility with the standard serialization process as far as the corresponding Obj e c t I n p u t S t r e a m class is provided. However, when inheriting these classes, depending on how the new classes will be used, we sometimes need to respect the standard serialization process. For instance, how to deal with the u s e P r o t o c o l V e r s i o n method when we do not respect the standard protocol to achieve efficiency? Moreover, these classes, Obj ectOutputStream and Obj ectInputStream, may evolve with the protocol, and they have in the past. Such evolutions may lead to incompatibilities with legacy sub-classes 2. Second, we consider that Obj e c t ( O u t p u t / I n p u t ) S t r e a m is a low level API since it does not hide the stream management of object serialization. Such a stream oriented API is not well suited for transport layers that do not rely on stream based hardware or libraries [6, 7]. Of course one can inherit - and this is the common approach - Obj e c t O u t p u t S t r e a m and give a brand new implementation for non stream based hardware. Even if we believe this is unnatural and error-prone, it is not a technical issue. The real problem is that, in order to receive an object, one must use an instance of the Obj e c t I n p u t S t r e a m class and especially the r e a d O b j e c t method. This implies that a thread must be waiting on this blocking method for an object to be received. This prevents one to get the benefits of using special architectures and/or libraries that allow, for instance, one-sided initiator data-transfers [6]. For all these reasons, we claim that a new non stream oriented API has to be defined. One may argue that RMI could already be implemented to take advantage of one-sided initiator data-transfers. Nevertheless, this implies to re-implement RMI what, in turn and as stated in section 1, implies to deal with a lot of other high level challenges instead of just focusing on the exchange of objects: distributed garbage collection, method invocation, registration management, etc. This is why an object exchange oriented API has to be defined, independently of RMI. The problem of defining adapted APIs for the easy replacement of the transport layer or of the serialization process has already been addressed. However, it generally leads to an unfortunate dividing line between the serialization process and the transport layer. For instance in 2The best example being the writeObjectOverride method that allows one to define a new serialization process. It only exists sincejdk 1.2.
137 KaRMI [ 1] a notion of technology is defined. The serialization is performed by KaRMI and the result of this serialization is sent using a given transport technology. A new technology can be defined to enhance the network transfer or to provide a new type of network support. However, in some improvement approaches [3, 9], the data to be transferred may depend on the remote/local JVM and/or on the available communication mechanisms: for instance the assumption that the two communicating computers are running similar JVMs allows to directly send memory regions. It may also be the case that the transport layer makes it possible not to buffer the data to be transferred. In such a case, separating the serialization process from the transport process prevents one from providing improvements based on these characteristics. This is why we believe that this separation is cumbersome.
3. J T O E : THE API
Program 1 The JToe API
public interface Node { void copy(Serializable object)
}
throws JToeException;
public interface CopyListener { void copied(Serializable object) ;
}
From the previous observations, we defined a simple powerful API: JToe. It is mainly composed of two interfaces: Node and CopyListener (see program 1). When one wants to send a copy of an object to a remote node, he has to get the Node object 3 representing the node he wants to communicate with and then invoke the c o p y method passing it the object to send as an argument. On the server side, the JToe layer will inform the application, by means of a call-back mechanism, that a new object has arrived, using the c o p i e d method of the C o p y L i s t e n e r interface. The action of copying an object to a remote node is a one-sided action. No user thread has to wait for an object to be received. This is the responsibility of the implementation to accept and receive any new object and then signal this arrival to the application using an event driven programming model through C o p y L i s t e n e r . c o p i e d . This approach is not only easy to understand and to use, but also allows to really take advantage of one-sided communication libraries [6]. Moreover, as shown in figure 1, there is no limitation on the way the Node interface can be implemented. The c o p y method is responsible for the whole process of copying an object to another address space, that is the construction of the data to send (serialization) and the transfer of these data through a transport layer. This way, no artificial dividing line between the serialization process and the transport layer is introduced. We claim that this allows any serialization improvement approach to be implemented. 3The problem of retrieving a Node is not managed by JToe since it is not a transfer of object problem but more a registry one. General naming services may be used such as JNDI [8].
138
Application
Application cop~
copy(o)
Node JV
i-
:# i i
: i i
JVM
i
System
System
Figure 1. General JToe behavior.
4. J T O E IMPLEMENTATIONS We have already developed three implementations of JToe [10]. Two of them are totally written in Java, one relying on Java RMI, the other using TCP sockets and the standard Java serialization. These allow any JToe application to be 100% Java compatible and portable. They also serve as reference implementations for regression tests. The third is a JVM level implementation, i.e. at a level where we can directly access the memory representation of objects. The goal of this implementation of JToe is to provide high performance in clusters of homogeneous computers and homogeneous JVMs by directly sending memory data instead of going through the serialization process. This implementation uses the JikesRVM virtual machine [14]. JikesRVM allows to directly access memory and interact with the garbage collector from Java code. It is an interesting experimentation platform and allowed us to implement a prototype in a reasonable time. In this implementation our concern is to perform efficiently and interact smartly with the garbage collector. JikesRVM supports various garbage collection policies. Parts of our implementation are garbage collector dependent. This implementation mainly behaves as follows: when an object is being copied, the corresponding graph of objects is computed. Then the data of the objects of the graph are sent with zero copy 4. On the receiver side, memory is allocated in a garbage collector dependent s p a c e mainly the n u r s e r y - and data are directly written into this area. The pointers are then updated to reflect the original structure. Figure 2 shows the performances of the JikesRVM specific implementation of JToe compared to the 100% Java implementation. Both are using TCP as their communication layer. The values represented by the curves are the average time of one round-trip for the specified object in a ping-pong like application. The RVN_RVIVl curve shows the performances of the JikesRVM dedicated JToe implementation when run with JikesRVM - note that it cannot be run with another JVM. The TCP_RVM curve shows the performances of the 100% Java implementation of JToe running with JikesRVM without any optimization. The TCP_SUN and TCP_IBM curves are for the same implementation running on, respectively, the Sun Java virtual machine version 4When we talk about zero copy we mean that our code does not perform any copy even if the actual communication layer will since we rely on TCE Future releases based on zero copy communication layers will actually achieve real zero copy.
139 120
RVM RVM
RVM_RVM TCP RVM
TCP_-RVM TCP_SUN
--
TCP-SUN -TCISIBM
--
100
._~
8o
80
8o
60
--
o~
"5 ._= =
~ 4o
40
20
200
400
0
600
0
200
U--7 I I
600
RvM_RvM --- ]
/
TCP_RVM-
/
800
1000
(b) Ping-pong with Vectors.
(a) Ping-pong with TreeSets. 120
400 size of the Vector
size of the TreeSet
TCP_SUN_ ---
-
RVM_RVM ~ ] TCP RVM -/ TCP-_SUN - - - - A
/
/
100
i
8O
~
60 0
40 0
20
0
o
200o0
40000
60000
8000o
size of the array
(c) Ping-pong with arrays of ints.
1ooooo
0
0
20000
40000
6OO00
80000
100000
size of the array
(d) Ping-pong with arrays of doubles.
Figure 2. Ping-pong performances. 1.4. l_01-b01 and the IBM Java virtual machine version 1.4.0. All the experimentations were achieved using two Linux 2.4.18 computers with Intel 1.7GHz processors and 512MB of RAM connected with Ethernet 100Mbps. We can see that, generally speaking, JikesRVM does not perform as well as the two other virtual machines for our ping-pong application. However, for the ping-pong of vectors, we see that our JikesRVM specific JToe implementation outperforms the 100% Java one by 30% to 40% when both are running on JikesRVM. This is a really promising result since it allows one to think that the same sort of enhancement may lead to same sort of performance enhancement with the other virtual machines. For the TreeSets, our implementation does not give interesting results; it performs even worse than the default serialization. Our code performs a suboptimal graph browsing to collect the objects to send where the ad hoc serialization of the j ava. u t i l . T r e e S e t class directly and linearly writes the objects it contains. We believe that this is the reason of this poor performance. Future releases of JToe for JikesRVM will improve the graph browsing algorithm by marking visited objects with JikesRVM specific techniques. Finally for arrays of int and double, our implementation totally outperforms the standard serialization on JikesRVM. The round-trip times are close to those obtained with the Sun or IBM virtual machines despite the poor performances of JikesRVM. This allows one to think that similarly optimized implementations of JToe for the Sun or IBM virtual machines
140 will lead to comparable improvements. The issue of fast transfer of objects has already been addressed in other works. Expresso [ 11 ] is a framework aimed at transferring Java objects efficiently using zero copy mechanisms. Expresso relies on the Kaffe [12] virtual machine. Expresso uses the notion of cluster to avoid graph exploration and pointer update. A cluster is a contiguous memory area where objects can be allocated. To transfer an object to a remote node, one actually transfers the cluster containing that object. All the other objects in the cluster are also transfered. With Expresso, objects must explicitly be allocated in the correct cluster to be transfered and the references between objects in different clusters are not conserved. Ibis [ 13] is a grid computing environment for Java. It defines the IPL, an API on top of which higher level distributed environments such as RMI can be implemented. Ibis differs from our work since the IPL not only defines the way objects are sent between computers but also how to access topology information, monitoring data, etc. Moreover, object serialization optimization in Ibis relies on byte code rewriting, the aim of which is to add the serialization code to classes to avoid dynamic type inspection. 5. CONCLUSION In this paper we have presented a new API which we call JToe that is dedicated to the exchange of objects between JVMs. This API makes it possible to take advantage of the knowledge we have of both the source and target JVMs and of the underlying network and the associated communication libraries. This leads to very efficient exchange of objects between JVMs. We have three implementations running. Two are 100% Java (one over RMI and one over TCP), the third is a low level one dedicated to Jikes RVM. This last implementation is still a work in progress. We are currently working to enhance the graph browsing algorithm using JikesRVM specific mechanisms. We also plan to release a LAPI and a Myrinet version of JToe. REFERENCES
[1] [2]
[3]
[4] [5] [6]
Christian Nester, Michael Philippsen and Bernhard Haumacher. A More Efficient RMI for Java. In Java Grande, pages 152-159, 1999. Jason Maassen, Rob Van Nieuwpoort, Ronald Veldema, Henri E. Bal, Thilo Kielmann, Ceriel J. H. Jacobs and Rutger F. H. Hofman. Efficient Java RMI for parallel programming. Programming Languages and Systems, 23(6):747-775, 2001. Fabian Breg and Constantine D. Polychronopoulos. Java virtual machine support for object serialization. In Proceedings of the 2001 joint ACM-ISCOPE conference on Java Grande, pages 173-180. ACM Press, 2001. Fabian Breg and Dennis Gannon. A Customizable Implementation of RMI for High Performance Computing. In Proc. of Workshop on Java for Parallel and Distributed Computing of IPPS/SPDP99, pages 733-747, 1999. Sun Microsystem. Serialization specification, http://java.sun.com/ Shah G., Nieplocha J., Mirza C.and Harrison R., Govindaraju R.K., Gildea K., DiNicola P. and Bender C. Performance and experience with LAPI: a new high-performance communication library for the IBM RS/6000 SP. In International Parallel Processing Symposium, pages 260-266, 1998.
141 [7] [8] [9]
[ 10] [11 ] [12] [13]
[ 14]
Myricom. Myrinet software, http://www.myri.com/scs/ Sun Microsystem. Java Naming and Directory Interface. http://j ava. sun. com/products/jndi/ K. Kono and T. Masuda. Efficient RMI: Dynamic Specialization of Object Serialization. In Proc. ofIEEE Int'l Conf. on Distributed Computing Systems (ICDCS), pages 308-315, 2000. Pascal Grange and Pierre Vign6ras. The JToe project, http://jtoe.sf.net L. Courtrai, Y. Mah6o and F. Raimbault. Expresso: a Library for Fast Transfers of Java Objects. In Myrinet User Group Conference, Lyon, 2000. The Kaffe Java virtual machine, http://www.kaffe.org/ Rob Van Nieuwpoort, Jason Maassen, Rutger Hofman, Thilo Kielmann, Henri E. Bal. Ibis: an Efficient Java-based Grid Programming Environment. Joint ACM Java Grande ISCOPE 2002 Conference, pp 18-27, 2002. B. Alpern, C. R. Attanasio, A. Cocchi, D. Lieber, S. Smith, T. Ngo, J. J. Barton, S. F. Hummel, J. C. Sheperd, and M. Mergen. Implementing Jalapefio in Java. In Proceedings of the 1999 ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages & Applications, OOPSLA 99, Denver, Colorado, November 1-5, 1999, volume 34(10) of ACM SIGPLAN Notices, pages 314 324. ACM Press, Oct. 1999.
This Page Intentionally Left Blank
Parallel Computing: SoftwareTechnology,Algorithms, Architecturesand Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
143
A M o d u l a r D e b u g g i n g I n f r a s t r u c t u r e for Parallel P r o g r a m s D. Kranzlmtiller ~, Ch. Schaubschl~igel~, M. Scarpa a, and J. Volkert ~ aGUP, University of Linz, Altenbergerstr. 69, 4040 Linz, Austria Debugging parallel and distributed programs is a difficult activity due to multiple concurrently executing and communicating tasks. One major obstacle is the amount of debugging data, which needs to be analyzed for detecting errors and their causes. The debugging tool DeWiz addresses this problem by partitioning the analysis activities into different, independent modules, and distributing these modules on the available, possibly distributed, computing environment. The target of the analysis activities is the event graph, which represents a program's behavior in terms of program state changes and corresponding relations. In DeWiz we distinguish three different kinds of modules, namely Event Graph Generation Modules, which generate an event graph stream from the collected trace data, (Automatic) Analysis Modules, whose task is to perform various analysis techniques on the event graph, and Data Access Modules, which present the results of the event graph analysis to the user in a meaningful way. The interconnection between the different modules is established by a dedicated protocol, which is currently based on TCP/IP or uses shared memory. This allows any user of DeWiz to easily integrate arbitrary software analysis tools with DeWiz. 1. O V E R V I E W O F P R O G R A M A N A L Y S I S W I T H D E W I Z 1.1. Motivation and related work
Software development for High-Performance Computers (HPC) faces several serious challenges. One example is the possible scale of the programs, determined by long execution times and large numbers of participating processes. As a result huge amounts of program state data have to be processed during program analysis activities, such as testing and tuning. This problem is addressed by the debugging tool DeWiz (Debugging Wizard), which evolved out of the Monitoring and Debugging environment MAD [6] as a possible solution. The original approach of MAD was repeatedly affected by the amount of debugging data. Long waiting times during interactive debugging sessions and the impracticability of certain analysis techniques represented substantial drawbacks, especially for real-world applications. Similar approaches to MAD are provided by commercial products, such as VAMPIR [1], or other related tools in this area, e.g. AIMS [18], Paje [2], Paradyn [11 ], Paragraph [3], and PROVE [4]. Each of these tools employs some kind of space-time diagram as a representation of program behavior. By analyzing these related approaches in the field, some basic characteristics and differences are obtained: 9 Analysis goals: performance analysis vs. error debugging Most software tools support only one analysis area, focusing either on performance as-
144 pects of a program or on the correctness of its behavior. Some tools provide additional support for other tasks of the software life-cycle, e.g. visual parallel programming. 9 Means of communication/execution: shared memory vs. message passing Depending on the underlying hardware architecture, runtime system, or development environment, software tools usually support only one kind of programming paradigm. An additional distinction is the parallel execution itself, e.g. whether threads or processes are used. 9 Levels of abstraction: high-level code vs. machine instructions Only few tools provide program analysis activities on different levels of abstraction, while most are pre-determined by the chosen level of instrumentation. 9 Approaches to instrumentation: static source vs. dynamic binary instrumentation Instrumentation of programs allows to distinguish between various different approaches, with static instrumentation of source code on one side of the spectrum, and dynamic instrumentation of binary machine code on the other side. 9 Connection between monitor and analysis tool: on-line vs. post-mortem Another related characteristic of software tools is the connection between the monitoring part, that extracts the behavioral data, and the analysis part, that investigates the program's behavior. In brief, on-line program analysis is required whenever changes to the execution behavior should be applicable on the fly, while post-mortem analysis is chosen if the analysis technique requires information about the complete execution history of a program. The reasons for all these distinctions are often given by the particular interests of the involved software tool developers. Besides that, there are also some tools supporting characteristics that seem converse at first. For example, the VAMPIR tool supports both, shared-memory OpenMP programs [ 14] and message-passing program using MPI [ 13]. This originates from the fact, that some of todays architectures are best utilized by using a hybrid MPI/OpenMP programming style [ 15], and more tools have already been proposed to follow this mixed-mode programming style. During our work on the MAD environment, some more evidence occurred: The event manipulation technique originally developed for message-passing programs required only minor changes to be useful for shared-memory codes [5]. With little extensions, most of them in the monitoring code, MAD was also applicable for performance analysis activities. These experiences motivated to extend the originally conservative approach of using the event graph for message-passing codes only into a more universal solution, whose result is the program analysis too 1 DeWiz. The ideas of DeWiz, namely 9 the representation of a program's behavior as event graph, 9 the modular, hence flexible and extensible approach, and 9 the graphical representation of a program's behavior will be described in more detail in the following sections.
145 2. EVENT GRAPH The theoretical fundament of the DeWiz debugging environment is the event graph, which has been defined as follows:
Definition 1 (Event Graph [8]). An event graph is a directed graph G = (E, -~) , where E is the non-empty set o f events e E E, while --+ is a relation connecting events, such that x --~ y means that there is an edge from event x to event y in G with the "tail" at event x and the "head" at event y.
The events e C E are the recorded events observed during a program's execution, like for example send or receive events in a message-passing program or semaphore locks and unlocks in a shared memory environment, respectively. The relation connecting the events of an event graph is the so called happened-before relation which is defined as follows:
Definition 2 (Happened-before relation [10]). The happened-before relation -~ is defined as
s
where ~ is the sequential order o f events relative to a particular responsible object, while is the concurrent order relation connecting events on arbitrary responsible objects.
c
The sequential order relation ~s simply defines that if two events epi and ej occur on the same 9
process and event e;i occurred before event e3p,then ep 9
C
The concurrent order relatlon~ defines inter-process relations of events. In case of messagepassing programs it means, that if event epi occurs on process p and is a send event, and event e~ occurs on process q and is the corresponding receive event, then ep Based on these definitions we have implemented the DeWiz protocol, which allows to propagate an event graph stream through a DeWiz system.
3. DEWIZ PROTOCOL AND FRAMEWORK The DeWiz protocol defines the communication of event graph streams between different DeWiz modules. Based on this protocol we use the abstract concepts of interfaces and communication channels. One channel connects exactly two modules, channels are unidirectional, and each module has exactly one interface (incoming or outgoing) per channel. There are two approaches how these channels and interfaces are implemented, The first one ist to use TCP/IP and BSD sockets [! 6]. This approach enables the tool to be distributed across different computing resources or even to use grid infrastructures, which are currently deployed all over the world. For example, in a DeWiz debugging session it might be feasible to execute an event graph generation module on the same computer where the application under observation is executed. The event graph generated by this module could be forwarded to a (probably resource intensive) analysis module which is executed on a different computer, in order to provide as much computing power as possible to the observed application (and not having to share resources with analysis tasks). The analysis module could then be connected to
146 a visualization module, which resides on a third computer, e.g. the workstation of the DeWiz user. However, in some cases this approach might not be useful, e.g. when the amount of trace data, or the size of the event graph respectively, exceeds certain limits. In this case it is not efficient, or even impossible, to send the event graph stream over the network, simply because of it's size. In such a case the event graph generation module and the analysis module(s) must reside on the same computer to avoid copying of the event graph. For that purpose DeWiz provides shared memory interfaces in addition to TCP/IP. When two modules are connected via such a shared memory interface, the "producer" module writes the event graph data into a shared memory segment, while possible "consumer" modules can read the data from that segment, thus following the "zero copy" paradigm. To enable the propagation of an event graph with the DeWiz protocol, several data structures have been defined. Simplified, an event graph stream consists of two kinds of such data structures: event: epi = (p, i, type, data) concurrent order relation: ep The event data structure corresponds to a particular state change observed in a program's execution. The variables p and i identify the responsible object on which the event occurred and its relative sequential order, respectively. The variable type determines the kind of observed event, e.g. in message-passing programs a send or receive. The data field is used for optional attributes, which describe the event in more detail depending on the intended analysis activities. The concurrent order relation represents a subset of the happened-before relation stated above. It is used to mark corresponding events on distinct processes, whose operations are somehow connected, e.g. a corresponding pair of send and receive operations. (Please note, that events on the same host object p are already ordered by their sequential identifier i.) The DeWiz Framework offers the required functionality to implement DeWiz modules for a particular programming language. At present, the complete functionality of DeWiz is supported in Java, while smaller fragments of the analysis pipeline are already available in C and C++. Each module must implement the following functionality: 9 Open event graph stream interface 9 Filter relevant events for processing 9 Process event graph stream as desired 9 Close event graph stream interface The functions to open and close interfaces are used to establish and destroy interconnections between modules. The interfaces transparently implement the DeWiz protocol, while filtering and processing events is performed within the main loop of each module.
147
Figure 1. DeWiz System during Event Graph Stream Processing Four monitors generate the event graph stream - presumably through on-line observation of 4 processes. A merger module and 2 buffer modules combine and cache the event graph stream. A pattern matching module and a group detection module perform automatic analysis, while the results of these analysis activities are presented in a visualization tool.
4. DEWIZ MODULES The processing of DeWiz is performed on the above mentioned data structures within DeWiz modules. The modules are assembled as a DeWiz system, which represents the intended program analysis pipeline for a particular debugging or performance analysis strategy. In Figure 1 an example DeWiz system is shown. Depending on their processing capabilities, different types of modules can be distinguished:
4.1. Event graph generation modules At least one module is required for generating an event graph stream corresponding to a program's execution. The events can be generated either on-line with a monitoring tool, or post-mortem by reading corresponding tracefiles. In case DeWiz is utilized as a plug-in for existing program analysis tools, the event graph stream is generated by converting the specific data structures of the host tool into the event graph stream protocol. Example modules already provided by the DeWiz framework include an on-line interface to OCM (OMIS Compliant Monitor) [ 17] and OPARI (OpenMP Pragma and Region Instrumentor) [12][5], as well as a trace reader for the monitoring tool NOPE [7]. 4.2. Automatic analysis modules The actual operations on the event graph are performed by automatic analysis modules. These modules extract the desired information and try to detect interesting characteristics of the program's behavior. Example modules are already provided for error detection, e.g. to determine communication errors in message-passing programs or race conditions at semaphore operations in shared memory programs. More elaborate examples include a pattern matcher for repeated behavioral patterns and a module for detection of process grouping and subsequently process isolation [9].
148 4.3. Data access modules
After or during processing the event graph stream, detected program characteristics can be presented to the user. Different kinds of data access modules support a variety of user interfaces. In most cases, DeWiz forwards the results to a visualization module that represents the event graph as a kind of space-time diagram. Example modules include ATEMPT, the visualization tool of the MAD environment, a failure notification mechanism for cellular phones, and a Java applet for arbitrary web browsers. 5. EXAMPLES The analysis functionality already implemented in DeWiz is described with the following two examples, namely extraction of communication failures, and pattern matching and loop detection. Communication failures can be detected by pairwise analyzing of communication partners. A set of analysis activities for message- passing programs is described in [8]. An example is the detection of different message length at send and receive operations, which may formally be defined as follows: Two events epi and e~9are called events with different message length, if epi c ej and
messageLength(e;) =fimessageLength(eJ). This formal definition can easily be mapped onto a DeWiz module with the available Java framework. During analysis, the module detects every communication pair, where the send operation transmits more or less bytes than the receive operation expects. A more complex analysis activity compared to the extraction of communication failures is pattern matching and loop detection. The goal of the corresponding DeWiz modules is to identify repeated process interaction patterns in the event graph. Some example event graph patterns are given in Figure 2. The leftmost pattern is called a simple exchange pattern, which describes the situation, where two arbitrary processes mutually exchange some kind of data item. The existence of this simple event graph pattern can easily be verified within a DeWiz module. More complex patterns can be specified and provided in a pattern database according to the needs of users and the characteristics of their programs. Vice versa, the user may even specify expected patterns of a program, and the program tries to locate them in the event graph of the observed execution. This allows to decrease the complexity of the analysis data, if patterns are detected in a parallel program, or to detect incorrect behavior, if expected patterns are missing. Some more details about pattern matching can be found in [8]. 6. S U M M A R Y
The modular approach of DeWiz allows parallel and distributed program analysis based on a set of independent analysis modules. The target representation of program behavior is the abstract event graph model, which allows a wide variety of analysis activities for different kinds of programs on different levels of abstraction. The analysis modules may be arbitrarily distributed across a set of available resources, which allows to process even large amounts of program state data in reasonable time. The latter is especially interesting with respect to grid computing, which may be just the fight environment for a future DeWiz grid debugging service.
149
S ->
simple exchange
Figure 2. Process interaction patterns The leftmost pattern is called a simple exchange pattern. In the middle a round robin pattern is shown, while the right screenshot shows a tree-like communication pattern.
REFERENCES
[1]
[2]
[3] [4]
[5]
[6] [7] [8]
[9]
[ 10] [11]
H. Brunst, H.-Ch. Hoppe, W.E. Nagel, and M. Winkler. Performance Optimization for Large Scale Computing: The Scalable VAMPIR approach. Proc. ICCS 2001, Intl. Conference on Computational Science, Springer-Verlag, LNCS, Vol. 2074, San Francisco, CA, USA (May 2001). J. Chassin de Kergommeaux, B. Stein. Paje: An Extensible Environment for Visualizing Multi-Threaded Program Executions. Proc. Euro-Par 2000, Springer-Verlag, LNCS, Vol. 1900, Munich, Germany, pp. 133-144 (2000). M.T. Heath, J.A. Etheridge. Visualizing the Performance of Parallel Programs. IEEE Software, Vol. 13, No. 6, pp. 77-83 (November 1996). R Kacsuk. Performance Visualization in the GRADE Parallel Programming Environment. Proc. HPC Asia 2000, 4th Intl. Conference/Exhibition on High Performance Computing in the Asia-Pacific Region, Peking, China, pp. 446-450 (2000). R. Kobler, D. Kranzlmiiller, and J. Volkert. Debugging OpenMP Programs using Event Manipulation. Proc. WOMPAT 2001, Intl. Workshop on OpenMP Applications and Tools, Springer-Verlag, LNCS, Vol. 2104, West Lafayette, Indiana, USA, pp. 81-89 (July 2001). D. Kranzlmfiller, S. Grabner, and J. Volkert. Debugging with the MAD Environment, Parallel Computing, Vol. 23, No. 1-2, pp. 199-217 (Apr. 1997). D. Kranzlmfiller, J. Volkert. Debugging Point-To-Point Communication in MPI and PVM. Proc. EuroPVM/MPI 98, Intl. Conference, Liverpool, GB, pp. 265-272 (Sept. 1998). Event Graph Analysis for Debugging Massively ParD. Kranzlmfiller: allel Programs, PhD thesis, GUP, Joh. Kepler Univ. Linz (Sept. 2000), http ://www. gup. uni-linz, ac. at/-dk/thesis. D. Kranzlmfiller: Scalable Parallel Program Debugging with Process Isolation and Grouping, Proc. IPDPS 2002, 16th International Parallel & Distributed Processing Symposium, Workshop on High-Level Parallel Programming Models & Supportive Environments (HIPS 2002), IEEE Computer Society, Ft. Lauderdale, Florida, (April 2002). L. Lamport. Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM, Vol. 21, No. 7, pp. 558-565, (July 1978). B.P. Miller, M.D. Callaghan, J.M. Cargille, J.K. Hollingsworth, R.B. Irvin, K.L. Kara-
150 vanic, K. Kunchithapadam, and T. Newhall. The Paradyn Parallel Performance Measurement Tool IEEE Computer, Vol. 28, No. 11, pp. 37-46 (November 1995). [12] B. Mohr, A.D. Malony, S. Shende, and F. Wolf. Design and Prototype of a Performance Tool Interface for OpenMP. Proc. LACSI Symposium 2001, Los Alamos Computer Science Institute, Santa Fe, New Mexico, USA (October 2001). [ 13] Message Passing Interface Forum. MPI: A Message Passing Interface Standard- Version I.I. http://www.mcs,
anl. gov/mpi/(June 1995).
[14] OpenMP Architecture Review Board. OpenMP C and C++ Application Program Interface - Version 2.0. h t t p : //www. openmp, o r g / ( M a r c h 2002). [15] R. Rabenseiffner. Communication and Optimization Aspects on Hybrid Architectures. Proc. EuroPVMMPI 2002, Springer-Verlag, LNCS, Vol. 2474, Linz, Austria, pp. 410-420 (2002). [ 16] W. Richard Steves. UNIX Network Programming. Prentice Hall, (1990). [ 17] R. Wism/iller, J. Trinitis, and T. Ludwig. O C M - a Monitoring System for Interoperable Tools. Proc. SPDT 98, 2nd SIGMETRICS Symposium on Parallel and Distributed Tools, ACM Press, Welches, Oregon, USA, pp. 1-9 (August 1998). [ 18] J.C. Yan, H.H. Jin, and M.A. Schmidt. Performance Data Gathering and Representation from Fixed-Size Statistical Data. Technical Report NAS-98-003,
http://www.nas .nasa.gov/Research/Reports/Techreports/
19 9 8 / n a s - 9 8 - 0 0 3. p d f , NAS System Division, NASA Ames Research Center, Moffet Field, California, USA (February 1999).
Parallel Computing: Software Technology, Algorithms, Architectures and Applications G.R. Joubert, W.E. Nagel, F.J. Peters and W.V. Walter (Editors) 9 2004 ElsevierB.V. All rights reserved.
151
Toward a Distributed Computational Steering Environment based on CORBA O. Coulaud ~, M. Dussere ~, and A. Esnard ~ ~Projet ScA1Applix, INRIA Futurs et LaBRI UMR CNRS 5800, 3 51, cours de la Lib6ration, F-33405 Talence, France This paper presents the first step toward a computational steering environment based on CORBA. This environment, called EPSN 1, allows the control, the data exploration and the data modification for numerical simulations involving an iterative process. In order to be as generic as possible, we introduce an abstract model of steerable simulations. This abstraction allows us to build steering clients independently of a given simulation. This model is described with an XML syntax and is used in the simulation by some source code annotations. EPSN takes advantage of the CORBA technology to design a communication infrastructure with portability, interoperability and network transparency. In addition, the in-progress parallel CORBA objects will give us a very attractive framework for extending the steering to parallel and distributed simulations. 1. I N T R O D U C T I O N Thanks to the constant evolution of computational capacity, numerical simulations are becoming more and more complex; it is not uncommon to couple different models in different distributed codes running on heterogeneous networks of parallel computers (e.g. multi-physics simulations). For years, the scientific computing community has expressed the need of new computational steering tools to better grasp the complexity of the underlying models. The computational steering is an effort to make the typical simulation work-flow (modeling, computing, analyzing) more efficient, by providing on-line visualization and interactive steering over the on-going computational processes. The on-line visualization appears very useful to monitor and detect possible errors in long-running applications, and the interactive steering allows the researcher to alter simulation parameters on-the-fly and immediately receive feedback on their effects. Thus, the scientist gains a better insight in the simulation regarding to the cause-andeffect relationship. A computational steering environment is defined in [ 1] as a communication infrastructure, coupling a simulation with a remote user interface, called steering system. This interface usually provides on-line visualization and user interaction. Over the last decade, many steering environments have been developed; they distinguish themselves by some critical features such as the simulation integration process, the communication infrastructure and the steering system design. A first solution for the integration is the problem solving environment (PSE) approach, like in SCIRun [2]. This approach allows the scientist to construct a steering application according to a visual programming model. As an opposite, CAVEStudy [3] only interacts with
1EPSN project (http ://www. labri, fr/epsn) is supported by the French ACI-GRID initiative.
152 the application through its standard input/output. Nevertheless, the majority of the steering environments, such as the well-known CUMULVS [4], are based on the instrumentation of the application source-code. We have chosen this approach as it allows fine grain steering functionalities and achieves good runtime performances. Regarding the communication infrastructure, there are many underlying issues especially when considering parallel and distributed simulations: heterogeneous data transfer, network communication protocols and data redistributions. In VIPER [5], RPCs and the XDR protocol are used to implement the communication infrastructure. Magellan & Falcon [6] communicates over heterogeneous environments through an event system built upon DataExchange. CUMULVS uses a high-level communication infrastructure based on PVM and allows to collect data from parallel simulations with HPF-like data redistributions. In EPSN project, we intend to explore the capabilities of the CORBA technology and the currently under development parallel CORBA objects for the computational steering of parallel and distributed simulations. In this paper, we first describe the basis and the architecture of EPSN. Then, we illustrate the integration process of a simulation. Finally, we present preliminary results on the EPSN prototype, called Epsilon.
2. THE E P S N ENVIRONMENT
2.1. Principles EPSN is a steering platform for the instrumentation of numerical simulations. In order to be as generic as possible, we introduce an abstract model of steerable simulations. This model intends to clearly identify where, when and how a remote client can safely access data and control the simulation. We consider numerical simulations as iterative processes involving a set of data and a hierarchy of computation loops modifying these data. Each loop is associated with a single counter and the association of all these counters enables EPSN to precisely follow the global time-step evolution. The "play/stop" control operations imply the definition of breakpoints at some stable states of the simulation. In practice, we can easily steer most of the simulations by simply instrumenting their main loop. The basic access operations consist in extracting and modifying the data. As these data alternate between coherent and incoherent states during the computation, it implies the definition of restricted areas where data are not accessible. Several data can be logically associated to define a group that enables the end-user to efficiently manipulate correlated data together. Moreover, we have extended the coherency definition for groups to guarantee that all group members are accessed at the same iteration. This model, which fits well parallel applications (SPMD), needs to define a global time extension to maintain the data coherency over coupled or distributed simulations (MPMD). Such a mechanism implies an explicit association of the loops and breakpoints of the different simulation components. The representation in the abstract model is obtained by pointing out the relevant informations on the simulation. First the user describes the simulation elements in an XML file, then he connects the simulation with its representation. To do so, he annotates the source code with EPSN API in order to locate the elements that he has identified and to mark their evolution through the simulation process. The XML description also intends to lighten and clarify this annotation phase.
153 ..............................................
i EPSN 'nfrastructu~ffs A ~ Steerable
_1~ :i _ .Simulati~ ..S~~
Side
~I i [.C_o : ~o ;, 'I 0 ~1~1
Client Side
....
i. . . . . . . . .
d . . . . . Is
dedi....dthread ~ ..............................................
Steering
r ~ ~.o
:o
SimulationLoop ~ ' : ' ~ q ~ <-,,,,-> C O R B A requests
i
. .................................................
(~) -~
UserC . . . . .
~
~
~
P..... In,.... , C. . . . .
'--'~VisualizationLo~ dthre
~
-...............................
,cation Layer (MP,)
iO
CORBAobject( .....
" []
client C O R B A reference
)
~
Cliont ~
~")[I
i(~ !
,cation Layer
Object Request Broker
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
:
/
Client ~j~ k J]l . .......................................
Figure 1. (a) Detail of the EPSN architecture. (b) EPSN parallel architecture with PaCO++.
2.2. Architecture and communication infrastructure EPSN is a distributed and dynamic infrastructure. It is based on a client/server relationship between the steering system (the client) and the simulation (the server) which both use EPSN libraries. The clients are not tight coupled with the simulation. Actually, they can interact onthe-fly with the simulation through asynchronous and concurrent requests. According to this model, different steering systems can concurrently access the same simulation and, reciprocally, a steering system can simultaneously access different simulations. These characteristics make EPSN environment very flexible and dynamic. The communication infrastructure of EPSN is based on CORBA, but it is completely hidden to the end-user. CORBA enables applications to communicate in a distributed heterogeneous environment with network transparency according to a RPC programming model. It also provides to EPSN the interoperability between applications running on different machine architectures. Although, CORBA is often criticized for its performance, some implementations are very efficient [7]. The principle of EPSN infrastructure is to run a permanent thread attached to each simulation process. This thread contains a multi-threaded CORBA server dedicated to the communications between the steering clients and the simulation. As shown on figure 1(a), the simulation thread consists of different CORBA objects corresponding to EPSN functionalities. For being fully asynchronous, EPSN uses oneway CORBA calls and the client thread also implements a callback object to receive data from the simulation. In other steering environments, like CUMULVS, the simulation is in charge of the communications that occur during a single blocking subroutine call. In EPSN, the thread accesses directly to a data through the shared memory of the process without any copy. Between a process and the EPSN thread, the communications use standard inter-thread synchronization mechanisms based on semaphores and signals. This strategy combined with the asynchronous CORBA calls allows to overlap the communications. In order to maintain a single representation of the simulation, we use a specific CORBA object, the proxy, running on the first simulation process. This object provides the description of the whole simulation and all the CORBA references needed by both the client and other simulation processes. As the proxy is registered to the CORBA naming service, remote clients can easily connect it. In order to achieve coherent steering operations on SPMD simulations, we have developed some protocols to synchronize the simulation processes. This synchronization implies to broadcast the request to all the involved processes and to synchronize on the first
154 breakpoint before achieving the parallel request. To reduce the synchronization cost, the parallel processes can also synchronize once at the beginning and keep going synchronously after that. The parallel extension of CORBA objects, like PaCO++ [8], reduces the synchronization cost thanks to a better use of the parallel infrastructure. PaCO++ typically exploits an internal communication layer based on MPI (Fig. l(b)). It also greatly eases the EPSN extension to parallel clients and for the inherent problem of data redistribution. On-going works are focusing on the integration of PaCO++ in EPSN and especially on the full support of regular data decomposition. 2.3. Functionalities EPSN consists in two C/Fortran libraries, the first one provides functions to build a steerable simulation and the second one enables to build a remote steering system. Control. The insertion of b a r r i e r function calls, acting as debugger breakpoints, in the simulation source code allows the user to control the execution flow of the simulation. The breakpoints can be remotely set "up" or "down" ( s e t b a r r i e r command) and they allow classical control commands (play, step, stop). Moreover, the calls to iterate function point out the evolution of simulation through the