Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen
2565
3
Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo
José M.L.M. Palma Jack Dongarra Vicente Hernández A. Augusto Sousa (Eds.)
High Performance Computing for Computational Science – VECPAR 2002 5th International Conference Porto, Portugal, June 26-28, 2002 Selected Papers and Invited Talks
13
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors José M.L.M. Palma A. Augusto Sousa Faculdade de Engenharia da Universidade do Porto Rua Dr. Roberto Frias, 4200-465 Porto, Portugal E-mail: {jpalma,augusto.sousa}@fe.up.pt Jack Dongarra University of Tennessee, Department of Computer Science Knoxville, TN 37996-1301, USA E-mail:
[email protected] Vicente Hernández Universidad Politécnica de Valencia Departamento de Sistemas Informáticos y Computación Camino de Vera, s/n, Apartado 22012, 46020 Valencia, Spain E-mail:
[email protected]
Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliographie; detailed bibliographic data is available in the Internet at
.
CR Subject Classification (1998): D, F, C.2, G, J.2, J.3 ISSN 0302-9743 ISBN 3-540-00852-7 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion by DA-TeX Gerd Blumenstein Printed on acid-free paper SPIN: 10871851 06/3142 543210
Preface
The 5th edition of the VECPAR series of conferences marked a change of the conference title. The full conference title now reads VECPAR 2002 — 5th International Conference on High Performance Computing for Computational Science. This reflects more accurately what has been the main emphasis of the conference since its early days in 1993 – the use of computers for solving problems in science and engineering. The present postconference book includes the best papers and invited talks presented during the three days of the conference, held at the Faculty of Engineering of the University of Porto (Portugal), June 26–28 2002. The book is organized into 8 chapters, which as a whole appeal to a wide research community, from those involved in the engineering applications to those interested in the actual details of the hardware or software implementation, in line with what, in these days, tends to be considered as Computational Science and Engineering (CSE). The book comprises a total of 49 papers, with a prominent position reserved for the four invited talks and the two first prizes of the best student paper competition. Chapter 1 deals with the application of discretization techniques to engineering problems. The invited lecture delivered by Rainald L¨ ohner sets the tone in this chapter, comprised of six more papers covering areas such as, for instance, combustion and geology. The talks by Vipin Kumar and Leif Eriksson open Chaps. 2 and 3 on data-mining techniques and computing in chemistry and biology. Problem-solving environments is the major topic in Chap. 4, initiated by the invited talk on the cactus software by Ed Seidel. Chapter 5 has nine papers on linear and nonlinear algebra. The papers in Chap. 6, though under the title of cluster computing, may be of general interest. Cluster computing is living up to its expectations and enabling high-performance computing at an affordable price. The last chapter, Chap. 8, has 12 papers under the general title of software tools and environments, and is thus close to computer science. Student Papers Awarding two first prizes for the best student paper was a consequence of the quality of and the difficulty in distinguishing between two good pieces of work originating from quite different fields. This was also the way found by the jury to honor the multidisciplinarity of the conference. At this point, we would like to wish the greatest success to the authors and in particular to the students Marc Lange and Arturo Escribano. – Investigation of Turbulent Flame Kernels Using DNS on Clusters by Marc Lange (High-Performance Computing Center, Stuttgart, Germany)
VI
Preface
– Mapping Unstructured Applications into Nested Parallelism by Arturo Gonz´ alez-Escribano (Universidad de Valladolid, Spain), Arjan J.C. van Gemund (Faculty of Information Technology and Systems, The Netherlands) and Valent´ın Carde˜ noso-Payo (Universidad de Valladolid, Spain) To conclude, we would like to state in writing our gratitude to all members of the Scientific Committee and additional referees. Their opinions and comments were essential in the preparation of this book and the conference programme. We hope that the knowledge and experience of many contributors to this book can be useful to a large number of readers. October 2002
Jos´e M.L.M. Palma Jack Dongarra Vicente Hern´andez A. Augusto Sousa
VECPAR is a series of conferences organized by the Faculty of Engineering of Porto (FEUP) since 1993.
Acknowledgments
Apart from those acknowledged in the following pages, we would like to thank V´ıtor Carvalho, who created and maintained the conference website, and Alice Silva for the secretarial work. A very special word of gratitude is deserved to the members of the Organizing Committee Jo˜ ao Correia Lopes and Jos´e Magalh˜ aes Cruz. The web-based submission and refereeing and the management of the VECPAR databases would have been impossible without their precious help. VECPAR 2002 was a landmark in many aspects. It was the first edition of the conference held at the new buildings of the Faculty of Engineering. It was the end of a series of conferences held in Porto; the next edition of the conference will be held in Valencia (Spain). We believe that now is the time to acknowledge the collaboration of many people, always behind the limelight, who have been essential to the success of the VECPAR series of conferences. To mention only a few, and extending our gratitude to all personnel under their supervision, it is our pleasure to thank Mrs. Branca Gon¸calves, from the External Relations Service of FEUP and Mr. Lu´ıs Machado, from Abreu Travel Agency.
Committees
Organizing Committee A. Augusto de Sousa (Chair) Jos´e Couto Marques (Co-chair) Jo˜ ao Correia Lopes Jos´e Magalh˜ aes Cruz Fernando Silva
Steering Committee Jos´e Laginha Palma, Universidade do Porto, Portugal Jack Dongarra, University of Tennessee, USA Vicente Hernandez, Universidad Polit´ecnica de Valencia, Spain Michel Dayd´e, ENSEEIHT-IRIT, France Satoshi Matsuoka, Tokyo Institute of Technology, Japan Jos´e Fortes, University of Florida, USA
Best Student Paper Award (Jury) ´ Alvaro Coutinho, Universidade Federal do Rio de Janeiro, Brazil Jos´e Fortes, University of Purdue, USA Ronan Guivarch, ENSEEIHT-IRIT, France Rainald L¨ ohner, George Mason University, USA Vipin Kumar, University of Minnesota, USA Jos´e Palma, Universidade do Porto, Portugal Fernando Silva, Universidade do Porto, Portugal Aad van der Steen, Utrecht University, The Netherlands
Scientific Committee R. Benzi L. Biferale A. Coutinho J.C. Cunha F. d’Almeida I. Duff N. Ebecken S. Gama J.M. Geib W. Gentzsch
Univ. de Roma, Italy Univ. de Roma, Italy Univ. Federal do Rio de Janeiro, Brazil Univ. Nova de Lisboa, Portugal Univ. do Porto, Portugal Rutherford Appleton Lab., UK Univ. Federal do Rio de Janeiro, Brazil Univ. do Porto, Portugal Univ. Sciences et Technologies de Lille, France SUN, USA
Committees
A. George L. Giraud D. Knight V. Kumar R. L¨ ohner E. Luque A. Nachbin A. Padilha M. Peric T. Priol I. Procaccia R. Ralha D. Ruiz H. Ruskin M. Stadtherr F. Tirado A. Trefethen M. Valero A. van der Steen B. Tourancheau E. Zapata
Univ. of Florida, USA CERFACS, France Rutgers Univ., USA Univ. Minnesota, USA George Mason Univ., USA Univ. Aut` onoma de Barcelona, Spain Inst. Matem´atica Pura e Aplicada, Brazil Univ. do Porto, Portugal CD Star, Germany IRISA/INRIA, France Weizmann Institute of Science, Israel Univ. do Minho, Portugal ENSEEIHT-IRIT, France Dublin City Univ., Ireland Univ. Notre Dame, USA Univ. Complutense de Madrid, Spain NAG, UK Univ. Polit´ecnica de Catalunya, Spain Utrecht Univ., The Netherlands ´ Ecole Normale Sup´erieure de Lyon, France Univ. de Malaga, Spain
Additional Referees A. Augusto de Sousa A. Pimenta Monteiro A. Teixeira Puga Aleksander Lazarevic Alexandre Denis Alfredo Bautista Ananth Grama Christian P´erez Christophe Calvin Cl´ audio L. Amorim Cyril Gavoille Didier Jamet El Mostafa Daoudi Enrique Fernandez-Garcia Fernando Alvarruiz Fernando Silva Ignacio Blanquer J.B. Caillau Jesus Corbal Jorge Barbosa Jos´e Diaz
Jos´e E. Roman Jos´e Gonzalez Jos´e L. Larriba-Pey Jos´e M. Alonso Jos´e M. Cela Josep Vilaplana Jos´e C. Marques Jos´e Nuno FIdalgo Jo˜ao Gama Julien Langou Kirk Schloegel Laurent Lefevre Loic Prylli Luis B. Lopes Luis Torgo M. Eduardo Correia M. Prieto Matias M. Pr´ ospero dos Santos M. Radlinski M. Velhote Correia Martin van Gijzen
IX
X
Committees
Michael Steinbach M´ ario Oldemiro Nuno Correia Paulo B. Vasconcelos Pedro D. Medeiros Pierre Manneback R.E. Goodman Ramdev Kanapady Renaud Lottiaux
Ricardo Lopes Ricardo Rocha Ronan Guivarch S. Oral Serge Miguet Vivek Sarin Xavier Pueyo Yvon Jegou
Invited Lecturers
– Yutaka Akiyama Computational Biology Research Center, Japan – Leif Eriksson Uppsala University, Sweden – Vipin Kumar University of Minnesota, USA – Rainald L¨ ohner George Mason University, USA – Edward Seidel Max-Planck-Institut f¨ ur Gravitationsphysik, Albert-Einstein-Institut, Germany
Sponsoring Organizations
The Organizing Committee is very grateful to the following organizations for their support
UP – Universidade do Porto FEUP – Faculdade de Engenharia da Universidade do Porto INESC Porto – Instituto de Engenharia de Sistemas e de Computadores do Porto FCT – Funda¸c˜ao para a Ciˆencia e Tecnologia POSI – Programa Operacional Sociedade de Informa¸c˜ao FEDER – Fundo Europeu de Desenvolvimento Regional EOARD – European Office of Aerospace Research and Development FCCN – Funda¸c˜ao para a Computa¸c˜ao Cient´ıfica Nacional FCG – Funda¸c˜ao Calouste Gulbenkian FLAD – Funda¸c˜ao Luso Americana para o Desenvolvimento FO – Funda¸c˜ao Oriente OE – Ordem dos Engenheiros CVRVV – Comiss˜ ao de Viticultura da Regi ao dos Vinhos Verdes SCW – Scientific Computing World OCB – Oporto Convention Bureau
Table of Contents
Chapter 1: Fluids and Structures Fluid-Structure Interaction Simulations Using Parallel Computers (Invited Talk) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Rainald L¨ ohner, Joseph D. Baum, Charles Charman, and Daniele Pelessone Investigation of Turbulent Flame Kernels Using DNS on Clusters (Best Student Paper Award: First Prize) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Marc Lange A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver for Incompressible Flow Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Margreet Nool and Michael M.J. Proot Edge-Based Interface Elements for Solution of Three-Dimensional Geomechanical Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Rubens M. Sydenstricker, Marcos A.D. Martins, Alvaro L.G.A. Coutinho, and Jos´e L.D. Alves Finite Element Simulation of Seismic Elastic Two Dimensional Wave Propagation: Development and Assessment of Performance in a Cluster of PCs with Software DSM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Francisco Quaranta, Maria C.S. de Castro, Jos´e L.D. Alves, and Claudio L. de Amorim Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Carmen N.M. Paz, Luiz F. Martha, Jos´e L.D. Alves, Eduardo M.R. Fairbairn, Nelson F.F. Ebecken, and Alvaro L.G.A. Coutinho An A Posteriori Error Estimator for Adaptive Mesh Refinement Using Parallel In-Element Particle Tracking Methods . . . . . . . . . . . . . . . . . . . . . . 94 Jing-Ru C. Cheng and Paul E. Plassmann Chapter 2: Data Mining High Performance Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Vipin Kumar, Mahesh V. Joshi, Eui-Hong (Sam) Han, Pang-Ning Tan, and Michael Steinbach Data Mining for Data Classification Based on the KNN-Fuzzy Method Supported by Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Jos´e L.A. Rosa and Nelson F.F. Ebecken
XIV
Table of Contents
Chapter 3: Computing in Chemistry and Biology Lignin Biosynthesis and Degradation – A Major Challenge for Computational Chemistry . . . . . . . . . . . . . . . . . . . . . . . . . 137 Bo Durbeej, Yan-Ni Wang, and Leif A. Eriksson High Performance Computing in Electron Microscope Tomography of Complex Biological Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Jos´e J. Fern´ andez, Albert F. Lawrence, Javier Roca, Inmaculada Garc´ıa, Mark H. Ellisman, and Jos´e M. Carazo Visualization of RNA Pseudoknot Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Wootaek Kim, Yujin Lee, and Kyungsook Han
Chapter 4: Problem Solving Environments The Cactus Framework and Toolkit: Design and Applications (Invited Talk) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Tom Goodale, Gabrielle Allen, Gerd Lanfermann, Joan Mass´ o, Thomas Radke, Edward Seidel, and John Shalf Performance of Message-Passing MATLAB Toolboxes . . . . . . . . . . . . . . . . . . . . . 228 Javier Fern´ andez, Antonio Ca˜ nas, Antonio F. D´ıaz, Jes´ us Gonz´ alez, Julio Ortega, and Alberto Prieto Evaluating the Performance of Space Plasma Simulations Using FPGA’s . . 242 Ben Popoola and Paul Gough Remote Parallel Model Reduction of Linear Time-Invariant Systems Made Easy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Peter Benner, Rafael Mayo, Enrique S. Quintana-Ort´ı, and Gregorio Quintana-Ort´ı An Approach to Teaching Computer Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Ester M. Garz´ on, Inmaculada Garc´ıa, and Jos´e Jes´ us Fern´ andez
Chapter 5: Linear and Non-linear Algebra Fast Sparse Matrix-Vector Multiplication for TeraFlop/s Computers . . . . . . 287 Gerhard Wellein, Georg Hager, Achim Basermann, and Holger Fehske Performance Evaluation of Parallel Gram-Schmidt Re-orthogonalization Methods . . . . . . . . . . . . . . . . . . 302 Takahiro Katagiri
Table of Contents
XV
Toward Memory-Efficient Linear Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Allison Baker, John Dennis, and Elizabeth R. Jessup A Parallel Newton-GMRES Algorithm for Solving Large Scale Nonlinear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Jes´ us Peinado and Antonio M. Vidal Preconditioning for an Iterative Elliptic Solver on a Vector Processor . . . . . 343 Abdessamad Qaddouri and Jean Cˆ ot´e 2-D R-Matrix Propagation: A Large Scale Electron Scattering Simulation Dominated by the Multiplication of Dynamically Changing Matrices . . . . . . 354 Timothy Stitt, N. Stan Scott, M. Penny Scott, and Phil G. Burke A Parallel Implementation of the Atkinson Algorithm for Solving a Fredholm Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 Filomena Dias d’Almeida and Paulo Beleza Vasconcelos SLEPc: Scalable Library for Eigenvalue Problem Computations . . . . . . . . . . . 377 Vicente Hern´ andez, Jose E. Rom´ an, and Vicente Vidal Parallelization of Spectral Element Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 St´ephane Airiau, Mejdi Aza¨ıez, Faker Ben Belgacem, and Ronan Guivarch Chapter 6: Cluster Computing Mapping Unstructured Applications into Nested Parallelism (Best Student Paper Award: First Prize) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .407 Arturo Gonz´ alez-Escribano, Arjan J.C. van Gemund, and Valent´ın Carde˜ noso-Payo An Efficient Parallel and Distributed Algorithm for Counting Frequent Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 Salvatore Orlando, Paolo Palmerini, Raffaele Perego, and Fabrizio Silvestri A Framework for Integrating Network Information into Distributed Iterative Solution of Sparse Linear Systems . . . . . . . . . . . . . . 436 Devdatta Kulkarni and Masha Sosonkina Efficient Hardware Implementation of Modular Multiplication and Exponentiation for Public-Key Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . 451 Nadia Nedjah and Luiza de Macedo Mourelle Real-Time Visualization of Wake-Vortex Simulations Using Computational Steering and Beowulf Clusters . . . . . . . . . . . . . . . . . . . . . . 464 Anirudh Modi, Lyle N. Long, and Paul E. Plassmann PALM: A Dynamic Parallel Coupler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Andrea Piacentini and The PALM Group A Null Message Count of a Conservative Parallel Simulation . . . . . . . . . . . . . . 493 Ha Yoon Song, Sung Hyun Cho, and Sang Yong Han
XVI
Table of Contents
Chapter 7: Imaging Static Scheduling with Interruption Costs for Computer Vision Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .509 Francisco A. Candelas, Fernando Torres, Pablo Gil, and Santiago T. Puente A Parallel Rendering Algorithm Based on Hierarchical Radiosity . . . . . . . . . . 523 Miguel Caballer, David Guerrero, Vicente Hern´ andez, and Jose E. Rom´ an A High–Performance Progressive Radiosity Method Based on Scene Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 Antonio P. Guerra, Margarita Amor, Emilio J. Padr´ on, and Ram´ on Doallo Wavelet Transform for Large Scale Image Processing on Modern Microprocessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 Daniel Chaver, Christian Tenllado, Luis Pi˜ nuel, Manuel Prieto, and Francisco Tirado
Chapter 8: Software Tools and Environments An Expandable Parallel File System Using NFS Servers . . . . . . . . . . . . . . . . . . .565 F´elix Garc´ıa, Alejandro Calder´ on, Jesus Carretero, Jose M. P´erez, and Javier Fern´ andez Scalable Multithreading in a Low Latency Myrinet Cluster . . . . . . . . . . . . . . . .579 Albano Alves, Ant´ onio Pina, Jos´e Exposto, and Jos´e Rufino Minimizing Paging Tradeoffs Applying Coscheduling Techniques in a Linux Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593 Francesc Gin´e, Francesc Solsona, Porfidio Hern´ andez, and Emilio Luque Introducing the Vector C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608 Patricio Buli´c and Veselko Guˇstin Mobile Agent Programming for Clusters with Parallel Skeletons . . . . . . . . . . . 622 Rocco Aversa, Beniamino Di Martino, Nicola Mazzocca, and Salvatore Venticinque Translating Haskell# Programs into Petri Nets . . . . . . . . . . . . . . . . . . . . . . . . . . . .635 Francisco Heron de Carvalho Junior, Rafael Dueire Lins, and Ricardo Massa Ferreira Lima
Table of Contents
XVII
An Efficient Multi-processor Architecture for Parallel Cyclic Reference Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 650 Rafael Dueire Lins ROS: The Rollback-One-Step Method to Minimize the Waiting Time during Debugging Long-Running Parallel Programs . . . . . . . . . . . . . . . . . . . . . . . 664 Nam Thoai, Dieter Kranzlm¨ uller, and Jens Volkert Distributed Paged Hash Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 679 Jos´e Rufino, Ant´ onio Pina, Albano Alves, and Jos´e Exposto A Meta-heuristic Approach to Parallel Code Generation . . . . . . . . . . . . . . . . . . 693 Barry McCollum, Pat H. Corr, and Peter Milligan Semidefinite Programming for Graph Partitioning with Preferences in Data Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703 Suely Oliveira, David Stewart, and Takako Soma A Development Environment for Multilayer Neural Network Applications Mapped onto DSPs with Multiprocessing Capabilities . . . . . . . . . . . . . . . . . . . . . 717 Eugenio S. C´ aner, Jos´e M. Seixas, and Rodrigo C. Torres Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .731
Fluid-Structure Interaction Simulations Using Parallel Computers Invited Talk Rainald L¨ ohner1 , Joseph D. Baum2 , Charles Charman3 , and Daniele Pelessone4 1
4
School of Computational Sciences and Informatics M.S. 4C7, George Mason University, Fairfax, VA 22030-4444, USA 2 Applied Physics Operations Science Applications International Corp., McLean, VA 22102, USA 3 General Atomics, San Diego, CA 92121, USA Engineering Software System Solutions, Solana Beach, CA 92075, USA
Abstract. A methodology to simulate large-scale fluid-structure interaction problems on parallel machines has been developed. Particular emphasis was placed on shock-structure interaction problems. For the fluid, a high-resolution FEM-FCT solver based on unstructured grids is used. The surface motion is handled either by moving, body fitted grids, or via surface embedding. For the structure, a Lagrangean large-deformation finite element code is employed. The coupled system is solved using a loose coupling algorithm, with position and velocity interpolation and force projection. Several examples, run on parallel machines, demonstrate the range of applicability of the proposed methodology.
1
Introduction
The threat of intentional (Lockerbie (1988) [2], World Trade Center (1993) [3], Khobar Towers (1996), Nairobi and Dar Es Salaam (1998), Aden (2000), New York (2001), Washington (2001), etc.) or accidental (Seveso (1976), Toulouse (2001), oilrigs, oil refineries, etc.) explosions is a reality that planners have to take seriously when designing buildings, particularly those that will have a large concentration of people (airports, embassies, etc.). Any comprehensive methodology that seeks to predict or reproduce these events must account for the relevant physical phenomena, which include: – – – – – –
Shock wave propagation through fluid and solid media; HE ignition and detonation; Afterburn phenomena; Particle transport and burn; Load and position/velocity transfer between fluid and structure; Possible catastrophic failure of the structure with consequent changes to the flowfield domain/topology.
At the same time, the simulations required must be run as fast as possible in order to have an impact on design and/or understanding. This implies that J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 3–23, 2003. c Springer-Verlag Berlin Heidelberg 2003
4
Rainald L¨ ohner et al.
– Integrated pre- and post-processing for fluid and structural dynamics; – Fully automatic grid generation; and – Scalability on parallel computers should be an integral part of any such simulation methodology. This paper reports on recent developments towards and simulations with such a methodology.
2 2.1
CFD Solver Equations Solved
Most of the shock phenomena present in explosions are well described by the Euler equations, given by: u,t + ∇ · Fa = S ,
(1)
u = {ρ , ρvi , ρe} ,
(2)
Faj = {(vj − wj )ρ , (vj − wj )ρvi + pδij , (vj − wj )ρe + vj p)} ,
(3)
where ρ, vi , wj , e and p denote, respectively, the density, cartesian fluid and mesh velocity component in direction xi , total energy and pressure. This set of equations is closed by adding an equation of state for the pressure, given as a function of the density and internal energy ei : 1 p = p(ρ, ei ) , ei = e − vj vj . 2 For ideal gases, this reduces to: p = (γ − 1)ρei ,
(4)
(5)
where γ denotes the ratio of specific heats. In order to solve numerically these PDEs, an explicit finite element solver on unstructured grids is employed. For the spatial discretization linear tetrahedral elements are used, and the resulting discretization is cast in an edge-based formulation to reduce indirect addressing and operation count. Monotonicity, a key requirement to propagate accurately shocks, is maintained via Flux-corrected transport (FEM-FCT) [14]. 2.2
Moving Bodies/Surfaces
For problems that exhibit surface movement, two classic solution strategies have emerged:
Fluid-Structure Interaction Simulations Using Parallel Computers
5
a) Body-Conforming Moving Meshes: here, the PDEs describing the flow are cast in an arbitrary Lagrangean-Eulerian (ALE) frame of reference, the mesh is moved in such a way as to minimize distortion, if required the topology is reconstructed, the mesh is regenerated and the solution reinterpolated as needed. While used extensively (and successfully) [15, 4, 5, 23, 24, 35], this solution strategy exhibits some shortcomings: the topology reconstruction can sometimes fail for singular surface points, and there is no way to remove subgrid features from surfaces, leading to small elements due to geometry; reliable parallel performance beyond 16 processors has proven elusive for most general-purpose grid generators; the interpolation required between grids will invariably lead to some loss if information; and there is an extra cost associated with the recalculation of geometry, wall-distances and mesh velocities as the mesh deforms. b) Embedded Fixed Meshes: here, the mesh is not body-conforming and does not move. At every time-step, the edges crossed by CSD faces are identified and proper boundary conditions are applied in their vicinity. While used extensively - particularly in the context of adaptive Cartesian grids - [36, 28, 33, 11, 32, 12, 1], this solution strategy also exhibits some shortcomings: the boundary, which typically has the most profound influence on the ensuing physics, is also the place where the solution accuracy is worst; no stretched elements can be introduced to resolve boundary layers; adaptivity is essential for most cases; there is an extra cost associated with the recalculation of geometry (when adapting) and the crossed edge information. We have recently explored the second type of solution strategy. This switch was prompted by the inability of CSD codes to ensure strict no-penetration during contact. Several blast-ship interaction simulations revealed that the amount of twisted metal was so considerable that any enforcement of strict no-penetration (required for consisted topology reconstruction) was hopeless. The embedded surface technique used is the simplest possible. Given the CSD triangulation and the CFD mesh, the CFD edges cut by CSD faces are found and deactivated. If we consider an arbitrary field point i, the time-advancement of the unknowns uveci is given by: C ij (Fi + Fj ) , (6) Mi ∆ui = ∆t Ω
where Mi , ∆ui , ∆t, C ij , Fi denote, respectively, the lumped mass-matrix, increments of unknowns, time-step, edge-coefficients and fluxes. For any edge ij crossed by a CSD face, the coefficients C ij are set to zero. This implies that for a uniform state u = const. the balance of fluxes for interior points with cut edges will not vanish. This is remedied by defining a new boundary point to impose total/normal velocities, as well as adding a ‘boundary contribution’, resulting in: i i ij i C (Fi + Fj ) + CΓ Fi . (7) M ∆u = ∆t Ω
6
Rainald L¨ ohner et al.
Fig. 1a. Shock Tube Problem: Embedded Surface
t=20
Density: Usual Body−Fitted Mesh t=20
Density: Embedded CSD Faces in Mesh
Plane Cut With Embedded CSD Faces in Mesh
Fig. 1b. Shock Tube Problem: Density Contours The point-coefficients CΓi are obtained from the condition that ∆u = 0 for u = const. The mass-matrix M i of points surrounded by cut edges must be modified to reflect the reduced volume due to cut elements. Again, the simplest possible modification of M i is used. In a pass over the edges, the smallest cut edge fraction’ ξ for all the edges surrounding a point is found. The modified mass-matrix is then given by: 1 + ξmin i M . (8) 2 Note that the value of the modified mass-matrix can never fall below half its original value, implying that time-step sizes will always be acceptable. The embedded CSD technique is demonstrated by comparing the results on the Sod shock-tube problem for a ‘clean-wall’, body fitted mesh and an equivalent embedded CSD mesh. Initially, a diaphragm placed halfway down a tube separates two states ρ1 = p1 = 1.0 and ρ2 = p2 = 0.1. The embedded geometry can be discerned from Figure 1a. Figure 1b shows the results for the two techniques. Although the embedded technique is rather Mi∗ =
Fluid-Structure Interaction Simulations Using Parallel Computers
7
primitive, the results are surprisingly good. The main difference is little bit more noise in the contact discontinuity region, which may be expected, as this is a linear discontinuity. The motion of the surfaces attached to deforming fluids is imposed from the structural dynamics solver. If body-fitted grids and mesh movement are used, the mesh motion is smoothed using a nonlinear Laplacian [19] in order to reduce the amount of global and local remeshing required. For the embedded formulation, all the face-edge intersections checks are carried out incrementally, and points that ‘cross states’ due to CSD face movement need to be extrapolated appropriately. At the end of each fluid time-step, the loads exerted on the ‘wetted surfaces’ belonging to the structural dynamics solver are computed and transferred back. 2.3
HE Ignition and Detonation
Most high explosives are well modeled by the Jones-Wilkins-Lee (JWL) equation of state, given by: ω ω −R1 v +B 1− (9) e e−R2 v + ωρei , p=A 1− R1 v R2 v where v denotes the relative volume of the gas: v=
V ρ0 . = V0 ρ
(10)
Observe that in the expanded state (v → ∞), the JWL equation of state reduces to p = ωρei = (γ − 1)ρei ,
(11)
where the correlation of ω and γ becomes apparent. The transition to air is made by comparing the density of air to the density of the high explosive. Given that A >> B while R1 = O(R2 ), the decay of the first term in Eqn.(1) with increasing v (decreasing ρ) is much faster. This implies that as v increases, we have p → Be−R2 v + ωρe .
(12)
The mixture of high explosive and air is considered as air when the effect of the B-term may be neglected, i.e. p = = Be−R2 v , pcj
(13)
where pcj denotes the Chapman-Jouget pressure and = O(10−3 ). Afterburning is modeled by adding energy via a burn coefficient λ that is obtained from λ,t = a
p p0
16
√ 1−λ ,
(14)
8
Rainald L¨ ohner et al.
where λ = 0 for the unburned state, and λ = 1 for the fully burned material. After updating λ, the energy released is added as follows: (ρe)|
n+1
n
= (ρe)| + ρQ(λn+1 − λn ) ,
(15)
where Q is the afterburn energy. Compared to the five unknowns required for the Euler equations with an ideal air equation of state, we require an additional two: the burn fraction b to determine which part of the material has ignited, and the afterburn coefficient λ. Both of these scalars have to be transported through the mesh. For a typical explosion simulation, the major portion of CPU time is required to simulate the burning material. This is because the pressures are very high, and so are the velocities of the fluid particles. Once the material has burned out, one observes a drastic reduction of pressures and velocities, which implies a dramatic increase in the allowable time-step. Even though shocks travel much larger distances, this post-burn ‘diffraction phase’ takes less CPU-time than the burn phase. In order to speed up the simulation, the portions of the grid outside the detonation region are deactivated. The detonation velocity provides a natural speed beyond which no information can travel. Given that the major loops in an unstructured-grid flow solver are processed in groups (elements, faces, edges, etc.) for vectorization, it seemed natural to deactivate not the individual edge, but the edge-group. In this way, all inner loops can be left untouched, and the test for deactivation is carried out at the group-level. The number of elements in each edge-, face-, or element-group is kept reasonably small (O(128)) in order to obtain the highest percentage of deactivated edges without compromising performance. The points are renumbered according to their ignition time assuming a constant detonation velocity. In this way, the work required for point-loops is minimized as much as possible. The edges and points are checked every 5-10 time-steps and activated accordingly. This deactivation technique leads to considerable savings in CPU at the beginning of a run, where the time-step is very small and the zone affected by the explosion only comprises a small percentage of the mesh.
3
CSD Solver
For the solid, a Lagrangian finite element code based on the large displacement formulation is used. Truss-, beam-, triangular and quadrilateral shells, as well as hexahedral solid elements have been incorporated. Innovative features of this structural dynamics code include adaptive refinement [29, 30], advanced contact algorithms [31] and the ability to fail elements or open cracks based on physics (i.e. not predetermined). At the beginning of each structural dynamics time-step, the loads from the fluid are updated. At the end of each structural dynamics time-step, the positions and velocities of the ‘wetted surfaces’ are extracted and transferred back to the flow solver.
Fluid-Structure Interaction Simulations Using Parallel Computers
4
9
Load and Position/Velocity Transfer
Optimal discretizations for the structure and the fluid are, in all probability, not going to be the same. As an example, consider an explosion in a room. For an accurate fluid solution, element sizes below 1cm have proven reliable. For the structure, shell or brick elements of 10cm are more than sufficient. There may be situations where fluid and structural surfaces are represented by abstractions of different dimensionality. A typical example here would be a commercial aircraft wing undergoing aeroelastic loads. For an accurate fluid solution using the Euler equations, a very precise surface representation with 60-120 points in the chordwise direction will be required. For the structure, a 20 × 40 mesh of plate elements may be more than sufficient. Any general fluid-structure coupling strategy must be able to handle efficiently the information transfer between different surface representations. This is not only a matter of fast interpolation techniques [17, 27], but also of accuracy, load conservation [6, 7], geometrical fidelity [7], and temporal synchronization [13, 7]. 4.1
Load Transfer and Conservation
When considering different mesh sizes for the CSD and CFD surface representation, the enforcement of accuracy in the sense of: σs (x) ≈ σf (x) and conservation in the sense of: f = σs ndΓ = σf ndΓ
(16)
(17)
proves to be non-trivial. The best way to date to handle this problem for similar surface representations is via an adaptive Gaussian quadrature [6]. In some cases, the conservation of forces may not be as important as conservation of other physically meaningful quantities, e.g. work or torque. It is important to note that conservation of only one of these: force, torque or work, can be guaranteed. The reliable and accurate treatment different levels of abstraction or dimensionality between fluid and structures represents an open question to date. 4.2
Geometrical Fidelity
In many instances, the structural model will either be coarse as compared to the fluid model, or may even be on a different modeling or abstraction level. On the other hand, it is the structural model that dictates the deformation of the fluid surface. It is not difficult to see that an improper transfer of structural deformation to the fluid surface mesh can quickly lead to loss of geometrical fidelity. For the cases considered here, the fluid surface was ‘glued’ to the surface of the structure. For ‘non-glued’ fluid and structural surfaces a number of recovery techniques have been proposed to date [9, 18, 7]. However, we consider
10
Rainald L¨ ohner et al. Current CAD Data Base (Analytical + Discrete Data)
Remove Discrete Data
Updated CAD Data Base (Analytical + Discrete Data)
Updated CSD Data
Recover Discrete CAD Data
Fig. 2. Automatic Surface Reconstruction
the proper treatment of surface deformation for ‘non-glued’ fluid and structural surfaces, as well as some form of error indicator to warn the unsuspecting user, to be areas deserving further study.
5
Topology Change
Suppose that due to cracking, failure, spalation, etc., the ‘wetted surface’ of the structure has been changed. This new surface, given by a list of points and faces, has to be matched with a corresponding fluid surface. The fluid surface data typically consists of surface segments defined by analytical functions that do not change in time (such as exterior walls, farfield boundaries, etc.), and surface segments defined by triangulations (i.e. discrete data) that change in time. These triangulations are obtained from the ‘wetted structure surface’ at every time-step. When a change in topology is detected, the new surface definition is recovered from the discrete data, and joined to the surfaces defined analytically, as indicated in Figure 2. The discrete surface is defined by a support triangulation, with lines and end-points to delimit its boundaries. In this sense, the only difference with analytically defined surfaces is the (discrete) support triangulation. The patches, lines and end-points of the ‘wetted structure surface’ are identified by comparing the unit surface normals of adjacent faces. If the scalar product of them lies below a certain tolerance, a ridge is defined. Corners are defined as points that are attached to: – Only one ridge; – More than two ridges; or – Two ridges with considerable deviation of unit side-vector.
Fluid-Structure Interaction Simulations Using Parallel Computers
11
L6 L5 S1 L1 L7
L2
L4
S2 L3
Fig. 3. Discrete Surface Recovery
Between corners, the ridges form discrete lines. These discrete lines either separate or are embedded completely (i.e. used twice) in discrete surface patches. Figure 3 sketches the recovery of surface features and the definition of discrete surface patches for a simple configuration. For more information, see [20]. For the old surface definition data set, the surface patches attached to wetted structure surfaces are identified and all information associated with them is discarded. The remaining data is then joined to the new wetted structure surface data, producing the updated surface definition data set. This data set is then used to generate the new surface and volume grids. The surface reconstruction procedure may be summarized as follows: – For the Updated Discrete Data, Obtain: • Surface Patches + B.C. • Lines • End-Points • Sources – For the Old Analytical+Discrete Data: • Remove Discrete Data • Reorder Arrays – Merge: • Old Analytical Data • Updated Discrete Data A topology change as described above will imply a complete surface mesh regeneration, with subsequent flowfield mesh regeneration. If topology change occurs frequently, this can represent a considerable portion of the total computing cost. A way that has proven effective in reducing this cost is to identify which surfaces have actually changed, and to regenerate locally only the regions where changes have occurred. This local mesh generation procedure may be summarized as follows:
12
Rainald L¨ ohner et al.
– For the Updated and Old Surface Data: • Compare Surface Patches • Compare Lines • Compare End-Points – Remove from the Old Surface Data: • Surfaces that Have Disappeared • Lines that Have Disappeared • Points that Have Disappeared – Remove from the Current Mesh: • Faces of Surfaces that Have Disappeared • Elements Close to These Faces – Obtain Surface Mesh for New Surfaces – Locally Remesh Close to New Surfaces The procedure is shown schematically in Figure 4.
6
Scalability on Parallel Computers
From an operational point of view, the fluid-structure simulations considered here are characterized by: – Accent on complex physics: Accuracy in the modeling of the important physical phenomena is much more important than raw speed. While some operational runs require very fast turnaround, most of the runs can be performed in a few days without affecting negatively the analysis process. This, speed by itself is not the critical factor. For complex geometry and complex physics, data assimilation (geometry, boundary conditions, materials, etc.) and input, as well as the evaluation of results, requires many more man-hours than the actual run. a) Current CAD and Mesh
b) New CAD Data
c) After Correlation of CAD−Data and Local Element Removal
d) Completed Mesh
Fig. 4. Topology Change With Local Remeshing
Fluid-Structure Interaction Simulations Using Parallel Computers
13
– Interaction of several field solvers: The simulations are, by their nature, multidisciplinary (CFD, CSD, Transfer Library); at present, CFD consumes the vast majority of CPU time; this, however, may change if larger and more sophisticated CSD models are employed; – Potentially large load imbalances: Remeshing (for regions with distorted elements and/or topology change [15, 23]), adaptive mesh refinement (for accuracy [16]) particle-mesh techniques (for afterburn) and contact detection and force evaluation [10] are but four of many potential sources of severe load imbalances. In order to obtain sustainable efficiency for distributed memory machines, dynamic load balancing is indispensable. – Limited size: Experience indicates that the majority of the simulations can be performed with less than 108 elements. This implies that the useful number of parallel processors is limited (< 128). With the appearance of sufficiently powerful departmental servers that allow shared memory parallel processing (e.g. SGI-O3K, IBM-SP4), blast-induced fluid-structure simulation has become a very tractable problem. If we add to this list the fact that: – Some parts (e.g. remeshing) can be quite tedious to code, debug and maintain on distributed memory parallel machine; – Distributed memory parallel computing, with its many domain decompositions (CFD mesh, CFD particles, CSD mesh, CSD contact, CFD/CSD transfer), multiple file structures, represents an extra burden or ‘pain factor’ for the user; and – Distributed memory parallel computing implies extra development effort and debug time (that are purely artificial and complicate most algorithms), it is not difficult to see why the shared memory paradigm has remained highly popular. From cost considerations alone, parallelizing 99% of all operations required by these codes is highly optimistic. This implies that the maximum achievable gain will be restricted to O(1 : 100). There exist a number of platforms already that allow for efficient shared memory processing with O(100) processors. Instead of the spatial or logical decompositions encountered in typical distributed memory applications, for shared memory machines renumbering techiques [22, 26] play an important role in achieving acceptable scaling. Figure 5 shows the performance of FEFLO, the fluid code used in the present work, on a variety of common US-DOD-HPC platforms. One can see that the speedup obtained using shared and distributed memory approaches is similar.
7
Examples
We show three cases to illustrate the range of problems covered by the capabilities developed. The coupling complexity increases with each example. For the first example, the bodies move freely, based on the loads of the structure.
14
Rainald L¨ ohner et al.
32 O2KSH O2KMPI SP2MPI DAXMPI SP3SH Ideal
Speedup
16
8
4
2
1
1
2
4 8 Nr. of Processors
16
32
Fig. 5. Performance of FEFLO on Different Platforms
The last two examples show structural failure, rupture and topology change. All cases were run on SGI Origin 2000 machines in shared memory mode, using up to 64 processors. 7.1
Weapon Fragmentation
As the first example, we consider a weapon fragmentation experiment. The weapon used consisted of a thick wall cylinder, serrated into 32 rows of fragments, 16 fragments per row, for a total of 512 ‘small’ fragments, and thick nose and tail plates. While the cutting pattern was identical for all rows, the pattern was rotated between rows by angles that produced no symmetric pattern. Thus, the whole weapon had to be modeled. Each fragment weighed about 380 grams. The serrated weapon had tabs that kept the minimum distance between the fragments to 0.5 mm, while the average size of each fragment was a 1 − 4 cm per side. In contrast, the room size was several meters. This huge disparity in dimensions required the use of sources attached to each flying body in order to ensure a uniform, high-resolution mesh about each fragment. Figure 6a shows the computational domain as well as snapshots of the surface grid employed. Each of the fragments is treated as a separate, freely flying rigid body whose velocity and trajectory were determined by integrating the six ordinary differential equations describing the balance of forces and moments. The mesh used for the CFD domain is body conforming and moves along with the fragments. Figures 6b6i show a sequence of snapshots of pressure, detonation products velocity, and
Fluid-Structure Interaction Simulations Using Parallel Computers
15
Fig. 6a. Computational Domain
Fig. 6b,c. Pressure Contours and Fragment Velocities at t = 72 µsec and t = 139.7 µsec
16
Rainald L¨ ohner et al.
Fig. 6d,e. Pressure Contours, Detonation Products, and Fragment Velocity at t = 225.7 µsec
Fig. 6f,g. Pressure Contours, Detonation Products, and Fragment Velocity at t = 297.3 µsec
Fluid-Structure Interaction Simulations Using Parallel Computers
17
Fig. 6h,i. Pressure Contours and Fragment Velocities at t = 501 µsec and t = 900 µsec fragment and mesh velocity, at several times. The pressure results are shown for a planar cut. Note that this is not a plane of symmetry. The figures show the detonation wave, the accelerating fragments, and the sharp capturing of the shock escaping through the opening gaps. The results indicate that acceleration to the final velocity takes about 120 microseconds. The predicted velocities fall within the 10% error band of the experimental measurements. Given the complexity of the physical phenomena being modeled, such a correlation with experimental data is surprisingly good. Typical meshes for this simulation were of the order of 3-4 Mtets. The simulation was carried out on an 8 processor SGI Origin 2000, and took approximately 2 weeks. For more details, see [5]. 7.2
Generic Weapon Fragmentation
The second example shows a fully coupled CFD/CSD run. The structural elements were assumed to fail once the average strain in an element exceeded 60%. At the beginning, the fluid domain consisted of two separate regions. These regions connected as soon as fragmentation started. In order to handle narrow gaps during the break-up process, the failed structural elements were shrunk by a fraction of their size. This alleviated the time-step constraints imposed by small elements without affecting the overall accuracy. The final breakup led to approximately 1200 objects in the flowfield. Figure 7 shows the fluid pressure and surface velocity of the structure at three different times during the simulation. As before, the CFD is body conforming and moves according to the fragments. Typical meshes for this simulation were of the order of 8.0 Mtets, and the simulations required of the order of 50 hours on the SGI Origin 2000 running on 32 processors.
18
Rainald L¨ ohner et al.
Fig. 7a. Pressure at Different Times
Fig. 7b. Surface Velocity at Different Times
Fluid-Structure Interaction Simulations Using Parallel Computers
19
Fig. 8a. Surface Pressure at Different Times
7.3
Shock Interaction with a Ship Hull
The third example shows another fully coupled CFD/CSD run. In this case, an explosion close to a generic ship hull was considered. The structure was modeled with quadrilateral shell elements. The structural elements were assumed to fail once the average strain in an element exceeded 60%. As the shell elements fail, the fluid domain underwent topological changes. The damage is rather severe, and strict contact could not be maintained by the CSD solver. After several unsuccessful attempts using the body conforming, moving mesh option, the case was finally run using the embedded surface technique. The CFD mesh had approximately 16.0 Mtets, and was refined in the regions where the initial shock impact occurs. Figure 8 shows the pressure contours on the surface, as well as the velocity of the structure, at several times during the run. The influence of walls can clearly be discerned for the velocity. Note also the failure of the structure, and the invasion of high pressure into the chamber.
8
Conclusions
A methodology to simulate large-scale fluid-structure interaction problems on parallel machines has been developed. Particular emphasis was placed on shockstructure interaction problems. For the fluid, a high-resolution FEM-FCT solver based on unstructured, moving, body fitted grids is used. For the structure, a Lagrangian large-deformation finite element code is employed. The coupled system is solved using a loose coupling algorithm, with position and velocity interpolation and force projection. The described methodology has been maturing
20
Rainald L¨ ohner et al.
Fig. 8b. Surface Velocity at Different Times
Fig. 8c. Plane Cut at T=0.000255 (Pressure, Grid)
Fluid-Structure Interaction Simulations Using Parallel Computers
21
rapidly over the last years, and is used on a routine basis to predict the complex phenomena encountered in shock-structure interaction problems. Areas of current research include: – Time-accurate implicit solvers for the drag phase; – RANS griding for arbitrary body shapes; – Automatic element size reduction for gaps and narrow openings during topology change; – Parallel grid generation for large number of processors; – Failure criteria for concrete based in first principle physics; – ‘Hard’ contact algorithms that guarantee no penetration; and – Further parallelization of fluid, structure and coupling libraries.
Acknowledgments This work was partially supported by DTRA, with Dr. Michael Giltrud as the technical monitor, as well as AFOSR, with Dr. Leonidas Sakell as the technical monitor.
References [1] M. J. Aftosmis, M. J. Berger and G. Adomavicius - A Parallel Multilevel Method for Adaptively Refined Cartesian Grids with Embedded Boundaries; AIAA-000808 (2000). 5 [2] J. D. Baum. H. Luo and R. L¨ ohner - Numerical Simulation of a Blast Inside a Boeing 747; AIAA-93-3091 (1993). 3 [3] J. D. Baum, H. Luo and R. L¨ ohner - Numerical Simulation of Blast in the World Trade Center; AIAA-95-0085 (1995). 3 [4] J. D. Baum, H. Luo, R. L¨ ohner, C. Yang, D. Pelessone and C. Charman - A Coupled Fluid/Structure Modeling of Shock Interaction with a Truck; AIAA-96-0795 (1996). 5 [5] J. D. Baum, H. Luo and R. L¨ ohner - The Numerical Simulation of Strongly Unsteady Flows With Hundreds of Moving Bodies; AIAA-98-0788 (1998). 5, 17 [6] J. R. Cebral and R. L¨ ohner - Conservative Load Projection and Tracking for Fluid-Structure Problems; AIAA J. 35, 4, 687-692 (1997a). 9 [7] J. R. Cebral and R. L¨ ohner - Fluid-Structure Coupling: Extensions and Improvements; AIAA-97-0858 (1997b). 9 [8] J. R. Cebral and R. L¨ ohner - Interactive On-Line Visualization and Collaboration for Parallel Unstructured Multidisciplinary Applications; AIAA-98-0077 (1998). [9] G. P. Guruswamy and C. Byun - Fluid-Structural Interactions Using NavierStokes Flow Equations Coupled with Shell Finite Element Structures; AIAA93-3087 (1993). 9 [10] E. Haug, H. Charlier, J. Clinckemaillie, E. DiPasquale, O. Fort, D. Lasry, G. Milcent, X. Ni, A. K. Pickett and R. Hoffmann - Recent Trends and Developments of Crashworthiness Simulation Methodologies and their Integration into the Industrial Vehicle Design Cycle; Proc. Third European Cars/Trucks Simulation Symposium (ASIMUTH), Oct. 28-30 (1991). 13
22
Rainald L¨ ohner et al.
[11] S. L. Karman - SPLITFLOW: A 3-D Unstructured Cartesian/ Prismatic Grid CFD Code for Complex Geometries; AIAA-95-0343 (1995). 5 [12] A. M. Landsberg and J. P. Boris - The Virtual Cell Embedding Method: A Simple Approach for Gridding Complex Geometries; AIAA-97-1982 (1997). 5 [13] M. Lesoinne and Ch. Farhat - Geometric Conservation Laws for Flow Problems With Moving Boundaries and Deformable Meshes, and Their Impact on Aeroelastic Computations; Comp. Meth. Appl. Mech. Eng. 134, 71-90 (1996). 9 [14] R. L¨ ohner, K. Morgan, J. Peraire and M. Vahdati - Finite Element Flux-Corrected Transport (FEM-FCT) for the Euler and Navier-Stokes Equations; ICASE Rep. 87-4, Int. J. Num. Meth. Fluids 7, 1093-1109 (1987). 4 [15] R. L¨ ohner - Three-Dimensional Fluid-Structure Interaction Using a Finite Element Solver and Adaptive Remeshing; Computer Systems in Engineering 1, 2-4, 257-272 (1990). 5, 13 [16] R. L¨ ohner and J. D. Baum - Adaptive H-Refinement on 3-D Unstructured Grids for Transient Problems; Int. J. Num. Meth. Fluids 14, 1407-1419 (1992). 13 [17] R. L¨ ohner - Robust, Vectorized Search Algorithms for Interpolation on Unstructured Grids; J. Comp. Phys. 118, 380-387 (1995a). 9 [18] R. L¨ ohner, C. Yang, J. Cebral, J. D. Baum, H. Luo, D. Pelessone and C. Charman - Fluid-Structure Interaction Using a Loose Coupling Algorithm and Adaptive Unstructured Grids; AIAA-95-2259 [Invited] (1995b). 9 [19] R. L¨ ohner and Chi Yang - Improved ALE Mesh Velocities for Moving Bodies; Comm. Num. Meth. Eng. 12, 599-608 (1996a). 7 [20] R. L¨ ohner - Regridding Surface Triangulations; J. Comp. Phys. 126, 1-10 (1996b). 11 [21] R. L¨ ohner - Extensions and Improvements of the Advancing Front Grid Generation Technique; Comm. Num. Meth. Eng. 12, 683-702 (1996c). [22] R. L¨ ohner - Renumbering Strategies for Unstructured-Grid Solvers Operating on Shared-Memory, Cache-Based Parallel Machines; Comp. Meth. Appl. Mech. Eng. 163, 95-109 (1998). 13 [23] R. L¨ ohner, Chi Yang, J. D. Baum, H. Luo, D. Pelessone and C. Charman - The Numerical Simulation of Strongly Unsteady Flows With Hundreds of Moving Bodies; Int. J. Num. Meth. Fluids 31, 113-120 (1999a). 5, 13 [24] R. L¨ ohner, C. Yang, J. Cebral, J. D. Baum, H. Luo, E. Mestreau, D. Pelessone and C. Charman - Fluid-Structure Interaction Algorithms for Rupture and Topology Change; Proc. 1999 JSME Computational Mechanics Division Meeting, Matsuyama, Japan, November (1999b). 5 [25] R. L¨ ohner - A Parallel Advancing Front Grid Generation Scheme; AIAA-00-1005 (2000). [26] R. L¨ ohner and M. Galle - Minimization of Indirect Addressing for Edge-Based Field Solvers; AIAA-02-0967 (2002). 13 [27] N. Maman and C. Farhat - Matching Fluid and Structure Meshes for Aeroelastic Computations: A Parallel Approach; Computers and Structures 54, 4, 779-785 (1995). 9 [28] J. E. Melton, M. J. Berger and M. J. Aftosmis - 3-D Applications of a Cartesian Grid Euler Method; AIAA-93-0853-CP (1993). 5 [29] D. Pelessone and C. M. Charman - Adaptive Finite Element Procedure for NonLinear Structural Analysis; 1995 ASME/JSME Pressure Vessels and Piping Conference, Honolulu, Hawaii, July (1995). 8 [30] D. Pelessone and C. M. Charman - An Adaptive Finite Element Procedure for Structural Analysis of Solids; 1997 ASME Pressure Vessels and Piping Conference, Orlando, Florida, July (1997). 8
Fluid-Structure Interaction Simulations Using Parallel Computers
23
[31] D. Pelessone and C. M. Charman - A General Formulation of a Contact Algorithm with Node/Face and Edge/Edge Contacts; 1998 ASME Pressure Vessels and Piping Conference, San Diego, California, July (1998). 8 [32] R. B. Pember, J. B. Bell, P. Colella, W. Y. Crutchfield and M. L. Welcome - An Adaptive Cartesian Grid Method for Unsteady Compressible Flow in Irregular Regions - sl J. Comp. Phys. 120, 278 (1995). 5 [33] J. J. Quirk - A Cartesian Grid Approach with Hierarchical Refinement for Compressible Flows; NASA CR-194938, ICASE Report No. 94-51, (1994). 5 [34] R. Ramamurti and R. L¨ ohner - Simulation of Flow Past Complex Geometries Using a Parallel Implicit Incompressible Flow Solver; pp. 1049,1050 in Proc. 11th AIAA CFD Conf. , Orlando, FL, July (1993). [35] D. Sharov, H. Luo, J. D. Baum and R. L¨ ohner - Time-Accurate Implicit ALE Algorithm for Shared-Memory Parallel Computers; First International Conference on Computational Fluid Dynamics, Kyoto, Japan, July 10-14 (2000). 5 [36] D. de Zeeuw and K. Powell - An Adaptively-Refined Cartesian Mesh Solver for the Euler Equations; AIAA-91-1542 (1991). 5
Investigation of Turbulent Flame Kernels Using DNS on Clusters Best Student Paper Award: First Prize Marc Lange High-Performance Computing-Center Stuttgart (HLRS) Allmandring 30, D-70550 Stuttgart, Germany Phone: +49 711 685 2507 Fax: +49 711 685 7626 [email protected]
Abstract. A parallel code for the direct numerical simulation (DNS) of reactive flows using detailed chemical kinetics and multicomponent molecular transport is presented. This code has been used as a benchmark on several commodity hardware clusters including two large systems with current state-of-the-art processor and network technologies, i. e. up to 180 Alpha EV68 833 MHz and up to 400 AthlonMP 1.4 GHz CPUs have been used, which are arranged in dual-nodes with Myrinet2000 interconnect in both cases. The other part of the paper presents the results of an investigation of flames evolving after induced ignition of a premixed mixture under turbulent conditions.
1
DNS of Turbulent Reactive Flows
Energy conversion in numerous industrial power devices like automotive engines or gas turbines is still based on the combustion of fossil fuels. In most applications, the reactive system is turbulent and the reaction progress is influenced by turbulent fluctuations and mixing in the flow. The understanding and modeling of turbulent combustion is thus vital in the conception and optimization of these systems in order to achieve higher performance levels while decreasing the amount of pollutant emission. During the last few years, direct numerical simulations (DNS), i. e. the computation of time-dependent solutions of the Navier-Stokes equations for reacting ideal gas mixtures (as given in Sect. 2), have become an important tool to study turbulent combustion at a detailed level. As DNS does not make use of any turbulence or turbulent combustion models, this technique may be interpreted as high-resolution (numerical) experiments, enabling new investigations on fundamental mechanisms in turbulence-chemistry-interaction and aiding in the refinement of turbulent combustion models. However, many of the DNS carried out so far have used simple one-step chemistry, although important effects cannot be captured by simulations with such oversimplified chemistry models [1, 2]. By making efficient use of the computational power provided by massively parallel systems, it is possible to perform J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 24–38, 2003. c Springer-Verlag Berlin Heidelberg 2003
Investigation of Turbulent Flame Kernels Using DNS on Clusters
25
DNS of reactive flows using detailed chemical reaction mechanisms at least in two spatial dimensions. Nevertheless, computation time is still the main limiting factor for the DNS of reacting flows, especially when applying detailed chemical schemes.
2
Governing Equations
In the context of DNS, chemically reacting flows are described by a set of coupled partial differential equations expressing the conservation of total mass, chemical species masses, momentum and energy [3]. Using tensor notation1 these equations can be written in the form ∂ ∂(uj ) + ∂t ∂xj ∂(Yα ) ∂(Yα uj ) + ∂t ∂xj ∂(ui ) ∂(ui uj ) + ∂t ∂xj ∂e ∂((e + p)uj ) + ∂t ∂xj
=0 ,
(1)
∂(Yα Vα,j ) + Mα ω˙ α ∂xj ∂τij ∂p = − , ∂xj ∂xi ∂(uj τkj ) ∂qj = − , ∂xk ∂xj =−
(α = 1, . . . , NS )
,
(2) (3) (4)
wherein is the density and ui the velocity component in ith coordinate direction. Yα , V α and Mα are the mass fraction, diffusion velocity and molar mass of the species α. NS is the number of chemical species occurring in the flow, τij denotes the viscous stress tensor and p the pressure, q is the heat flux and e is the total energy given by NS ui ui e= + hα Yα − p (5) 2 α=1 with hα being the specific enthalpy of the species α. The chemical production rate ω˙ α of the species α, which appears in the source-term on the right-hand sides of the NS species mass equations (2), is given as the sum over the formation rate equations for all NR elementary reactions ω˙ α =
NR λ=1
(p) kλ (ναλ
−
(r) ναλ )
NS
ν
(r)
cααλ
.
(6)
α=1
The rate coefficients kλ of the elementary reactions are given by a modified Arrhenius law Eaλ kλ = Aλ T bλ exp − , (7) RT 1
Summation convention shall be applied to roman italic indices (i, j, k) only, i. e. no summation is carried out over greek indices (α, λ).
26
Marc Lange
the parameters Aλ and bλ of the pre-exponential factor and the activation energy Eaλ are determined by a comparison with experimental data [4]. The chemical reaction mechanism for the H2 /O2 /N2 system which has been used in all simulations presented in this paper contains NS = 9 species and NR = 37 elementary reactions [5]. The viscosity and multicomponent diffusion velocities are computed using standard formulae [6, 7], thermodynamical properties are computed using fifthorder fits of experimental measurements [4]. The set of equations is complemented by the ideal-gas state-equation p=
RT M
,
(8)
wherein R is the gas constant and M the mean molar mass of the mixture.
3
Structure of the Parallel DNS Code
Due to the broad spectrum of length and time scales apparent in turbulent reactive flows, a very high resolution in space and time is needed to solve the system of equations given in the prvious section. The computation of the chemical sourceterms and the multicomponent diffusion velocities are the most time-consuming parts in such DNS. Therefore, almost all DNS carried out so far has been (at least) restricted to the use of simplified models (e. g. one global reaction and equal diffusivities) or to two-dimensional simulations. Even with these restrictions it is crucial to make efficient use of HPC-systems, to be able to carry out DNS of reactive flows in acceptable time. A code has been developed for the DNS of reactive flows on parallel computers with distributed memory using message-passing communication [8, 9, 2]. In favor of being able to include detailed models for the chemical reaction kinetics and the molecular transport as outlined in Sect. 2, only two-dimensional simulations are performed. The spatial discretization in DNS codes is typically done by spectral methods or high-order finite-difference schemes. The main advantages of the latter ones are a greater flexibility with respect to boundary-conditions and the possibility of a very efficient parallelization, as in a fully explicit formulation no global data-dependencies occur. We chose a finite-difference scheme with sixth-order central-derivatives, avoiding numerical dissipation and leading to very high accuracy. The integration in time is carried out using a fourth-order fully explicit Runge-Kutta method with adaptive time-stepping. The control of the timestep is based on (up to) three independent criteria: A Courant-Friedrichs-Lewy (CFL) criterion and a Fourier criterion for the diffusion terms are checked to ensure the stability of the integration and an additional accuracy-control of the result for one or more selectable variables is obtained through timestep doubling. The fully explicit formulation leads to a parallelization strategy, which is based on a regular two-dimensional domain-decomposition. For a given computational grid and number of processors it is tried to minimize the length of
Investigation of Turbulent Flame Kernels Using DNS on Clusters
27
Table 1. Scaling behavior on a Cray T3E-900 for a simulation with 9 species and 37 reactions on a 5442 points grid # PEs speedup efficiency / %
1
4
8
16
32
64
128
256
512
1.0 100.0
4.3 106.6
8.1 100.7
15.9 99.2
30.5 95.3
57.9 90.4
108.7 84.9
189.0 73.8
293.6 57.4
the subdomain boundaries and thus the amount of communication. After the initial decomposition, each processor node controls a rectangular subdomain of the global computational domain. By using halo-boundaries, an integration step on the subdomain can be carried out independently from the other nodes. After each integration step, the values in the three points wide halo-regions are updated by point-to-point communications with the neighboring nodes. MPI is used for the communication. Besides the computations on clusters discussed in this paper, our main platform for production runs have been Cray T3E systems. Table 1 lists the speedups and corresponding parallel efficiencies achieved on a Cray T3E-900 for a test case with a constant global size of 544 × 544 grid points [10]. The average performance per PE within this simulation (including I/O) for the computation using 64 processors was 86.3 MFLOP/s. For a problem with a scaled size of 32 × 32 grid points per processor, a speedup of 197 has been obtained using 256 PEs of a Cray T3E-900 corresponding to an efficiency of 79 % .
4
Performance on Clusters
A big trend in parallel computing today is the use of clustered systems. By using commodity hardware for the nodes as well as for the network, clusters of PCs can provide an excellent performance/price ratio – at least for problems which they are well suited for. Therefore, there is a broad interest in the performance behavior of real world applications on such clusters. We have studied the performance of several clusters with different types of nodes and network hardware for our application. As a starting point of this investigation, benchmarks for different problem sizes using one and 16 processors of the following clusters have been performed: Alpha-Myrinet (Alpha-M) Nodes: 64 Compaq DS10 workstations, each having one 466 MHz Alpha 21264 (EV6) processor (2 MB cache) and 256 MB RAM. Interconnect: Myrinet. Dual-Pentium-Myrinet (DualP3-M) Nodes: 8 IBM dual-processor Netfinity servers, each having two 600 MHz Pentium III processors and 896 MB RAM. Interconnect: Myrinet. Pentium-Ethernet (P3-E) Nodes: 16 PCs with 660 MHz Pentium III processors and 512 MB RAM. Interconnect: 100 MBit Ethernet.
28
Marc Lange
Table 2. Monoprocessor performance for different problem sizes related to the performance of the Cray T3E-1200 solving the 50 × 50 problem Problem Size T3E-1200 Alpha-M DualP3-M P3-E
50 × 50
100 × 100
200 × 200
1.00 2.39 1.21 1.28
1.11 2.40 1.24 1.25
1.19 2.33 1.20 1.22
400 × 400 1.23 (not enough memory)
1.19 1.26
Table 2 lists the relative performance of one processor of each of these systems for a computation on a grid with 502 , 1002 , 2002, 4002 points, respectively. The relative performance is obtained from normalizing the inverse of the CPU-time needed to carry out a fixed number of timesteps with the problem size and dividing this value by the corresponding value of a Cray T3E-1200 PE for the 50 × 50 points case. The relative performance of a T3E-900 PE for the 502 grid points computation is 0.82 . The PE of the T3E performs better for the larger grids whereas the differences of the other systems with respect to the differently sized problems remain within a few percent. The behavior of the T3E may be attributed to the stream buffers, but does not hold for even larger grid sizes as can be seen from the results given in the last section: the superlinear speedup for 4 PEs (see Table 1) is possible only due to the relative performance per PE being better for a problem size of 2722 than of 5442 grid points. The Pentium III systems are slightly faster in these tests than the Alpha 21164, and the Alpha 21264 is almost twice as fast as the Pentium III nodes. Of course the exact relations between the different nodes strongly depend on the levels of optimization performed by the compilers. In this case, the default compilers and switches called from the mpif90 command were used on the Linux-clusters, i. e. PGI V3.1-3 with -o on the Pentium III systems and Compaq Fortran CFAL 1.0 with -O4 on the Alpha-M. On the T3Es CF90 3.5 with -O3 has been used. In Table 3, the measured parallel efficiencies Es for 16 processors of each of the clusters are given for two problems with a fixed load per processor of 502 and 1002 grid points respectively. For the parallelization on the DP3-M cluster, at least from the application programmers point of view, no differences between intra-node communication and inter-node communication have been made. The Table 3. Scaled parallel efficiency (in percent) using 16 processors for two different loads per processor Problem Size per Processor T3E-1200 Alpha-M DualP3-M P3-E
50 × 50
100 × 100
91.0 80.5 83.0 77.5
92.0 90.4 81.1 85.7
Investigation of Turbulent Flame Kernels Using DNS on Clusters
29
Table 4. Time for communication and synchronization tc /s for 10 timesteps using 16 processors Problem Size per Processor T3E-1200 Alpha-M DP3-M (-o) P3-E
50 × 50
100 × 100
0.80 0.81 1.36 1.82
2.52 1.43 6.02 4.30
loads per processor chosen here correspond to real production runs, for which global grids with between 4002 and 10002 grid points are typical. The main result is, that all the systems tested deliver reasonable parallel performance – at least for our application and up to 16 processors. For our application, it is possible to calculate the overhead for the parallelization, i. e. the time spent for communication and synchronization, from the run-times of the serial run and the scaled parallel run [11]. Table 4 lists the time spent for communication and synchronization tc for a fixed number of timesteps using 16 processors. The number of messages in these tests is independent from the load per processor, and the length of the messages between neighboring nodes scales with the length of the domain-boundaries which is in these benchmarks of the same size as in typical production runs. From the times achieved on the Alpha-Myrinet cluster it can be seen, that for our application and 16 nodes, the Myrinet is only slightly slower than the T3E-network in the case with the smaller messages and 76 % faster in the case with the larger messages. As expected, the network performance of the Ethernet cluster is clearly lower than that of these two systems. The Dual-Pentium-Myrinet cluster suffers obviously from the very low parallel efficiency inside the nodes. The memory conflicts in the bus-based shared memory architecture also cause the performance of the dual-nodes to be more sensitive to the problem size. The main questions arising from these benchmarks concern the efficiency of using dual-nodes and the scaling behavior for large configurations. These are adressed by measurements performed on two large clusters with current stateof-the-art processor and interconnect technology: Dual-Alpha-Myrinet2000 (API-M2K) Nodes: 96 API NetWorks CS20 1U servers, each having two 833 MHz Alpha 21264B (EV68) processors (4 MB L2 cache) and 2 GB RAM. Interconnect: Myrinet 2000. Dual-Athlon-Myrinet2000 (HELICS) Nodes: 256 PCs with two 1.4 GHz AMD AthlonMP processors and 1 GB RAM. Interconnect: Myrinet 2000. Access to the first system has been granted by Cray Inc., the latter system has been installed at the Interdisciplinary Center for Scientific Computing (IWR) of Heidelberg University together with the University Computing Centers on Heidelberg (URZ) and Mannheim (RUM) in March 2002 (HEidelberger LInux Cluster System, see http://helics.iwr.uni-heidelberg.de/) and is currently
30
Marc Lange
Table 5. Monoprocessor performance for different problem sizes related to the performance of the Cray T3E-1200 solving the 50 × 50 problem Problem Size
50 × 50
100 × 100
200 × 200
400 × 400
4.40 4.22
4.18 3.87
3.67 3.84
2.69 3.69
API-M2K HELICS
one of the most powerful PC-clusters worldwide. Table 5 lists the relative performance of one processor of each of these two systems analogous to Table 2. Both processors perform at a similar level, but the decrease of the performance with increasing problem size is much stronger on the Alpha than on the Athlon system. Compaq Fortran 1.0-920 has been used on the API-M2K. As the HELICS system had just been installed and was still in its testing phase, another IA32Linux-cluster has been used as a development platform (http://www.hlrs.de/hwaccess/platforms/volvox/). The Intel Fortran Compiler ifc 6.0β has been used on this system to produce the binaries for the HELICS. To be able to distinguish the influence of the intra-node parallelization from the influence of the network performance, several test cases have been run using both processors of p/2 nodes as well as using only one processor on each of the p nodes used, where p is the number of processors to be used in the considered test case. In the following, this is denoted by the suffixes -dual and -single. The parallelization overhead times tc are listed in Table 6. In our benchmark, symmetric boundary conditions are used in the x-direction and periodic boundary conditions in the y-direction. In the cases using two processors with a load of N × N grid points per processor, the global grid has 2N points in the y-direction and the boundary between the subdomains is parallel to the x-axis. Due to the periodicity each of the processors has to exchange the values at two sides of its subdomain-boundary with the other processor. In the cases with 16 and more processors of course all processors, except for those at the left and right boundaries of the global domain, have to exchange the values on all four sides of their four subdomain-boundaries with the neighboring processors. Thus, the typical number of messages to be exchanged per processor in the two-processor case is half of that of the runs with 16 and 64 processors. The communication perTable 6. Parallelization overhead tc /s for 10 timesteps Number of Processors Problem Size per Processor T3E-1200 API-M2K-single API-M2K-dual HELICS-single HELICS-dual
2
2
16
16
64
50 × 50
400 × 400
50 × 50
100 × 100
50 × 50
0.35 0.19 0.57
21.67 3.91 36.55
0.80 0.56 0.33 0.75
2.52 1.33 1.68 0.85 4.36
0.80 0.81 0.58 0.36 0.80
Investigation of Turbulent Flame Kernels Using DNS on Clusters
31
Table 7. Overall time (in seconds) for computing ten timesteps on a 400 × 400 grid Number of Processors T3E-1200 API-M2K-single API-M2K-dual HELICS-single HELICS-dual
1
16
64
419.64 191.33 191.33 139.54 139.54
31.55 9.02 9.38 9.16 12.67
8.88 2.43 2.41 2.27 2.71
formance in the single-processor per node configurations for the Myrinet 2000 on the HELICS is clearly better than the communication performance of the T3E. The tests with two processors in the dual-case again show, that (for our test case) the parallelization overhead for the intra-node parallelization strongly depends on the problem size. The efficiency of the intra-node parallelization is higher on the API-M2K than on the HELICS. In addition, the strong decrease of this efficiency occurs at a larger problem size on the API-M2K as can be seen from the value of tc in the case with 1002 grid points on each of 16 processors which is still good for the API-M2K but quite poor for the HELICS. The computation times for computing ten timesteps in the case with 400×400 grid points (global) are given in Table 7. From these times, speedups for this problem of fixed size can easily be computed, but the resulting values are dominated by the dependence of the single processor performance from the problem size. The computing times however give an overview of the overall performance of these systems for our application. From the presented results it is already expected that these clusters provide an excellent platform for our application also for very large configurations. This is confirmed by the parallel efficiencies for a scaled problem with 50 × 50 grid points per processor as given in Table 8. As the time spent for global communication in our application during the normal integration in time is negligible, the scaled parallel efficiency should stay almost constant for runs with 16 and more processors as it is the case here. There is some global communication involved when gathering data for I/O, but this is not taken into account here as the accumulated time for gathering data and writing it to disk is only about one percent of the total time of a typical production run. From the benchmarks presented, it is clear that the performance of a configuration with p single-processor nodes is superior to a configuration with p/2 dualnodes. However, the latter configuration is typically much less expensive and the price/performance ratio is often better for a system with dual-nodes. The main question in this context is therefore often, if it is worthwhile for a system with a given number of nodes to invest into dual-nodes instead of single-nodes. If we compare the computation time for a fixed problem size of 400 × 400 grid points when using both processors of the dual nodes with that when only one processor per node is used, we find a performance gain of 55.3 % from the additional CPUs
32
Marc Lange
Table 8. Scaled parallel efficiency (in %) for a load of 50 × 50 grid points per processor Number of Processors T3E-1200 API-M2K-single API-M2K-dual HELICS-single HELICS-dual
16
64
180
400
91.0 76.7 85.2 71.7
90.6 75.3 75.9 84.0 70.5
74.6 83.5 69.9
70.1
for 64 nodes of HELICS (i. e. the run time is reduced by a factor of 1/1.553). When using 256 processors on 128 nodes instead of only one processor on each of the 128 nodes for the same case with 4002 grid points the performance gain is 69.6 %. In the following section, results of DNS of turbulent flame kernels are presented. Two simulations with 704 × 704 grid points each have been performed, one using 256 PEs of a Cray T3E-900 and one using 256 processors of HELICS (128 nodes). The computation time for advancing the solution over a physical time of 0.1 ms (from t = 0.45 ms to t = 0.55 ms) was 14857 seconds on the T3E-900 and 3639 seconds on HELICS. These times include writing four full solutions to disk, i. e. all local values of the main physical variables were saved after every 0.025 ms. About 13600 timesteps were computed for each of these parts of the integrations in time with the difference of the number of timesteps in both simulations (caused by the slightly different initial conditions) being less than 0.3 %.
5
Investigation of Turbulent Flame Kernels
Induced ignition and the following evolution of premixed turbulent flames is a phenomenon of large practical importance, e. g. in Otto engine combustion and safety considerations. DNS studies of this process have been performed using simple one-step chemistry in a model configuration of an initially uniform premixed gas under turbulent conditions which is ignited by an energy source in a small region at the center of the domain [12]. Our investigation extends these studies by using a detailed reaction mechanism (9 species, 37 elementary reactions) and a detailed transport model including thermodiffusion. The Soret effect (i. e. diffusion caused by temperature gradients) has often been neglected in simulations of reacting flows with multicomponent transport, but we found it to be of importance in onedimensional simulations of induced ignition of laminar H2 /O2 /N2 mixtures. In the simulations carried out in this study, a cold (T = 298 K) uniform mixture consisting of 23.3 % H2 , 11.7 % O2 , and 65 % N2 (mole fractions) has been superimposed with a turbulent flow field computed by inverse FFT from a vonK´ arm´ an-Pao-spectrum with randomly chosen phases. The mixture is ignited by
Investigation of Turbulent Flame Kernels Using DNS on Clusters
p (Pa):
96000
100000
104000
108000
33
112000
Fig. 1. Temporal evolution of pressure and temperature during the ignition of a turbulent hydrogen-air mixture
34
Marc Lange
Fig. 2. Temperature (top) and vorticity (bottom) in a flame (Kernel A) evolving after induced ignition of a turbulent mixture at t = 0.2 ms, t = 0.4 ms, and t = 0.6 ms (left to right) an energy-source in a small round region at the center of the domain, which is active during the first 15 µs of the simulation. Above a minimum ignition energy an expanding flame kernel is observed. The first phase of such a DNS is shown in Fig. 1. During the time in which the energy-source is active, the mixture at the center of the domain heats up to about 3100 K, radicals are formed, and the mixture ignites. A pressure wave is observed which propagates outwards towards the boundaries of the domain. Non-reflecting outflow conditions based on characteristic wave relations [13] are imposed on all boundaries to allow the pressure wave to leave the domain without disturbing the solution as can be seen from Fig. 1. Figure 2 shows the temporal evolution of one such flame kernel (Kernel A) in a turbulent flow field after the ignition. The first row of images shows the temperature at t = 0.2 ms, t = 0.4 ms, and t = 0.6 ms, respectively. In the row below, the temporal evolution of vorticity is shown. The lack of a vortex-stretching mechanism in two-dimensional simulations of decaying turbulence leads to an inverse cascade with growing structures. There is a very strong damping of the turbulence in the hot region of the burnt gas due to the high viscosity. A comparison shows a good qualitative agreement of the DNS presented with the results of an experimental investigation of turbulent flames performed under similar conditions [14]. A DNS of a second flame kernel (Kernel B) has been carried out with the following two changes: 1. The energy-source causing the ignition has a smaller
Investigation of Turbulent Flame Kernels Using DNS on Clusters
35
Fig. 3. H2 O2 mass fraction in a turbulent flame kernel (Kernel B) at t = 0.2 ms, t = 0.4 ms, and t = 0.6 ms
Fig. 4. OH mass fraction in two turbulent flame kernels (left: Kernel B, right: Kernel A) 0.6 ms after ignition caused by differently sized energy sources
diameter of dign = 0.4 mm instead of 1.6 mm while the amount of energy released per unit time and area is the same. 2. The individual realization of the turbulent flow field is different whereas it features the same statistical properties, i. e. the same input parameters – a root mean square of the velocity fluctuations of 3 m/s and an integral length scale of 2 mm – have been chosen for the initialization. Figure 3 shows the spatial distribution of the mass fraction of H2 O2 , a radical which is confined to a very thin layer in the flame front, at a physical time of 0.2 ms, 0.4 ms, and 0.6 ms. (The variations of the H2 O2 concentration in the reaction zone along the flame are more clearly visible in Fig. 6.) The hydroxy radical (OH) is a very important intermediate species which is often measured in experiments. The spatial distributions of this radical in the two flame kernels at t = 0.6 ms are shown in Fig. 4. High concentrations of OH are found behind the flame front in regions with negative curvature (convex towards the side with the burnt gas). In the case of the larger ignition source (Kernel A), OH has been formed in excess during the ignition phase leading to the peak of OH mass fraction near the center of the domain.
36
Marc Lange
An important quantity in turbulent combustion modeling is the reaction intensity. There are several possibilities to define the reaction intensity, e. g. the local heat release rate NS Yα ω˙ α hα (9) q˙ = α=1
can be used for this purpose. The temporal evolution of the total heat release rate, which is obtained by integrating q˙ over the computational domain, is shown in Fig. 5 for both turbulent flame kernels as well as for the corresponding laminar flames. The increase of total heat release is related to flame surface growth which is enhanced by the wrinkling of the flame front by the turbulence. The decrease of total heat release starting at t ≈ 0.7 ms in the case of Kernel A is caused by parts of the flame front leaving the computational domain. The total heat release as well as some other values like maximum heat release and maximum temperature in the domain are computed and written to disk by the application at frequent intervals during the run (here: every twentieth timestep). While this is feasible for some global physical variables, a quantitative analysis generally requires specialized tools for the postprocessing of the large and complex datasets generated by DNS. An extensible tool has been developed for this purpose which allows to extract several features like e. g. strain, stretch, curvature, and flame thickness from these datasets. This is illustrated in Fig. 6 which provides an insight into the correlations of some of these properties in the flame front of Kernel B. In this figure, the reaction zone thickness, the flame front curvature and the H2 O2 mass fraction along the flame front of Kernel B at t = 0.6 ms are shown for a subset of the DNS data with T = 1140 K ≈ (Tmax + Tmin )/2. The values of the reaction zone thickness are based on the local temperature gradient and given relative to the reaction zone thickness of the corresponding circular flame propagating in a laminar mixture with the same chemical composition. High concentrations of the H2 O2 radical in the reaction zone evidently occur in regions with negative curvature, i. e. convex towards the burnt gas. (The maxima of the H2 O2 mass fraction in flame normal direction are however located at temperatures around 625 K.) Such dependencies of the local chemical composition on other variables like e. g. strainrate or curvature constitute a central point in many combustion models. As the changes of the chemical composition in curved flame fronts are caused by the combined effects of curvature and preferential diffusion, these dependencies can only be found from the analysis of data obtained by high-resolution experiments or simulations with a sufficiently accurate description of molecular diffusion and chemical kinetics.
Acknowledgements The author would like to thank Cray Inc. for providing access to the DualAlpha(EV68)/Myrinet-2000 cluster and M. Wierse for carrying out the benchmark runs on this cluster. The author gratefully acknowledges the Interdisciplinary Center for Scientific Computing (IWR) at Heidelberg for granting him access to the Dual-AthlonMP/Myrinet-2000 cluster (HELICS).
Investigation of Turbulent Flame Kernels Using DNS on Clusters 200
30
-1
∫ q dA (kJ m s )
Y(H2O2) / ppm
dign = 1.6mm, turbulent (Kernel A) dign = 1.6mm, laminar dign = 0.4mm, turbulent (Kernel B) dign = 0.4mm, laminar
150
37
45
60
-1
1.20 1.15 1.10
100
rel. thickness
.
50
0
1.05 1.00 0.95
0 5
0.90
10
0.85
15 20
-7500 -5000 cu r v -2500
0
0.25
t (ms)
0.5
0.75
Fig. 5. Temporal evolution of overall heat-release in flame kernels evolving in turbulent and laminar mixtures after induced ignition
atu 0 re (ra 2500 d/m 5000 )
25 30
m s (
m)
35 40
Fig. 6. Reaction zone thickness, curvature, andH2 O2 mass fraction along the flame front of Kernel B at t = 0.6 ms
References [1] Mantel, T., Samaniego, J.-M.: Fundamental Mechanisms in Premixed Turbulent Flame Propagation via Vortex-Flame Interactions, Part II: Numerical Simulation, Combustion and Flame 118 (1999) 557–582 24 [2] Lange, M., Warnatz, J.: Investigation of Chemistry-Turbulence Interactions Using DNS on the Cray T3E, in High Performance Computing in Science and Engineering ’99 (Krause, E., J¨ ager, W., eds.) Springer (2000) 333–343 24, 26 [3] Bird, R. B., Stewart, W. E., Lightfoot, E. N.: Transport Phenomena, Wiley (1960) 25 [4] Warnatz, J., Maas, U., Dibble, R. W.: Combustion, 2nd ed., Springer (1999) 26 [5] Maas, U., Warnatz, J.: Ignition Processes in Hydrogen-Oxygen Mixtures, Combustion and Flame 74 (1988) 53–69 26 [6] Hirschfelder, J. O., Curtiss, C. F., Bird, R. B.: Molecular Theory of Gases and Liquids, Wiley (1954) 26 [7] Kee, R. J., Dixon-Lewis, G., Warnatz, J., Coltrin, M. E., Miller, J. A.: A Fortran Computer Code Package for the Evaluation of Gas-Phase Multicomponent Transport Properties, Tech. Rep. SAND86-8246 (1986) 26 [8] Th´evenin, D., Behrendt, F., Maas, U., Przywara, B., Warnatz, J.: Development of a Parallel Direct Simulation Code to Investigate Reactive Flows, Computers and Fluids 25(5) (1996) 485–496 26 [9] Lange, M., Th´evenin, D., Riedel, U., Warnatz, J.: Direct Numerical Simulation of Turbulent Reactive Flows Using Massively Parallel Computers, in Parallel Computing: Fundamentals, Applications and New Directions (D’Hollander, E., Joubert, G., Peters, F., Trottenberg, U., eds.) Elsevier (1998) 287–296 26 [10] Lange, M., Warnatz, J.: Direct Simulation of Turbulent Reacting Flows on the Cray T3E, in Proceedings of the 14th Supercomputer Conference ’99 in Mannheim (Meuer, H.-W., ed.) (1999) 27
38
Marc Lange
[11] Lange, M.: Direct Numerical Simulation of Turbulent Flame Kernels Using HPC, in High Performance Computing in Science and Engineering ’01 (Krause, E., J¨ ager, W., eds.) Springer (2002) 418–432 29 [12] Echekki, T., Poinsot, T. J., Baritaud, T. A., Baum, M.: Modeling and Simulation of Turbulent Flame Kernel Evolution, in Transport Phenomena in Combustion (Chan, S. H., ed.) vol. 2 Taylor & Francis (1995) 951–962 32 [13] Baum, M., Poinsot, T. J., Th´evenin, D.: Accurate Boundary Conditions for Multicomponent Reactive Flows, Journal of Computational Physics 116 (1995) 247–261 34 [14] Renou, B., Boukhalfa, A., Puechberty, D., Trinit´e, M.: Local Scalar Flame Properties of Freely Propagating Premixed Turbulent Flames at Various Lewis Numbers, Combustion and Flame 123 (2000) 507–521 34
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver for Incompressible Flow Problems Margreet Nool1 and Michael M. J. Proot2 1
2
CWI, P.O. Box 94079, 1090 GB Amsterdam, The Netherlands [email protected] Delft University of Technology, Kluyverweg 1, Delft, The Netherlands [email protected]
Abstract. The paper deals with the efficient parallelization of leastsquares spectral element methods for incompressible flows. The parallelization of this sort of problems requires two different strategies. On the one hand, the spectral element discretization benefits from an elementby-element parallelization strategy. On the other hand, an efficient strategy to solve the large sparse global systems benefits from a row-wise distribution of data. This requires two different kinds of data distributions and the conversion between them is rather complicated. In the present paper, the different strategies together with its conversion are discussed. Moreover, some results obtained on a distributed memory machine (Cray T3E) and on a virtual shared memory machine (SGI Origin 3800) are presented.
1
Introduction
Least-squares spectral element methods are based on two important and successful numerical methods: spectral/hp element methods and least-squares finite element methods. Least-squares methods lead to symmetric and positive definite algebraic systems which circumvent the Ladyzhenskaya-Babuˇska-Brezzi stability condition and consequently allow the use of equal order interpolation polynomials for all variables. The accuracy of a least-squares spectral element discretization of the Stokes problem (cast in velocity-vorticity-pressure form) has been reported in [5, 6] for different boundary conditions. The present paper deals with the efficient parallelization of this solver. Parallelization of the least-squares spectral element method (LSQSEM) requires two different kinds of distribution of data and the conversion is rather complicated. The spectral element structure enables one to calculate the local
Funding for this work was provided by the National Computing Facilities Foundation (NCF), under project numbers NRG-2000.07 and MP-068. Computing time was also provided by HPαC, Centre for High Performance Applied Computing at the Delft University of Technology.
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 39–52, 2003. c Springer-Verlag Berlin Heidelberg 2003
40
Margreet Nool and Michael M. J. Proot
matrices corresponding to each spectral element (also called cells), simultaneously. After the (parallel) calculation of the local systems, we have to switch from a local numbering to a global numbering to complete the gathering procedure. When this has been completed, one obtains a global Compressed Sparse Row (CSR) formatted matrix which can be easily distributed along an arbitrary number of processors. Each processor has to send data from one cell to a few other processors or possibly to itself, a very unbalanced task due to the chosen numbering. However, if this task is completed, each processor contains a part of the global assembled matrix, and the data per processor will be balanced again. Since the global system is symmetric and positive definite, the robust preconditioned conjugate gradient method (PCG) can be applied directly. In this paper, we describe results obtained with the simple, easy to parallelize, Jacobi or diagonal preconditioning. In the future, we plan to test the efficiency of other preconditioning methods: block-Jacobi, FEM-matrix and Additive Schwarz methods [3, 4, 7]. The results will be discussed in forthcoming work. The parallelization of the complete algorithm, including the important conversion of a spectral element-by-element distribution to a row-wise CSR-format distribution, and numerical results of large-scale problems arising in scientific computing are also discussed in the present paper.
2
The Least-Squares Spectral Element Formulation of the Stokes Problem
In order to obtain a bona fide least-squares formulation, the Stokes problem is first transformed into a system of first order partial differential equations by introducing the vorticity as an auxiliary variable. By using the identity ∇ × ∇ × u = −∆u + ∇(∇ · u) and by using the incompressibility constraint ∇ · u = 0, the governing equations subsequently read ∇p + ν∇ × ω = f in Ω, (1) ω − ∇ × u = 0 in Ω,
(2)
∇ · u = 0 in Ω,
(3) T
where, in the particular case of the two-dimensional problem, uT = [u1 , u2 ] represents the velocity vector, p the pressure, ω the vorticity, f T = [f1 , f2 ] the forcing term (if applicable) and ν the kinematic viscosity. For simplicity it is further assumed that the density equals ρ = 1. In two dimensions, system (1)(3), consists of four equations and four unknowns and is uniformly elliptic of order four. The velocity boundary condition (u given) is used to supplement the governing equations (1)-(3). The linear Stokes operator and its right-hand-side read: ∂ ∂ 0 0 ν ∂x u1 f1 ∂x1 2 ∂ ∂ 0 u 2 f2 0 −ν ∂x1 ∂x2 in Ω. L (U ) = F ⇐⇒ ∂ = (4) ∂ 1 0 ω 0 ∂x2 − ∂x1 ∂ ∂ p 0 0 0 ∂x1 ∂x2
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
3 3.1
41
General Implementation Aspects Discretization
The domain is discretized with a mesh of K non-overlapping conforming quadrilateral spectral elements of the same order N . Each quadrilateral spectral element is mapped on the parent spectral element Ω e by using an iso-parametric mapping to the bi-unitsquare [−1, 1] × [−1, 1] with local coordinates ξ1 and ξ2 . In the parent element all variables, located at the Gauss-Legendre-Lobatto collocation (GLL) points, can be approximated by the same Lagrangian interpolant, since the least-squares formulation is not constrained by the LadyzhenskayaBabuˇska-Brezzi stability condition. Most spectral element methods are based on the GLL numerical integration for reason of accuracy. For the two-dimensional Stokes problem, the discrete spectral element approximation yields uˆ1 N2 N1 uˆ2 , Uh = hi (ξ1 ) hj (ξ2 ) (5) ω ˆ j=0 i=0 pˆ i,j where hi (ξ1 ) with 0 ≤ i ≤ N1 and hj (ξ2 ) with 0 ≤ j ≤ N2 represent the Lagrange interpolants in the ξ1 and ξ2 -direction through the GLL points, respecT tively. The vector [ˆ u1 , u ˆ2 , ω ˆ , pˆ] is the vector of unknown coefficients, evaluated at the GLL collocation point. Hence, each spectral element gives rise to a local system of the form: Ai zi = fi ,
with i = 1, · · · , K,
where the matrices Ai and right-hand side vectors fi are given by T Ai = [L (ψ0,0 ) , · · · , L (ψN,N )] [L (ψ0,0 ) , · · · , L (ψN,N )] dΩ,
(6)
(7)
Ωe
and
T
[L (ψ0,0 ) , · · · , L (ψN,N )]
fi =
F dΩ,
(8)
Ωe
respectively and where ψi,j = hi (ξ1 ) hj (ξ2 ). 3.2
Local Numbering versus Global Numbering
Due to the continuity requirements between the C 0 -spectral elements, some of the variables zi corresponding to an internal edge will belong to more than one local system which necessitates the introduction of a global numbering. In Fig. 1 an example is given of a domain discretized with a mesh of four spectral elements. Each spectral element contains nine local nodes, numbered from 1 to 9 (small-size digits). In the same figure, also a global numbering (normal-size digits) is shown. First, the internal nodes or variables are numbered (1, · · · , 9), then the knowns (10, · · · , 25) given by the boundary conditions.
42
Margreet Nool and Michael M. J. Proot knowns
19 7
20 8
9
24
25
7
8
9
5
6
23
variables
18
4
5
6
5
4
III 17
1 7
2
3
1
2
3
8
9
7
8
9
5
6
4
5
6
2
3
2
3
4
4
1
10
1
11
22
IV 7
6
I 12
9
8
21
II 2
1
13
3
14
16
15
Fig. 1. Example of local and global numbering. The domain has been divided into four cells: I, II,III, IV. Each cell contains 9 nodes, denoted by a ◦ Since each local variable corresponds to a global variable, one can establish the local-global mapping operator gm for each spectral element. For the given example, we have gmI gmII gmIII gmIV
= [ 10, 11, 13, 12, 1, 2, 17, 4, 6 ], = [ 13, 14, 15, 2, 3, 16, 6, 7, 21 ], = [ 17, 4, 6, 18, 5, 8, 19, 20, 24 ], = [ 6, 7, 21, 8, 9, 22, 24, 25, 23 ].
(9)
The local-global mapping operator gmI can also be expressed by the sparse gathering matrix Gi which has nonzero entries according to Gi (i, gmI (i)) = 1, I = I, · · · , IV. The global assembly of the K local systems (6) can now readily be obtained with: K
K T KU = F ⇔ Gi Ai Gi U = GiT fi , (10) i=1
i=1
where the matrix K represents the symmetrical, globally gathered matrix of full bandwidth and the vectors U and F represent the global nodes (e.g., degrees of freedom and knowns) and the global right-hand side function, respectively. Since the known values are numbered last, one can subdivide the vector U into an unknown component U1 and a known component U2 . Consequently, the T matrix K can be factored into submatrices K1,1 , K1,2 , K1,2 and K2,2 . Also the right-hand side vector F can be factored into the vectors F1 and F2 . Hence, system (10) has the following matrix structure K1,1 K1,2 U1 F1 = , (11) T K1,2 K2,2 U2 F2
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
43
which readily allows “static condensation” of the knowns, leading to the following sparse symmetric and positive definite system K1,1 U1 = F1 − K1,2 U2 ,
(12)
which can be solved efficiently in parallel with the Jacobi preconditioned conjugate gradient method. The results of the iterative solver in [2] revealed that it is necessary to construct the global system (12) in parallel to obtain a good scalable solver. To this end, the matrices Ai , the vectors fi , the local-global mapping operator gmI and the gathering procedure, must be performed in parallel. 3.3
Investigation of the Grid
In order to construct the matrices Ai and vectors fi , one only needs the coordinates of the corners of the cells and its GLL orders in ξ1 - and ξ2 direction, N1 and N2 , respectively. For the calculation of the local-global mapping operator gmI , detailed grid information regarding the neighbouring cells is required to obtain the global numbering in parallel. Hence, the coordinates of the corners, the neighbouring cells and the GLL order together with a unique global number (see Section 3.4) are the minimal information needed to construct the global system (12) in parallel. These data will be collected on one processor when the grid is investigated and broadcasted from the root processor to min ((K, P)) processors, where P is the number of available processing elements. This information can be collected the following way. The order of the spectral elements p = N1 = N2 is an input variable. When the grid file is read, the nodes, the nodal coordinates and the vertices of the spectral element mesh become available. The neighbors of the spectral elements can be obtained by investigating the mesh. This can be done the following way. The vertices of a spectral element are given in fixed order, see Fig. 2. In case two elements share the same node, their position with respect to each other is known. E.g., if node 0 of element S5 equals node 3 of element S2 then S2 must be the South-neighbor of S5 . If node 2 of S5 equals node 3 of S6 then S6 will be the East-neighbor of S5 , and so on. If a spectral element has less than four neighbors it must be a boundary element. For the parallel global numbering of the GLL collocation points, it is necessary to assign the points to one particular cell. For the inner points this is trivial, but the situation is more complicated for points located at the interfaces. Therefore, we must define to which spectral element the vertices and edges belong. There are several possibilities, with the same complexity, and we choose one of them: – a spectral element consists of all its internal GLL points, the lower-left vertex, the collocation points at the South and West edge, without the corner nodes. – if a spectral element has an external boundary, the set is extended with the boundary points and possibly by vertices that do not belong to other spectral elements.
44
Margreet Nool and Michael M. J. Proot 3 0 3
0
1 0 2 3
S
4
0 3
2 3
S7
1 0 2 3
S1
1 0
S8
2 3 1 0 2 3
S
5
S2
S9 S
6
1 0 2 3 1 0
S3
2 1 2 1 2 1
Fig. 2. Ordering of vertices: the vertices are numbered in fixed order, such that it is easy to recognize which spectral elements are neighbors 3 3 3 3
3 3
3 3
4 4 4 4
4 4
4 4 4 4
3 3
3
3
4 4
4
4 4
3 3 3 3
3 3
3 3
4 4 4 4
4 4
4 4 4 4
1 1
1
1
2 2
2
2 2
1 1
1
1
2 2
2
2 2
1 1 1 1
1 1
1 1
2 2 2 2
2 2
2 2 2 2
Fig. 3. An example of a grid discretized into four spectral elements. The nodes numbered 1 belong to spectral element 1, those numbered 2 belong to spectral element 2, and so on. The solid line corresponds to the physical boundary Fig. 3 displays a mesh of four spectral elements and the numbers 1 to 4 elucidate to which spectral element the nodes belong. Communication is required to obtain the missing values on the internal North and East interfaces: first data on the edge are sent from North to South. Subsequently, data on the edge are sent from East to West. After these steps, also the right-upper vertex has got its correct component values. 3.4
Unique Global Numbering
To obtain a unique global numbering, first the interface points are numbered which belong to a single spectral element: this local numbering starts with 1. k Then we know exactly how much degrees of freedom Ndof are present per element and transfer this information along the processors, such that we can compute Mi Mi =
i−1 k=1
k Ndof ,
(13)
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
45
being the sum of the degrees of freedom on preceding spectral elements k, k < i. The value Mi will be added to the local number. The knowns are numbered K analogously. The numbering starts with MK + Ndof + 1. Hence, one obtains the global node numbering discussed in Section 3.2. 3.5
The Construction of Matrix K
The parallel global assembly of the K local systems can now be performed resulting in the sparse symmetric positive definite global matrix K. Since the solution of eq. (12) is by far the most time consuming part, it must be optimally performed. Therefore, K will be distributed along the processors into balanced parts of rows. The sparsity of K asks for a CSR-formatted storage approach. The symmetry is not exploited: all nonzero elements of K are stored. However, the CG method can only be applied successfully if K is symmetric and positive definite. The conversion of the local systems Ai into the global system K is complicated and requires communication. Beforehand, it is difficult to say which processors will communicate with each other and how long the messages will be. The global number determines to which processor the data will be sent. Therefore, a type gambit has been defined which couples the global number with the processor number and the row number on that particular processor: TYPE :: gambit INTEGER :: globalnr INTEGER :: procnr INTEGER :: rownr END TYPE Let Ndof denote the degrees of freedom. Because only the first Ndof rows of matrix K are desired, the value procnr is not important in case the global number is larger than Ndof : to avoid confusion a negative value is added to procnr. For each nonzero element of Ai it is then easy to determine whether it contributes to K1,1 , to K1,2 , or whether it can be neglected (row number > Ndof ). All nonzero elements of Ai are considered and on account of its global number it is known to which processor it has to be sent. Before such an element and corresponding information about column number in K is sent, all data are gathered and put into one message intended for a particular processor. Three different cases can occur: 1. The message must be sent to the processor on which it is already present. In that case, a send instruction is superfluous and the data can be processed immediately, or after the other messages have been sent. 2. The message is empty: no communication will take place, except that a message will be sent to the receiving processor that no contribution of the send processor can be expected.
46
Margreet Nool and Michael M. J. Proot
3. The message is not empty and the send and receive processors are unequal. In that case communication is necessary. The order in which the messages will be received is not known, and does not matter either. If nonzero elements are already present (see case 1 above) the first very sparse matrix Lj can be built up. This matrix Lj contains the nonzero values of an Ai descendent of a spectral element stored in CSR-format. Contributions of other processors will be added to Lj - i.e. each step the sum of two sparse matrices, stored in CSR-format, is calculated with a SPARSKIT [9] subroutine. In case no data are present, e.g., the rank of the processor is larger than the number of cells, the first data received will be used to build up the basis matrix Lj . At the end, Lj will be the j-th strip of the matrix K. In the worst case, to create Lj , data from K spectral elements are needed. In practice, this number will be less. We do not investigate to find for all (K, P) combinations a favorite global numbering. We observe that two steps are required to construct the matrix K of (11) completely: one to construct K1,1 and one to construct K1,2 of eq. (12). The distribution of K1,1 and K1,2 along the processors is similar.
4
Parallel Platform and Implementation Tools
The calculations have been performed on – Cray T3E system Vermeer (named after the Dutch painter) at HPαC with 128 user processing elements (PE’s) interconnected by the fast 3D torus network with a peak performance of 76.8 Gigaflop/s. The T3E uses the DEC Alpha 21164 for its computational tasks. Each node in the system contains one PE which in turn contains a CPU, memory, and a communication engine that takes care of communication between PE’s. The bandwidth between nodes is quite high: 325 MB/s, bi-directional. Each PE is configured with
A1
A2
A3
A4
P0
P1
P2
P3
P0
P1
P2
P3
P4
P5
K1
K2
K3
K4
K5
K6
Send
Receive
Fig. 4. Example of mapping of local matrices Ai into the global matrix K, partitioned into matrix parts Kj of consecutive rows
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
47
128 Mbytes of local memory, providing more than 16 Gbytes of globally addressable distributed memory. – The SGI Origin 3800 Teras with 1024 500 MHz RI 14000 processors, subdivided into six partitions, two (interactive) 32-CPU partitions and four batch partitions of 64, 128, 256 and 512 CPU’s, respectively. The theoretical peak performance is 1 Teraflop/s. Parallelization is done either automatically or explicitly by the user. A message passing model is allowed on the Origin using optimized SGI versions of e.g. MPI. For more information about the Cray T3E used, we refer to [10]. To get good portable programs which may run on distributed-memory multiprocessors, networks of workstations as well as shared-memory machines we use MPI, Message Passing Interface. Standard or blocking communication mode is used: a send call does not return until the message data have been safely stored away so that the sender is free to access and overwrite the send buffer. Besides the standard communication mode, it appears to be necessary to use the buffered-mode send operation MPI BSEND (see e.g., [8]). This operation can be started whether or not a matching receive has been posted. Its completion does not depend on the occurrence of a matching receive. In this way a deadlock situation, in which all processes are blocked is prevented. The execution of the send-receive process demonstrated in Fig. 4 profits of the buffered-mode send operation. All routines have been implemented in FORTRAN 90, making frequently use of dynamic allocation of memory and derived types. As described in Sect. 3.5 the conversion of the cell-wise distribution into the row-wise CSR-format storage is difficult to implement and asks for a high level of flexibility. On fore hand it is not clear which processing elements will communicate and how much data will actually transferred. An upper bound for the number of processors which may need data of cell i can be K, the number of cells involved, whereas an upperbound of data from cell i to PE j can be the whole contents of that particular cell i. Apparently, multiplication of these two upperbounds leads to a storage demand per processor which grows with the number of processors used in the execution. The solution can be found in the usage of data structures with pointers to variable array sizes.
5 5.1
Numerical Results The h- and p-Refinement Approach and Its Accuracy
In (least-squares) spectral element applications, two different kinds of refinement strategies are commonly used: h-refinement and p-refinement. The purpose of the numerical simulations is to check the accuracy performance for both refinement strategies. To this end, the least-squares spectral element formulation of the velocity-vorticity-pressure formulation of the Stokes problem is demonstrated by means of the smooth model problem of Gerritsma-Phillips [1] with v = 1. This model problem involves an exact periodic solution of the Stokes problem
48
Margreet Nool and Michael M. J. Proot
defined on the unit-square [0, 1] × [0, 1]. The velocity boundary condition is used for all the numerical simulations. The pressure constant is set at the point (0, 0). Six different grids are used to check the accuracy of the h-refinement. As can be observed in Table 1, the polynomial order of all the spectral elements equals 4, which means that each direction has four Gauss-Legendre-Lobatto(GLL) collocation points, and the number of spectral elements is varied from 4 to 144. In the middle column of Tables 1 and 2 the order of the large sparse global system is given together with the number of iterations required to solve this system using CG. The right column in the tables lists the L2 norm of the different components, like the velocity (L2 norm of x− and y−components agree), the vorticity and pressure. Only four different grids have been used to check the accuracy in case of the p-refinement (see Table 2). Each grid contains four spectral elements. The order of the approximating polynomial varies from 4 to 10 and is the same in all the variables. A growth of the polynomial order in the p-refinement case will increase the number of nodes per cell and likewise the amount of computational effort per cell. 5.2
The Parallel Performance
The least-squares spectral element solver (LSQSEM) code allows for the following situations: K ≥ P : At least several processors contain more than one spectral element. It is not required that P is a true divisor of K, or put it another way, the number of spectral elements per PE may differ.
Table 1. The different grids used for the investigation of the h−refinements Spectral GLLsize of # elements order global system iterations Velocity 2×2 4 259 132 9.2 10−4 4×4 4 1027 232 5.0 10−5 6×6 4 2307 326 5.2 10−6 8×8 4 4099 431 1.1 10−6 10 × 10 4 6403 569 3.2 10−7 12 × 12 4 9219 707 1.2 10−7
L2 norm Vorticity 4.8 10−2 1.6 10−3 2.8 10−4 8.7 10−5 3.5 10−5 1.7 10−5
Pressure 1.8 10−2 7.1 10−4 6.9 10−5 1.3 10−5 3.6 10−6 1.3 10−6
Table 2. The different grids used for the investigation of the p-refinements Spectral GLLsize of # elements order global system iterations Velocity 2×2 4 259 132 9.2 10−4 2×2 6 579 224 8.7 10−6 2×2 8 1027 305 6.5 10−8 2×2 10 1603 388 4.4 10−10
L2 norm Vorticity 4.8 10−2 7.5 10−4 7.1 10−6 4.5 10−8
Pressure 1.8 10−2 1.9 10−3 1.6 10−6 7.6 10−9
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
49
K < P : A less plausible possibility. As we may conclude from Tables 1 and 2 that in order to increase the accuracy of the solution it is to preferred to enlarge the GLL order rather than to add more spectral elements. The latter gives rise to much larger global systems and accordingly more execution time. Indeed, during the construction of the local systems P − K processors are idle, but for the same accuracy, the wall-clock time can be less. We remark, that MPI gives the opportunity to define an intracommunicator, which is used for communicating within a group, for instance within the group of PE’s which correspond to a spectral element. In the code we may consider several phases in the parallelization. The phases described correspond to the parts shown in Tables 3 and 4: – The decomposition of the domain, or the investigation of the grid (cf. Sect. 3.3). In this phase, information is read from file: the number of spectral elements, the coordinates of the corners of the spectral elements, and of each spectral element its vertices are listed on file. Further the position of the spectral elements with respect to its neighbors is derived. This phase will be performed on a single processor. – the second phase called GAMBIT, performs the computation on the internal GLL nodes and on the edge and vertices. Also the local-global mapping (cf. eq. (10)) is calculated. Communication is required for internal edges and vertices. – The third phase computes the local systems and the right hand side values. This part is the most time-consuming part of the calculations on spectral elements and as can be conclude from Table 3 and 4, it is well scalable. – The fourth and fifth phases consider the conversion of the local systems into the global systems. In SUM the contribution of the local systems to the construction of matrix K is gathered. CSR performs the actual construction of K1,1 and K1,2 of eq. (11) by means of computing the sum of sparse matrices. Also the communication as described by Fig. 4 is included in CSR. – In the last phase of the program PCG is used to solve the global system. In [2] it is described how the matrix-vector product in PCG are computed in parallel and gives pictures of speed-ups of this part of the code achieved on Teras and Vermeer. The last column of the tables gives the wall-clock time for a complete run. We emphasize that the conversion part may not be neglected, especially the man power that was needed to give the code the flexibility for allowing an arbitrary number of spectral elements with respect to the number of processors involved, but for 32 processors it takes about 1.2 % of the total wall clock time. Notice that super linear speed up is achieved in the CSR-phase. The wall-clock times of a cylinder grid as shown in Fig. 5 are listed in Table 4. It concerns a grid of 86 spectral elements including 66 boundary elements. The no-slip condition holds at the cylinder boundary, the velocity boundary condition on the outside boundary and the forcing term is zero. On Vermeer, the GLL order
50
Margreet Nool and Michael M. J. Proot
Table 3. Wall-clock timings in seconds for the different parts in the parallel LSQSEM code obtained for the grid of 12 × 12 spectral elements and GLL order=6 P 4 8 16 32
Spectral elements read GAMBIT Stokes 0.19 0.38 118.95 0.20 0.30 59.44 0.18 0.23 29.70 0.19 0.21 16.54
Conversion SUM CSR 3.01 21.94 1.56 7.41 0.85 2.65 0.59 1.11
PCG time 584.57 308.55 176.34 121.67
Execution time 729.19 377.65 210.15 140.77
Y
0.5 0
-0.5 -1
0
X
1
2
3
Fig. 5. Cylinder grid existing of 86 spectral elements with p = 4 Table 4. Wall-clock timings in seconds for the different parts in the parallel LSQSEM code obtained for the cylinder grid P 1 2 4 8 16 2 4 8 16 32
Teras, p = 8 Spectral elements Conversion read GAMBIT Stokes SUM CSR 0.04 0.15 61.30 3.45 44.85 0.06 0.06 30.60 2.05 18.66 0.04 0.04 15.65 1.05 10.53 0.06 0.03 7.93 0.59 2.83 0.11 0.08 4.40 0.33 1.24 Vermeer, p = 6 0.24 0.15 142.43 4.59 48.90 0.24 0.14 72.58 2.42 26.97 0.19 0.08 36.24 1.26 7.63 0.19 0.14 19.80 0.74 2.77 0.20 0.05 9.88 0.54 1.11
PCG time 1522.1 966.6 475.1 232.0 142.0
Execution time 1633.3 1018.2 502.5 243.6 150.3
853.2 444.0 228.0 120.4 68.7
1049.0 546.7 273.6 144.1 80.7
is six in both directions, on TERAS the order is eight. The number of iterations is 1261 and 1904, respectively. Finally, Fig. 6 shows the overall performance of the code on on a square domain partioned into decomposed into 8 × 8, 10 × 10 and 12 × 12 spectral elements. On Vermeer, it is not possible to run the largest problem on less than
A Parallel, State-of-the-Art, Least-Squares Spectral Element Solver
51
Vermeer,GLL=6 700
Wall clock time
600
500
400
300
200
12x12 100
10x10 8x8 0
4
8
12
16
24
# processors
32
Fig. 6. Wall-clock timings achieved for a grid decomposed into 8 × 8, 10 × 10 and 12 × 12 spectral elements four processors. We remark, that not in all cases the number of cells is a multiple of P, e.g., the 10×10 grid has only P = 4 as a true divisor. The speedup obtained when eight times more processors are used, (viz. from P = 4 to P = 32) is more than 5.2 and for the 8 × 8 is nearly 7.
6
Conclusions and Outlook
Least-squares spectral element methods result in symmetric and positive definite systems of linear equations which can be efficiently solved in parallel by PCG. The parallelization of this kind of problems requires two different strategies. Indeed, the spectral element discretization benefits from an element-by-element parallelization strategy whereas an efficient strategy to solve the large sparse global systems benefits from a row-wise distribution of data. The numerical results, obtained with a simple model problem, confirm the good parallel properties of the element-by-element parallelization strategy. The combination of this strategy with the parallel PCG solver resulted in a good parallelizable code to solve incompressible flow problems. In the near future, the code will be extended to allow the usage of unstructured grids. More effective preconditioning methods for least-squares spectral element methods will be developed, too.
52
Margreet Nool and Michael M. J. Proot
References [1] M. I. Gerritsma and T. N. Phillips. Discontinuous spectral element approximations for the velocity-pressure-stress formulation of the Stokes problem. Int. J. Numer. Meth. Eng., 43:1401–1419, 1998. 47 [2] M. Nool and M. M. J. Proot. Parallel implementation of a least-squares spectral element solver for incompressible flow problems. Submitted to J. Supercomput., 2002. 43, 49 [3] L. F. Pavarino and O. B. Widlund. Iterative substructuring methods for spectral element discretizations of elliptic systems. I: compressible linear elasticity. SIAM J. NUMER. ANAL., 37(2):353–374, 1999. 40 [4] L. F. Pavarino and O. B. Widlund. Iterative substructuring methods for spectral element discretizations of elliptic systems. II: Mixed methods for linear elasticity and Stokes flow. SIAM J. NUMER. ANAL., 37(2):375–402, 1999. 40 [5] M. M. J. Proot and M. I. Gerritsma. A least-squares spectral element formulation for the Stokes problem. J. Sci. Comp., 17(1-3):311–322, 2002. 39 [6] M. M. J. Proot and M. I. Gerritsma. Least-squares spectral elements applied to the Stokes problem. submitted. 39 [7] A. Quateroni and A. Valli. Domain Decomposition Methods for Partial Differential equations. Oxford University Press, 1999. 40 [8] Marc Snir, Steve W. Otto, Steven Huss-Lederman, David W. Walker, and Jack Dongarra. MPI: The Complete Reference. MIT Press, Cambridge, Massachusetts, 1996. 47 [9] Yousef Saad. SPARSKIT: a basic tool-kit for sparse matrix computations (Version 2) http://www.cs.umn.edu/research/arpa/SPARSKIT/sparskit.html 46 [10] Aad J. van der Steen. Overview of recent supercomputers, Issue 2001 National Computing Facilities Foundation, Den Haag, 2001. 47
Edge-Based Interface Elements for Solution of ThreeDimensional Geomechanical Problems Rubens M. Sydenstricker, Marcos A. D. Martins, Alvaro L. G. A. Coutinho and José L. D. Alves Center for Parallel Computing and Department of Civil Engineering, COPPE/Federal University of Rio de Janeiro PO Box 68506, Rio de Janeiro, RJ 21945-970, Brazil rubens, marcos, alvaro}@nacad.ufrj.br [email protected] http://www.nacad.ufrj.br
Abstract. An edge-based three-dimensional interface element for simulation of joints, faults and other discontinuities present in several geomechanical applications is proposed. Edge-based data structures are used to improve computational efficiency of Inexact Newton methods for solving finite element nonlinear problems on unstructured meshes. Numerical experiments in the solution of three-dimensional problems in cache based and vector processors have shown that memory and computer time are reduced.
1 Introduction Solving three-dimensional large-scale geomechanical problems undergoing plastic deformations is of fundamental importance in several science and engineering applications. Particularly, interface finite elements have been widely used to model joints in rocks, faults in geologic media, as well as soil/structure interaction. Essentially, these elements must allow slip and separation between two bodies in contact, or separated by a thin material layer. Stresses are constrained so that traction in the interface element does not occur, and shear stresses are limited by a yield criterion. Although the required mechanical behavior of interface elements is simple, many analysts have reported numerical problems, such as oscillatory response. Several interface elements have been proposed in the literature, as those proposed by Herrmann [4] and Goodman et al. [3] for two-dimensional analysis. Although they can be extended to the three-dimensional case, it can be shown that the resulting elements have the same features of the two-dimensional corresponding elements, including the kinematic inconsistency pointed by Kaliakin and Li [5] for the element proposed by Goodman et al. [3]. In this work, we develop a simple triangular interface element for three dimensional analysis, which may be viewed as an extension of the element proposed Goodman et al. [3]. A similar approach was used by Schellekens and De Borst [10] to derive quadrilateral interface elements. This element has been implemented into an edge-based nonlinear quasi-static solid mechanics code to solve three-dimensional problems on unstructured finite element meshes [1, 9]. Edge-based data structures are used to improve the computational efficiency of the sparse matrix-vector products needed in the inner iterative driver of the Inexact Newton
method. The interface element can be regarded as a prism with triangular base. Thus, we developed an edge-based decomposition of its stiffness matrix to preserve the good computational properties of the existing edge-based code. To minimize indirect addressing effects, we also introduced new superedge arrangements for interface elements. The remainder of this work is organized as follows. In the next section we briefly review the governing nonlinear finite element equations and the Inexact Newton method. Section 3 describes the interface element and its edge-based implementation. Section 4 shows numerical examples on the Athlon AMD 1.4 GHz, Intel Pentium IV 1.8 GHz, Cray SV1 and Cray T90 processors . The paper ends with a summary of the main conclusions.
2 Incremental Equilibrium Equations and the Inexact Newton Method The governing equations for the quasi-static deformation of a body occupying a volume Ω , in indicial notation, is:
σ ij x j
ρ bi 0
in
(1)
where σij is the Cauchy stress tensor, xj is the position vector, ρ is the weight per unit volume and b i is a specified body force vector. Equation (1) is subjected to the kinematic and traction boundary conditions,
ui ( x, t) ui ( x, t) in u ; σ ij n j hi ( x, t ) in h
(2)
where Γu represents the portion of the boundary where displacements are prescribed ( ui ) and Γh represents the portion of the boundary on which tractions are specified (h i). The boundary of the body is given by G Gu Gh , and t represents a pseudo-time (or increment). Discretizing the above equations by a displacement-based finite element method we arrive to the discrete equilibrium equation,
{ Fint} {Fext } {0}
(3)
where {Fint} is the internal force vector and {Fext} is the external force vector, accounting for applied forces and boundary conditions. Assuming that external forces are applied incrementally and restricting ourselves to material nonlinearities only, we arrive, after a standard linearization procedure, to the nonlinear finite element system of equations to be solved at each load increment,
[ KT ] {u} { R}
(4)
where [KT] is the tangent stiffness matrix, function of the current displacements, {Δu} is the displacement increments vector and {R} is the unbalanced residual vector, that is, the difference between internal and external forces. Remark. We consider here that solid elements are perfect-plastic materials described by Mohr-Coulomb yield criterion. Stress updating is performed by an explicit, Euler-forward subincremental technique [2].
When solving iteratively the finite element system of linear equations, it is straightforward to employ inexact versions of the standard Newton-like methods [6]. In this case, tolerances for the inner iterative driver may be adaptively selected to minimize computational effort towards the solution. The iterative driver is the preconditioned conjugate gradient method, implemented employing edge-based strategies [1], that is, computing matrix-vector products as,
[ K e]
m
[K se ]
(5)
s 1
where [ K se ] is the contribution of edge s to [ K e ] and m is the number of element edges.
3 Edge-Based Interface Elements The interface element, which will be named GTB_3D, is composed by two triangular faces, and contains six nodes with 3 degrees of freedom per node as shown in Fig. 1. wi
6 ui
z 4
3
x 1
i 5
vi
2
y
Fig. 1. Interface element
The element formulation is an extension of the original element proposed by Goodman et al. [3] for two-dimensional analysis. Displacements at upper and lower faces are independently interpolated as:
{dupper } [ N ]{ dˆup}
; {dlower } [ N ]{ dˆlw}
(6)
where {dupper } and {dlower } contain the displacement components as:
wupper {d upper } uupper v upper
;
wlower {d lower } ulower v lower
(7)
{dˆlw } and {dˆup} contain the nodal displacements organized as: {dˆ lw}T [{ dˆ1}T {dˆ2 }T {dˆ 3}T ] ; {dˆ up}T [{ dˆ 4 }T {dˆ 5}T {dˆ6 }T ] T
where {dˆ i } [ uˆ i vˆi wˆ i ] and [N] is the interpolation function matrix given by:
(8)
0 L2 0 0 L 3 0 0 L1 0 [ N ] L1 0 0 L 2 0 0 L3 0 0 0 L1 0 0 L 2 0 0 L3 0
(9)
where L1, L2 and L3 are area coordinates. The material behavior may be viewed as being elastic orthotropic with Ex =Ey =0, Gxy =0 and ν xy ν xz ν yz 0 . As a consequence, normal stresses σ x , σ y and the in-plane shear stress τ xy are null, and stress-strain relations may be stated as:
{σ } [ D] {ε }
(10)
{σ }T [σ z τ xz τ yz ] ; {ε}T [ε z γ xz γ yz ]
(11)
where:
and [D] is the constitutive matrix:
d11 [ D] d22 d33
(12)
Deformations are computed from relative displacements as:
wupper wlower w 1 1 {ε} u uupper ulower h h v vupper vlower
(13)
where h is the thickness of the element. From equations (6) and (13) we write:
{ε} [ B ]{dˆ}
(14)
where:
[ B]
1 * 1 [ B ] [ N ] [ N ] h h
(15)
and:
{dˆ } {dˆ } ˆ lw {dup }
(16)
Strain energy is evaluated as:
U
1 2
h 1 {σ } dV {dˆ}T [ B]T [ D][ B ] dA{dˆ} o A 2 1 {dˆ}T [ B * ]T [ DI ][ B* ] dA{ dˆ} A 2
V {ε }
T
where:
(17)
0 d11 0 0 DI11 0 [ DI ] 0 DI 22 0 1 0 d22 0 . h 0 0 DI 33 0 0 d33
(18)
The stiffness matrix is obtained from the last integral of equation (17), and is explicit given by:
A [ KK ] [ KK ] 12 [ KK ] [ KK ]
(19)
2[ DD] [ DD ] [ DD ] [ KK ] [ DD ] 2[ DD] [ DD ] [ DD ] [ DD ] 2[ DD]
(20)
0 DI 22 0 [ DD ] 0 DI 33 0 0 DI 11 0
(21)
[ KGTB3D ] where:
and A is the area of the triangular surface. In order to implement the interface element into our edge-based solver, we disassemble the element matrix into its edge contributions as shown in Fig. 2.
Fig. 2. Schematic representation of interface element (left) and its edge disassembling (right)
Observe in Fig. 2 that the interface element generates 6 geometrical edges and 9 computational edges. The resulting interface element geometrical edges are shared with the solid elements. Table 1 presents the number of floating point operations (flops) and indirect addressing (ia) for the matrix-vector multiplication needed in the preconditioned conjugate gradient method for element-by-element and edge-by-edge implementations. We show in column (a) the estimates for a tetrahedron and in column (c) for the interface element, while in columns (b) and (d) are shown the estimates for the edge implementation for both element types.
Table 1. Computational parameters for matrix-vector multiplication
Parameter Flops Ia
Tetrahedron (a) 252 36
Edge tet 1.5 (b) 54 27
Interface (c) 108 54
Edge int 6 (d) 36 108
We may observe in Table 1 the gain in flops when the edge based scheme is used for both elements. In columns (b) and (d) estimates are affected respectively by factors 1.5 and 6. These factors represent the ratio of edges to number of elements for each element type in a typical unstructured mesh. However, the interface element has more ia operations. This burden is alleviated by the small number of interface elements with respect to solid elements and the fact that algorithmic complexity is maintained, that is, instead of having two element types, we have just one, the edge. Table 2 shows the memory demand for storing stiffness coefficients element-by-element and edge-by-edge for both element types. Table 2. Memory demand for storing stiffness coefficients
Data Structure Element Edge
Tetrahedron 78 9
Interface 171 6
Besides the improvements over the element based structure, there are more advantages in ia and flops parameters by grouping edges as depicted in Fig. 3, in superedges [1, 8]. The superedges assembled from interface elements are indicated as superedge4 and superedge9, resulting from crossing edges and vertical edges, as indicated by bold lines in Fig. 3. The edges from the triangular faces are accounted by the tetrahedron elements.
Tetrahedron
Superedge3
Superedge6
Interface
Superedge4
Superedge9
Fig. 3. Schematic representation of superedges for tetrahedron and interface elements
The idea behind superedges is, when loading data from memory to cache or vector registers, keep it there at the most, in order to operate them completely. This concept is
completed considering that one edge carry out all contributions from its neighborhood elements, avoiding unnecessary floating points operations and several memory accesses for each element, when the edge is loaded to the processor and operated at once, so maximizing the flops/ia balance. Indeed, an ordered access to memory increases the processor performance for any kind of modern processors (cisc, risc) or shared or distributed memory parallel computers. One alternative to optimize the balance of flops/ia is to reordering the nodal points in such way that memory contention is minimized [7]. Improvements in flops and ia parameters are shown in Table 3 for several superedges groupings. However, observe that these gains are not for the whole mesh, since in several unstructured meshes just part of the edges can be grouped into these superedge arrangements [1, 9]. Table 3. Computational parameters for matrix-vector multiplication with superedges
Parameter Flops Ia Flops/ia
Simple Edge 36 18 2.0
Superedge3 130 27 4.8
Superedge4 190 36 5.3
Superedge6 268 36 7.4
Superedge9 436 54 8.1
As one can see in Table 3, the balance of flops/ia increases with the edge grouping indicating great advantages over the simple edge. However, cache size and register availability may be observed in order to best fit flops and ia operations in the architecture, taking into account code simplicity and maintenance.
4 Numerical Examples 4.1 Footing-Soil Interaction In this example we simulate a footing resting on a soil mass and submitted to a vertical and a skew horizontal force, as shown in Fig. 4. The angle between the force and x-axis is 30o as indicated. The vertical force was maintained constant and equal to one, and the skew horizontal force was monotonically increased from zero to 0.4 in 5 load steps. For the soil mass, all displacements at the base are prescribed, as well as all normal to the lateral bounding displacements. Soil mass and footing are separated by interface elements. Both soil and footing are linear elastic, and a Mohr-Coulomb criterion with a friction coefficient equal to 0.5 and null cohesion for the interfaces was adopted. Both soil mass and footing were represented by linear tetrahedra, and the interface between soil and block was represented by GTB_3D elements. Material properties for soil and footing are: Eblock=105
;
Esoil=103
;
νblock = soil = 0.0
and for the interface elements we considered: DI11= 1010
;
DI22= DI33=104
Fv
x y
Fh
0.5
x =30o 1.0
z
1.0
10.0
5.0
10.0
Fig. 4. Soil-footing interaction
In Table 4 we present the mesh properties and, in Fig. 5, we present details of the mesh. The problem was solved by using a tangent inexact Newton method [1], with PCG tolerances varying between 10-2 and 10-6. Table 4. Mesh properties for soil-footing interaction problem
Number of tetrahedra
Number of interfaces
Number of nodes
Number of equations
143,110
5,000
27,890
83,238
Fig. 5. Soil-footing interaction - details the mesh
In the first load step, almost all elements remain closed without slipping. For the remaining load steps, the interface path is shown in Fig. 6, in which interface elements that experienced opening and slipping are respectively indicated in white and gray, whereas the elements that remained closed without slipping are indicated in black. We note that, at load step 5, there is an opening region bellow the point where the skew horizontal force was applied. We see that, at this stage, almost the whole interface path is slipping.
Load step 2
Load step3
Load step 4
Load step 5
Fig. 6. Interface bonding (black), slipping (gray) and separation (white).
Table 5 shows the computational performance on the Athlon AMD 1.4 GHz and Intel Pentium IV 1.8 GHz for the solution of this problem. We may observe that the super-edge solutions were faster than the simple edge solutions in both machines.
Table 5. Computational performance for Athlon 1.4 GHz and Intel Pentium IV 1.8 GHz processors.
Processor AMD Athlon 1,4 Intel Pentium IV
Simple Edge (h) 9.1 7.9
Superedge (h) 7.0 6.1
Ratio Super/simp. 0.77 0.77
4.2 Block of a Sedimentary Basin In this example we simulate the interaction of four blocks of a sedimentary basin separated by three geological faults, as shown in Fig. 7. Geometry and material properties are similar to a portion of a sedimentary basin in north-east Brazil. The blocks were submitted to a longitudinal compression applied at the left lateral surface. All displacements on the top are free and, at the basis and lateral surfaces, displacements normal to the mesh are prescribed. The mesh comprises 26,860 nodes, 94,058 linear tetrahedral elements, 12,960 interface elements and 76,380 equations. The resulting number of edges is 187,266 and 40.08% were grouped in superedge6, 16.99% in superedge3, 8.94% in superedge9 and 0.88% in superedge4; the percentage of remaining simple edges is 33.12%. To improve data locality we reordered nodes to reduce wave front.
Fig 7. Mesh for a portion of a sedimentary basin
Table 6 presents memory demands for storing stiffness matrices for tetrahedral and interface elements using respectively element-by-element and edge-based data structures. Note that we achieved a remarkable saving over the standard element by element data structure, particularly for interface elements. We applied prescribed displacements in six increments, stopping before the full slipping of the first inner fault. On average two nonlinear iterations were performed per displacement increment, using a tangent inexact Newton method, with PCG tolerances varying between 10-2 and 10-6. An average number of 4,000 PCG iterations per increment were needed to solve the linearized systems. Table 6. Memory demand for storing stiffness matrices for element and edge data structures.
Data type Tetrahedra Interface
Element (Mw) 7.0 2.1
Edge (Mw) 1.6 0.27
Ratio El/Ed 4.4 7.8
In Fig. 8 we present the vertical displacements at the 6th displacement increment. The maximum displacement occurs at the top of the fault. Fig. 9 shows the evolution of slipping interface elements at three inner faults for 2nd and 6th steps. In this figure, the slipping elements are indicated as black and represent about 40% for 2nd step and almost 50% for 6th step of the total number of interface elements. After the 6th step we observed a full slipping of the first fault, producing a structural instability.
Fig 8. Vertical displacements for 6th step.
(a)
(b)
Fig 9. Slipping interface elements at inner faults for (a) 2nd and (b) 6th steps.
Table 7 shows the total CPU time obtained on the Athlon AMD 1.4 GHz, Intel Pentium IV 1.8, Cray SV1 and Cray T90 vector processor using only edges and all superedges arrangements for tetrahedral and interface elements. Note that performance improvements for superedges were greater in the scalars processors. We may associate this behavior to their cache based architectures.
Table 7. CPU times for symmetric edge-based matrix-vector product for different processors.
Processor Athlon AMD 1.4 Intel Pentium IV 1.8 Cray SV1 Cray T90
Simple Edge (s) 10,702.0 7,722.0 3,230.0 1,076.0
Superedge (s) 8,257.0 5.868.0 2,954.0 1.049.0
Ratio Super/Simp 0.77 0.76 0.89 0.97
5. Conclusions In this work we have introduced a new implementation of a three-dimensional interface element integrated to a highly optimized edge-based solver for geomechanical problems. Results indicate that this element represents correctly the mechanical behavior of geologic faults and geomechanical interfaces. To maximize individual processor performance we introduced new superedge groupings. These were particularly effective in cache-based processors. Memory costs for element data structure is almost eight times the observed for edge-based structure. However, since the number of interface elements was much smaller than the number of solid elements, the overall memory reduction is more modest.
Acknowledgments This work is supported by CNPq grant 522692/95-8 and by MCT/FINEP/PADCT project on Modeling of Sedimentary Basins. We are indebted to the Brazilian Petroleum Agency
Program on Civil Engineering Human Resources for Oil and Gas (PRH02-UFRJ-COPPE/PEC). Computer time was provided by the Center of Parallel Computation of COPPE/UFRJ, the National Center for Supercomputing, CESUP/UFRGS and Cray Inc.
References 1. Coutinho, A.L.G.A., Martins, M.A.D., Alves, J.L.D., Landau, L., Moraes, A.: Edge-Based Finite Element Techniques for Nonlinear Solid Mechanics Problems. Int. J. Num. Meth. Engng., 50 (2001) 2053-2058. 2. Crisfield, M.A., Non-Linear Finite Element Analysis of Solids and Structures. Advanced Topics, v. 2, John Wiley & Sons, London, UK, 1997. 3. Goodman, R.E., Taylor, R.L., Brekke, T.L.: A Model for the Mechanics of Jointed Rock. J. Soil Mech. Fdns. Div., ASCE, 99 (1968), 637-659. 4. Herrmann, L.R.: Finite Element Analysis of Contact Problems . J. Eng. Mech., ASCE, 104 (1978) 1043–1059. 5. Kaliakin, V.N., Li, J.: Insight into Deficiencies Associated with Commonly Used ZeroThickness Interface Elements. Computers and Geotechnics, 17 (1995) 225-252. 6. Kelley, C.T. : Iterative Methods for Linear and Nonlinear Equations. Frontiers in applied mathematics, SIAM, Philadelphia, 1995. 7. Löhner, R., Galle, M., Minimization of Indirect addressing for edge-based field solvers, Comm. Num. Meth. Engng. ,18 (2002) 335-343 8. Löhner, R., Edges, Stars, Superedges and Chains, Comp. Meth. Appl. Mech. Engng. (1994), 255-263. 9. Martins, M.A.D., Alves, J.L.D., Coutinho, A.L.G.A.: Parallel Edged-Based Finite Techniques for Nonlinear Solid Mechanics. In: Palma, J.M.L.M., Dongarra, J., Hernández, V. (eds.): Vector and Parallel Processing – VECPAR 2000. Lecture Notes on Computer Science, Vol. 1981, Springer-Verlag, Berlin Heidelberg (2001), pp 506-518. 10. Schellekens, J.C.J., De Borst, R.: On the Numerical Integration of Interface Elements, Int. J. Num. Meth. Engng., 36 (1993) 43-66.
Finite Element Simulation of Seismic Elastic Two Dimensional Wave Propagation: Development and Assessment of Performance in a Cluster of PCs with Software DSM Francisco Quaranta1, Maria C. S. de Castro2,3 , Jos´e L. D. Alves1 , and Claudio L. de Amorim2 1
2
Laborat´ orio de M´etodos Computacionais em Engenharia LAMCE/PEC/COPPE/UFRJ CP 68552 CEP 21949-900 Rio de Janeiro-RJ Brasil {quaranta,jalves}@lamce.ufrj.br http://www.lamce.ufrj.br Laborat´ orio de Computa¸ca ˜o Paralela - LCP/PESC/COPPE/UFRJ {clicia,amorim}@cos.ufrj.br 3 Departamento de Inform´ atica e Ciˆencia da Computa¸ca ˜o - UERJ
Abstract. In this work we approach the seismic wave propagation problem in two dimensions using the finite element method (FEM). This kind of problem is essential to study the structure of the earth’s interior and exploring petroleum reservoirs. Using a representative FEM-based application, we propose and evaluate two parallel algorithms based on the inverse mapping and on the mesh coloring, respectively. The distinguishing feature of our parallel versions is that they were implemented in a distributed shared-memory system (SDSMs), which offers the intuitive shared-memory programming model on a cluster of low-cost highperformance PCs. Our results for several workloads show that the inverse mapping scheme achieved the best speedup at 7.11 out of 8 processors, though the mesh coloring algorithm scales well. Overall, these preliminary results we obtained indicate that cluster-based SDSMs represent a cost-effective friendly-programming platform for developing parallel FEM-based applications.
1
Introduction
In solid mechanics, solving transient-state problems and the study of wave propagation effects are common research activities that are very important to several fields of engineering. In particular, a major application of interest in petroleum engineering is the analysis of seismic wave propagation in the interior of earth coupled with the study of seismic methods for determining the geological subsurface structure. Both the analysis and knowledge of sub-surface geology are essential requirements to find out petroleum reservoirs. However, applying analytical methods to such class of problems become a challenging task due to J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 65–78, 2003. c Springer-Verlag Berlin Heidelberg 2003
66
Francisco Quaranta et al.
the problem’s complexity. Therefore, an increasingly cost-effective alternative is to develop computational environments based on numerical methods such as the highly successful finite element method (FEM) to deal with the problem’s complexity. Nevertheless, given the large size of typical input data, even current high-performance microprocessors are not able to execute FEM-based wave propagation applications satisfactorily because either they are not fast enough or they do not provide sufficient RAM to efficiently process such large input data, or both. Usually, large-scale FEM-based applications run on high-performance vector computers or more recently on clusters of high-performance PCs using traditional message-passing libraries such as MPI and PVM. We note that past works [1, 2, 7, 12] examined performance of FEM-based applications on high-cost hardware shared-memory processors and described experiments using applications similar to the ones we use in this paper. In contrast to these works, we obtained a competitive solution using a low-cost cluster platform. On the other hand, recent research results show that software distributed shared-memory systems (SDSMs) offer a cost-effective platform for parallel computing that combines the intuitive shared-memory programming model with the low-cost of distributed memory systems, as offered by clusters of PCs. With this in mind, in this work, we design and implement parallel wave propagation applications in TreadMarks, a popular SDSM, running on a cluster of 8 high-performance PCs and evaluate performance of our applications for several workloads. We introduce two parallel versions of well-known methods to handle element assembling in FEM codes: 1) the inverse mapping scheme [12] and 2) mesh colouring[7]. Our preliminary results show that our parallel implementation of wave propagation applications attains speed-ups as much as 7.11 out 8 processors. These results suggest that cluster-based software DSMs represent a cost-effective platform for running FEM-based wave propagation applications. The remainder of this paper is organized as follows. Next section overviews the linear elasticity problem. Section 3 summarizes the TreadMarks software distributed shared-memory system we use. Section 4 describes our parallel versions of wave propagation application. In Section 5 we present the experimental methodology. Section 6 analyzes performance of our parallel applications on top of TreadMarks. Finally, in Section 7 we draw our conclusions and outline future work.
2
The Linear Elasticity Problem
In this section we treat the elasticity problem in solid mechanics by presenting a formal definition of the problem and its associated finite element method (FEM) semi-discrete form. In addition, we comment on relevant computational aspects of FEM.
Finite Element Simulation
2.1
67
The Elasticity Problem in Solid Mechanics
A solid body, can be submitted to tractions and stresses over a period of time, as the deformed body in the Figure 1 shows. The figure represents a physical system that consists of a body with domain Ω and boundary Γ in a reference time t. The body can be submitted to prescribed displacements, surface tractions, and volume forces, at any given time, so that the body will be displaced, deformed, and it will occupy its new domain in the next time step. For instance, the body occupies the domain 0 Ω in time t = 0 (left side) and the new domain t Ω in a time t > 0 (right side), after forces are applied to it. The system behavior is governed by four equation sets, namely, momentum conservation equation, constitutive equation, initial conditions, and boundary conditions. Both the equations and formulations for linear analysis in solid mechanics are described in detail in [3, 4, 6, 8, 13]. 2.2
Formal Definition
A formal definition of the problem can be described as follows. Consider the surface tractions t q in 0 ΓF and body forces t b in 0 Ω. The problem consists in finding the displacement fields t u in all times of the interval t ∈ [0, T ], that satisfy the problem conditions. The problem’s equations are shown in Figure 2. In Figure 2 t σij is the Cauchy stress tensor, t xj are material coordinates, t bi is the body force, tρ is the mass density, t ai are the accelerations, Cijkl is the elastic constitutive tensor, t εkl is the strain tensor and t u is a prescribed displacement. Displacement and velocity at the initial time are represented by u(x, 0) and v(x, 0), respectively. The FEM-based numerical solution for the problem can be found using a discretization of the domain. For instance, we used the bilinear quadrilateral element to obtain a discrete domain. The next section focus on the spatial and temporal finite element discretization. 2.3
Spatial and Temporal Discretization of the Problem
The spatial discretization of a solid body is achieved by subdividing the body domain Ω into finite elements with domain Ω e , as shown in Figure 3. In these
0 0
x3 x1
x2
ΓF
ΓD
t t
Γ
F
0
Ω
t
Ω
Fig. 1. System with a deformed solid body
ΓD
68
Francisco Quaranta et al.
1. Momentum conservation equation ∂ t σij + t bi = tρ t ai ∂ t xj
in
t
Ω
(1)
2. Constitutive equation t
σij = Cijkl t εkl
(2)
3. Initial conditions u(x, 0) =
0
u
in
0
Ω
v(x, 0) =
0
v
in
0
Ω
u = tu
in
t
in
t
(3)
4. Boundary conditions t t
t
t
σij nij = q i
ΓD
(4)
ΓF
(5)
Fig. 2. Formal definition of the problem 0 0
x3 x1
x2
ΓF
0
e
Ω
ΓD
t t
ΓF
t
ΓD
e
Ω
Fig. 3. Finite element discretization of the system
elements, the displacement field is approximated by shape functions that interpolate the displacement of nodal points [3, 4, 8, 13]. In addition, we use the central difference operator to discretize the interval time of interest [0, T ], yielding an explicit method of solution [4]. This method is used in the finite element solution of short transient problems in linear elasticity and is summarized by the algorithm in the Figure 4. In the next section, we show the routine that represents the algorithm of Figure 4 and comment on relevant computational aspects. 2.4
Computational Aspects
Figure 5 shows the pseudo-code that corresponds to the explicit solution algorithm (Figure 4). We notice that the percentage of execution time that is spent on I/O (data reading, variable initialization, and writing results) is negligible compared to the total execution time of the application. In fact, we focus on the portion of code responsible for approximately 98% of the total execution time of the application for all the input workloads we tested.
Finite Element Simulation
69
Initialization of variables t = 0, n = 0. Compute ∆t Apply the initial values u0 , v0 , F0p and F0int Compute the lumped mass matrix M Compute initial acceleration a0 = M−1 (F0p − F0int ) 1 1 vn− 2 + an ∆tn , if n > 0 5. Compute vn+ 2 = vn + 12 an ∆tn , if n = 0 1. 2. 3. 4.
6. 7. 8. 9. 10. 11.
1
Compute displacements un+1 = un + vn+ 2 ∆t Update un+1 , with boundary conditions t u n+1 Compute stress σ n+1 , internal forces Fn+1 int and external forces Fp n+1 n+1 −1 n+1 Compute acceleration a = M (Fp − Fint ) Update t = t + ∆t If t > T stop, otherwise update n = n + 1 and start again from item 5.
Fig. 4. Explicit solution algorithm ExpConv Routine Vector and matrix declarations Initialization of variables For all number of steps do !TIME INTEGRATION Call DISASSEMBLING vector of displacements Call DISASSEMBLING vector of velocity Call EVALUATE internal forces Call ASSEMBLING element contribution For all unknowns do !UPDATE GLOBAL UNKNOWNS Update accelerations, velocities, and displacements
Fig. 5.
Explicit solution routine
The explicit-solution algorithm divides the problem into the following four main tasks: gather operation, evaluation of internal forces, scatter operation, and update of global unknowns. The evaluation of internal forces and update of unknowns are relatively simple arithmetic operations that divide the loops of elements and equations among processors, since there is no data dependence between the components within the same loop. The scatter and gather tasks perform write and read accesses to memory. The scatter (disassembling) operation is immediate because array components are independently computed. However, scatter (assembling) operation requires special attention, since different processors can read and modify the same value thus requiring controlled access. Indeed, with an inadequate treatment, assembling operations can consume up to 40% of total computing time [1, 2, 12]. We consider and evaluate two well-known algorithms in the design of the assembling process. More specifically, the inverse mapping scheme introduced by Torigaki [12] uses the concept of equation connectivity while mesh colouring, presented in [7] uses a reordering of the finite element mesh in blocks of disjoint elements. Each algorithm has advantages and disadvantages that will
70
Francisco Quaranta et al.
be discussed in Section 6. A similar performance study of these algorithms was performed in [1, 2] using a shared-memory vector/parallel machine. Before describing our two parallel versions of the assembling task, we introduce next basic concepts related to software DSMs.
3
TreadMarks
Software distributed shared-memory systems or simply SDSM protocols implement the abstraction of shared-memory on a distributed memory system. Therefore, SDSM offer an attractive platform for parallel computing by combining the low-cost of distributed memory systems with the intuitive shared-memory programming model. However, software DSM protocols incur substantial overheads to ensure memory coherence over high-latency communication networks, that degrade SDSMs performance significantly[9]. In this work, we use TreadMarks (Tmk), a well-known page-based multiplewriter protocol SDSM as the reference SDSM protocol. Tmk protocol implements the lazy release consistency model [10](RLC) in which accesses to shared data are protected by acquire (e.g., lock) and release (e.g., unlock) operations on locks. Lock acquire is used to gain access to shared data and lock release to give such access away. In the LRC model, modifications made to shared data by a processor need to be seen only by the next processor that acquires the lock. To know which modifications precede the lock acquire Tmk divides the program execution into intervals in such a way that the intervals can be partially ordered by synchronization accesses. So, on lock acquire, the acquiring processor asks the last releaser of the lock for the information about the modifications made to shared data during the intervals that precede the acquiring processor’s current interval. A barrier operation can be seen as if each processor executes a lock release followed by a lock acquire. To alleviate the problem of false-sharing as Tmk use the large page as the coherence unit, Tmk implements a multiple-writer protocol that allows multiple processors to write to the same page concurrently. Further details on Tmk can be found in [11]. Overall, SDSM systems achieve respectable performance gains on regular applications in which memory accesses are coarse-grained [5]. Since our applications belong to this class, they are likely to achieve good speedups.
4
The Parallel Explicit-Solution Algorithm
In this section, we describe briefly the parallel versions of the assembling phase in the explicit-solution algorithm. In this application, we implemented the assembling phase using both the inverse mapping scheme and mesh colouring. We adapted these schemes and their data structures to run on a software distributedshared memory platform. As a result of the simplicity of the shared-memory model our parallel implementations were straightforward. More specifically, we use lock/unlock operations when pairs of processors need to synchronize and barriers to synchronize all processors.
Finite Element Simulation
71
Assembly inverse mapping Routine Distribute elements among processors For all private equations of processor do Assembly element contributions For all processors do If a shared equation is assigned to processor then Request exclusive access !LOCK acquire Assembly elements contributions Release exclusive access !LOCK release Synchronize all processors !BARRIER
Fig. 6.
Parallel inverse mapping scheme assembly
Parallel Inverse Mapping scheme Figure 6 shows the parallel inverse mapping scheme. The algorithm starts by dividing the total number of elements among the processors. Each processor first assembles the contributions of elements that are in its private data area. Next, the algorithm collects the contributions of the elements that have common nodes between two processors. To build these shared equations, it is necessary to use a lock operation to ensure exclusive access to shared data as well as to enforce data coherence. Lastly, a barrier is issued to synchronize all processors before initializing the next integration time step. Parallel Mesh Colouring Figure 7 shows the parallel mesh colouring algorithm. This algorithm executes a preliminary operation, not shown, that consists in reordering the elements in sets of disjoints elements. These sets are blocks of elements that do not have nodes in common so that elements in a block have the same colour. For each block, we divide the elements of the block among the processors. Next, we compute their contributions for all private and shared equations. At the end of block processing, a barrier primitive synchronizes all the processors. In the mesh colouring algorithm, we do not need to use a lock primitive, because of the characteristic of the elements that belong to the same block. As the elements within the same block do not have nodes in common, the processors never contribute to the same equation at the same time.
5
Experimental Methodology
In order to assess the performance of the two parallel versions of the explicit solution algorithm we used a cluster of 8 PCs. Each PC consists of a 650 MHz Dual-Pentium III, 512 MBytes of RAM, 256 KBytes of cache (L2), connected by a Fast-Ethernet switch with 100 Mbits/s. Our cluster runs Linux Red Hat 6.2 and our TreadMarks version uses only one processor per node. The application was implemented in Fortran. We ran each experiment 10 times and presented the average execution time in seconds. The standard deviation was less than 2%.
72
Francisco Quaranta et al. Assembly Mesh Colouring Routine For all element blocks do Divide elements of each block among processors For all elements assigned to processor do If is a privative element then Assembly elements contributions on private area Else Assembly elements contributions on shared area Synchronize all processors !BARRIER Synchronize all processors !BARRIER
Fig. 7.
5.1
Parallel mesh colouring assembly
Numerical Example
A bulk of layered soil is modeled for the numerical synthesis of seismic data for VSP (Vertical Seismic Profiling). This is a typical application within the scope of Geophysics with great importance for finding petroleum reservoirs [2]. In this kind of analysis, the propagation of seismic waves, generated from explosive charges detonation on the surface, through various subsoil layers is measured and interpreted to obtain geological data. The analyzed model considers three horizontal layers, each of which presents different speeds of propagation for the dilatational wave. Figure 8 illustrates the problem.
C L
1 0 0 1 0 1 Layer 2 0 1 0 1 0 1 Layer 3 0 1 0 1111111111111111 000000000000000 0 1 400 m Layer 1
Fig. 8. Problem description
100 m 100 m 100 m
P (t)
1 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Finite Element Simulation
73
Table 1. Discretization characteristics Mesh GEO5 GEO6 GEO7 GEO8
Domain Elements Nodes Equations 400m × 300m 19200 19481 38280 800m × 300m 38400 38841 76680 1600m × 300m 76800 77561 153480 3200m × 300m 153600 155001 307080
Table 2. The physical characteristics Layer Young modulus (KN/m2 ) Poisson ratio Density (KN/m3 ) 1 1.598917E+03 0.33 2.55E-01 2 3.197840E+02 0.33 2.04E-01 3 7.185440E+02 0.33 2.34E-01
For simulation of the physical behavior model and analysis of application performance, four different finite element meshes were used. In all cases mesh resolution was kept constant at a 2.5 square meters. In the vertical direction, the number of elements was kept always constant for each case. In the horizontal direction, the number of elements was twice the previous one. Table 1 shows the main features of each finite element mesh we used. It shows for each set of input data its label, the domain size, the number of elements, nodes, and equations. For all cases, due to the same dimension of the elements, an identical time step ∆t = 1.0E − 04s was used as well as 7000 steps for time integration. Table 2 presents relevant physical characteristics of the model. They are identical to those found in [2]. Figure 9 illustrates the results for the propagation of seismic waves in the addressed problem. The figures show distribution of nodal velocities in the mesh at analysis times t = 0.0700 s, t = 0.1575 s, t = 0.2275 s and t = 0.3150 s. Due the boundary conditions and physical characteristics adopted in this example, we observe at time t = 0.1575 s the effect of different wave speed and refraction of the waves between layers 1 and 2. At times t = 0.2275 s and t = 0.3150 s we observe the propagation of the waves after the reflection at the end of layer 3.
6
Performance Analysis
In this section, we present and discuss the performance results obtained from the analysis of the numerical example described in the previous section. For each parallel application, whether based on the inverse mapping scheme or the mesh colouring algorithm using the explicit solution algorithm we show the total execution time in seconds and speedups. We ran the application for 1, 2, 4, and 8 processors.
74
Francisco Quaranta et al.
(a) t = 0.0700 s
(b) t = 0.1575 s
(c) t = 0.2275 s
(d) t = 0.3150 s
Fig. 9. Seismic Wave Propagation
6.1
Inverse Mapping Scheme
The inverse mapping application achieved the best performance. Table 3 shows the execution time (ET) and speedups (Sp) for all the input data workloads. Figure 10 shows the speedup curve related to that table. The total execution times of the inverse mapping scheme were approximately proportional to the input data size. The speedups we obtained were very good for all input data and numbers of processors, revealing that the application is potentially scalable. The main reason is that it exhibits a coarse-grained access pattern. Also, the application presented superlinear speedups when we ran input data GEO7 for 2 processors. This is explained by the fact that as we move from 1 to 2 processors the larger memory size reduces cache pollution. Notice that the application achieved speed-up greater than 7 for GEO7. From 4 to 8 processors we obtained a slight decrease in application performance due to TreadMarks cache-coherence invalidate-based protocol. More specifically, as we increase the number of processors, so does the number of shared equations, which in turn causes the effects of page invalidations to get worse as we added more processors.
Finite Element Simulation
75
Table 3. Results using the inverse mapping scheme # Procs. GEO5 ET Sp 766.6 1.00 393.9 1.95 212.6 3.61 119.1 6.44
1 2 4 8
Input Data GEO6 GEO7 ET Sp ET Sp 1561.5 1.00 3231.7 1.00 789.3 1.98 1596.6 2.02 423.8 3.69 853.1 3.79 235.9 6.62 454.5 7.11
GEO8 ET Sp 6640.5 1.00 3342.1 1.99 1770.1 3.75 1068.8 6.21
8 Ideal Geo5 Geo6 Geo7 Geo8
7
Speedup
6 5 4 3 2 1
1
2
4
NProcs
8
Fig. 10. Results using the inverse mapping scheme
6.2
Mesh Colouring
The mesh colouring application attained a lower performance than the inverse mapping-based one. Table 4 shows the execution time (ET) and speedups (Sp) for all input data workloads. Figure 11 shows the speedup curve related to that table. For all input data sizes the speedups were inferior to that of the inverse mapping scheme, and total execution times were significantly greater for 4 and 8 processors. In general, the mesh colouring scheme did not show to be as scalable as the inverse mapping scheme, although it also has a coarse-grained access pattern. Note that for 8 processors and input data GEO8, it even slowed down. The reason is the same as in the inverse mapping scheme, TreadMarks’ cache-coherence protocol and memory consistency model. In addition, the mesh colouring algorithm deals with a different finite element mesh data structure that aggravates coherence-related overheads.
76
Francisco Quaranta et al.
Table 4. Results using mesh colouring # Procs. GEO5 ET Sp 804.8 1.00 432.7 1.86 232.2 3.47 140.3 5.73
1 2 4 8
Input Data GEO6 GEO7 ET Sp ET Sp 1641.6 1.00 3334.1 1.00 869.8 1.89 1720.9 1.94 468.2 3.51 921.4 3.62 273.0 6.01 536.0 6.22
GEO8 ET Sp 6852.5 1.00 3465.9 1.98 2145.2 3.20 1866.2 3.68
8 Ideal Geo5 Geo6 Geo7 Geo8
7
Speedup
6 5 4 3 2 1
1
2
4
NProcs
8
Fig. 11. Results using mesh colouring
6.3
Discussion
So far, the results we obtained confirm the potential parallelism of both algorithms. In the inverse mapping scheme, it is necessary to attribute the maximum number of elements that can be connected to every node within the finite element mesh, which limits the data structures that are suitable for a specific implementation. For instance, due to the two dimensional and regular characteristics of the meshes we used, our choice was 4 elements. In case of the data structure varies and the maximum value increases, a new assembling routine will be required. Although the mesh colouring algorithm performed worse than the inverse mapping scheme, it is more general and thus must be also taken into consideration.
Finite Element Simulation
7
77
Conclusions and Future Works
In this work, we proposed and evaluated the performance of two new parallel implementations of inverse mapping and mesh colouring algorithms that have been proposed for solving the elastic wave propagation problem in solid mechanics using the finite element method. We observed that this class of problem has great impact on several areas of engineering, and direct applications to some industrial sectors. In particular, it is afundamental problem in subsurface geology analysis to find reservoirs of petroleum. In contrast with previous works, our parallel algorithm versions are unique since they use the intuitive shared-memory model implemented in a cluster of PCs as provided by current software DSM systems. We evaluate our parallel algorithms for several workloads running on a cluster of 8 PCs under TreadMarks SDSMs. Our results show that the best speedup was approximately 7 out of 8 processors for the inverse mapping scheme, but almost all experiments we performed presented speedups as much as 6. The reason for the inferior mesh colouring performance was the larger communication overheads produced by TreadMarks cache-coherence protocol. We noticed that, TreadMarks executed relatively more synchronization operations for the mesh colouring implementation than it performed for the inverse mapping one, as we added more processors and increased the input data size. Overall, these preliminary performance results indicate that cluster-based SDSMs offer a cost-effective friendly-programming platform for developing parallel FEM-based applications. We still have opportunities for performance improvements. For instance, SDSM systems that use home-based cache coherence protocols is likely to improve performance of the parallel mesh colouring algorithm because it requires less memory than the inverse mapping scheme. This work is in progress at COPPE/UFRJ to examine related issues and also to evaluate other FEM-based applications, including experiments that use high-performance cluster networks such as Giganet and Myrinet. In this regard, preliminary results confirm performance improvements up to 20% for the mesh colouring algorithm using a Myrinet switch.
Acknowledgements This work was partially supported by FINEP grant “Projeto de Computa¸c˜ao Paralela”. Our experiments were performed on the cluster of PCs of Laborat´ orio de Computa¸c˜ao Paralela - LCP/PESC/COPPE/UFRJ. The authors also would like to acknowledge the collaboration and help-on-line of our colleagues of LCP team Ines Dutra, Lauro Whately, Raquel Pinto, Silvano Bolfoni, Thobias Salazar and Rodrigo Fonseca.
78
Francisco Quaranta et al.
References [1] Alvaro L. G. A. Coutinho, Jose L. D. Alves and Nelson F. F. Ebecken: A Study of Implementation Schemes for Vectorized Sparse EBE Matrix-Vector Multiplication. Adv. Eng. Software. Vol.13 N.3 (1991) 130–134 66, 69, 70 [2] J. L. D. Alves: An´ alise Transiente de Grandes Deforma¸co ˜es em Computadores de Arquitetura Vetorial/Paralela. Tese de Doutorado em Engenharia Civil, COPPE/UFRJ. (1991) 66, 69, 70, 72, 73 [3] K. J. Bathe: Finite Element Procedures in Engineering Analysis. Prentice-Hall, Englewood Cliffs. (1982) 67, 68 [4] T. Belytschko: An Overview of Semidiscretization and Time Integration Procedures. in T. Belytschko and T. J. R. Hughes (eds.), Computational Methods for Transient Analysis, North-Holland, Amsterdam. (1983) 1–65 67, 68 [5] M. C. S. Castro and C. L. Amorim: Efficient Categorization of Memory Sharing Patterns in Software DSM Systems. in Proc. of the 15th IEEE Int’l Parallel and Distributed Processing Symposium, IPDPS’2001, San Francisco/USA, April 2001 70 [6] R. D. Cook, D. S. Malkus, M. E. Plesha: Concepts and Application of Finite Element Analysis. 3rd. Ed., Wiley, New York. (1989) 67 [7] R. M. Ferencz: Element-by-Element Preconditioning Techniques for Large-Scale, Vectorized Finite Element Analysis in Nonlinear Solid and Structural Mechanics. Ph.D. Dissertation, Stanford University. (1989) 66, 69 [8] T. J. R. Hughes: The Finite Element Method: Linear Static and Dynamic Finite Element Analysis. Prentice-Hall, Englewood Cliffs (1987) 67, 68 [9] Y. Zhou, L. Iftode, K. Li, J. P. Singh, B. R. Toonen, I. Schoinas, M. D. Hill, D. A. Wood: Relaxed Consistency and Coherence Granularity in DSM Systems: A Performance Evaluation. PPOPP97 (1997) 193–205 70 [10] P. Keleher, A. L. Cox and W. Zwaenepoel: Lazy Release Consistency for Software Distributed Shared Memory. ISCA92. (1992) 13–21 70 [11] P. Keleher, S. Dwarkadas and A. L. Cox and W. Zwaenepoel: TreadMarks: Distributed Shared Memory on Standard Workstations and Operating Systems. Proc. of the 1994 Winter Usenix Conference. (1994) 70 [12] T. Torigaki: A Structure of an Explicit Time Integration Dynamic Finite Element Program Suitable for Vector and Parallel Computers. Proceedings of the 4th International Conference on Supercomputing and 3rd World, Supercomputer Exhibition (ICS’89). Santa Clara CA (1989) 449–453 66, 69 [13] O. C. Zienkiewicz, R. L. Taylor: The Finite Element Methods Vol.2: Solid and Fluid Mechanics, Dynamics and Non-Linearity. McGraw Hill, London. (1991) 67, 68
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete Carmen N. M. Paz1, Luiz F. Martha1, José L. D. Alves,2 Eduardo M. R. Fairbairn2, Nelson F. F. Ebecken2, and Alvaro L. G. A. Coutinho2 1
Department of Civil Engineering Pontifical Catholic University of Rio de Janeiro, (PUC-Rio) Street Marquês de São Vicente 225, 22453-900 - Brazil {mena,lfm}@tecgraf.puc-rio.br 2 COPPE/UFRJ - Program of Civil Engineering PO Box 68506, 21945-970 Federal University of Rio de Janeiro - Brazil [email protected] [email protected] [email protected] [email protected]
Abstract. This work presents computational strategies used in an implementation of the probabilistic discrete cracking model for concrete of Rossi suitable to parallel vector processor (PVP). The computational strategies used for each realization, within the framework of Monte Carlo simulation, are the inexact Newton method to solve the highly nonlinear problem and element-by-element (EBE) iterative strategies considering that nonlinear behavior is restricted to interface elements. The simulation of a direct tension test is used to illustrate the influence of adaptive inexact Newton strategy in code performance on a CRAY T90.
1
Introduction
Concrete, the most used construction material in the world, is a very heterogeneous material, because of its composite structure composed by aggregates, mortar, fibers, etc. and also because of the physical phenomena that take place during chemical and physical evolution: autogenous shrinkage, initial stresses and microcracking, etc. The material can also present fracture size effects that may introduce large differences for the ultimate loads and non-objectivity of the analysis. To handle with these characteristics, two basic approaches are generally used to model the mechanical behavior of the material: (i) the heterogeneous material is assimilated to a statistically equivalent homogeneous material and a deterministic approach together with a size effect law is used; (ii) homogeneity is no longer considered, at a certain scale, and the heterogene-
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 79-93, 2003. Springer-Verlag Berlin Heidelberg 2003
80
Carmen N. M. Paz et al.
ity of the material, introduced by a stochastic approach, is the responsible for the scale effect. Regarding concrete, the second approach is quite recent, and in this paper we refer to Rossi’s et al. [1,2,3,4] probabilistic model that seems to take into account size effects when the statistical parameters are well determined. This method uses Monte Carlo simulations to find the average behavior of the structure, which corresponds to the converged mean values of some typical results. The remaining of this work is organized as follows; section 2 describes the probabilistic model; section 3 describes the formulation of the interface element; section 4 describes the computational strategies employed and section 5 addresses the numerical experiments carried out. Finally, section 6 closes this work presenting some conclusions drawn from the experiments.
2
Probabilistic Model
Among several other relevant factors, such as water/cement ratio of the paste, casting and curing conditions, loading conditions, etc, concrete cracking depends on the random distribution of constituents and initial defects. The heterogeneity governs the overall cracking behavior and related size effects on concrete fracture. The probabilistic crack approach, based on the direct Monte Carlo method, developed by Rossi and co-workers ( Rossi et al. [1,2,3,4]) takes this stochastic process into account by assigning in finite element analysis, randomly distributed material properties, such as tensile strength and Young's modulus to both the solid elements and the contact elements interfacing the former, that is, a discrete crack approach. This approach considers that all nonlinearities are restricted to contact elements modeling cracks. Therefore, the stochastic process is introduced at the local scale of the material, by considering that cracks are created within the concrete with different energy dissipation depending on the spatial distribution of constituents and initial defects. The local behavior is assumed to obey a perfect elastic brittle material law. Thus, the random distribution of local cracking energies can be replaced by a random distribution of local strengths. The present probabilistic model involves a number of mechanical properties of the material to be determined, which constitutes the modeling data. From a large number of direct tensile tests, it was found that a normal law describes rather well the experimental distribution of the relevant material data (Rossi et al. [2]). These characteristics are the means of the tensile strength ( f ct , µ ) and of the Young’s modulus ( E µ ); the standard deviations of the tensile strength ( f ct ,σ ) and of the Young’s modulus ( Eσ ). The following analytical expressions were proposed (Rossi et al. [2]):
f ct , µ = 6.5 ⋅ C ⋅ (VT Vg ) − a Eµ = E
f ct ,σ = 0.35 ⋅ f ct , µ (VT Vg ) − b
Eσ / E = 0.15 (Vt / V g ) − c
(1)
(2)
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
81
a = 0.25 − 3.6 ⋅10 −3 ( f c C ) + 1.3 ⋅10 −5 ( f c C )
2
b = 4.5 ⋅10 − 2 + 4.5 ⋅10 −3 ( f c C ) + 1.8 ⋅10−5 ( f c C )
2
(3)
c = 0.116 + 2.7 ⋅10 −3 ( f c C ) − 3.4 ⋅10 −6 ( f c C )
2
where VT is the volume of the two finite elements contiguous to an individual contact element of the mesh, Vg is the volume of the coarsest grain, C = 1MPa , f c is the cylinder compressive strength determined by the standardized test (French standard) on a cylinder 160 mm in diameter and 320 mm high, and E is the mean value (obtained from tests) of the Young’s modulus which does not exhibit significant volume effects. In these expressions, the compressive strength f c represents the quality of the concrete matrix, while the volume of the coarsest aggregate V g , refers to the elementary material heterogeneity. Equations (1) to (3) show that the smaller the scale of observation, the larger the fluctuation of the local mechanical properties, and thus the (modeled) heterogeneity of the matter. In other words, the finer the mesh, the greater the modeled heterogeneity in terms of Young's modulus and tensile strength. The empirical expressions corresponding to equations (1) to (3) were calibrated to fit a number of material tests for different types of concretes and volumes. They give a first approximation to capture volume effects on concrete fracture, but they are not expected to be universal. Furthermore, little is known about the validity of these empirical formulas for small Vt / V g . These volume ratios, which may be used in finite element structural analysis,
are too little to be determined by means of experimental tests. The mesh has nv volume elements and ni interface elements. Each interface element follows a rigid-brittle constitutive law characterized by an individual tensile strength f ct ,i . The volume elements are elastic and have individual Young’s modulus referenced by Ev . Following Rossi et al. [2] findings, these individual local tensile strengths and Young’s modulus are represented by normal distributions having the densities:
g f ( f ct ) =
g E (E ) =
1 f ct ,µ exp − 2 f ct ,σ 2π
1E−E µ exp − 2 Eσ 2π
1 f ct ,σ
1 Eσ
2
(4)
2
(5)
82
Carmen N. M. Paz et al.
where g f ( f ct ) and g E (E ) are the density function for the tensile strength f ct and the Young’s modulus E , x µ and xσ denote the mean and standard derivation of the distribution of quantity x . For the problem at hand, it is possible to find a sample of
ni values f ct ,i , each value corresponding to an interface element, and nv values E v , each value corresponding to a volume element, by using a standard routine for generation of random numbers for a given normal distribution (Press et al., [6]). If the heterogeneous characteristics of the material are well established and quantified by the statistical moments it is possible that the model displays the size effects related to the material heterogeneity. The problem with this approach is that these statistical moments are not known, a priori, for the characteristic volume of the finite elements used. However, some methods have been proposed to determine these parameters by means of inverse analysis using neural networks (Fairbairn et al, [7,8]). The solution for this probabilistic approach is obtained by means of a Monte Carlo simulation. A number of n samples is generated for a given normal distribution, and some characteristic responses of the structure are computed; for example, stress crack-width σ − w curve, or load displacement P − δ curve. Let the jth samples correspond to, e.g., the jth σ − w curve. This jth σ − w curve is composed of discrete values, σ kj and wkj , where the superscript j indicates the sample and the subscript k the discrete value of the σ − w curve. The same discrete
w − values are assumed and
σ kj
. The mean curve composed
the response is defined exclusively by the values of by pairs
σ kmean , wk
σ kmean, j =
1 j
then simply reads j
∑σ
l k
(6)
l =1
The Monte Carlo simulation is stopped when
σ kmean, j − σ kmean, j −1 ≤ tol
(7)
where tol is the prescribed tolerance to check the convergence of the procedure. With the convergence of the procedure, the total number of samples is set to n = j . This total number of samples n, corresponding to a Monte Carlo converged simulation, clearly depends on tol, which is a measure of the precision required by the analysis. It also depends on the heterogeneity of the material represented by the standard deviation. The more heterogeneous is the material the greater is the number of samples necessary to obtain a converged solution. Our experience in this field indicates that 25 to 50 samples are sufficient to obtain a converged σ − w curve.
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
83
Fig. 1. An Interface Element and its degrees of freedom in a local system
3
Interface Elements for Modeling Discrete Cracking
The finite element cracking model is a discrete model for which volume elements are always elastic and cracking occurs in elastic-brittle (almost rigid brittle) contact elements placed between two volume elements. The three dimensions interface element (Paz, [5]) depicted in figure 1 can be thought as a triangular base prism connecting adjacent faces of neighboring tetrahedra. These elements are formulated to describe relative displacements between the triangular faces simulating crack opening. The constituve law of the 3D interface element is defined by equation (8) for non cracked elastic state characterized by σ n < f ct ,i . When the tensile strength is reached the elements attains the cracked stage and modulus E c and G c are set to zero (figure 2).
0 0 ∆wn ∆σ n Ec h ∆σ = D cr ∆ w = ∆σ s = 0 Gc h 0 ∆ws ∆σ 0 0 Gc h ∆wt t
Fig. 2. Elastic-Brittle Contact law
(8)
84
Carmen N. M. Paz et al.
In equation (8) the subscript n normal stands for, while stand for s and t tangential indicating the direction respective to the crack plane, w are the relative displacements between the two faces of the interface element, h is the width of interface element, E c and Gc are respectively the normal (Young’s) and the shear modulus along crack plane. Equations (8) and figure 2 define the elastic-britle constituve behavior. The thickness h plays the role of a penalization parameters and should be conveniently chosen not to affect the solution. The kinematic relation for the interface element is;
∆ w = B ∆ a le
(9)
where − 1 0 0 −1 0 0 −1 0 0 1 0 0 1 0 0 1 0 0 1 B = 0 −1 0 0 −1 0 0 −1 0 0 1 0 0 1 0 0 1 0 3 0 −1 0 0 −1 0 0 −1 0 0 1 0 0 1 0 0 1 0
(10)
∆ w is the crack opening incremental and ∆ a el is the vector incremental nodal displacements for the interface element
Applying a standard displacement based F. E. formulation, the resulting tangent stiffness matrix for the interface element is given is: K eIntf = ∫ BT Dcr B dΩ Ω
(11)
The interface elements are generated contiguous to the faces of selected tetrahedra elements. This selection is performed by the user, defining a 3D box inside the mesh that contains the target elements.
Remark Our experience indicates that to increase robustness of nonlinear solution process we have to limit only one interface element to “crack” at each nonlinear iteration.
4
Computational Strategies
In this work we employ the inexact Newton method (Kelley, [9]) to solve resulting nonlinear set of equations at each load or displacement increment. The outline of the nonlinear solution algorithm is presented in Box 1.
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
Given
u tol , rtol , η
85
relative and residual tolerance.
Compute stiffness tetrahedra matrix K Tetra do k=1,2….., number of load increments do Compute external forces vector − − k K Fext = Fnodal + Fvolume + F − − K tetra U k + K Intef U k σ
do i=1,2 ..., while convergence Compute internal forces vector,
(
Finti = Finti
)
Tetra
(
+ Finti
)
Intf
Compute residual vector, i ψ i = Finti − Fext
Update stiffness interface matrix K iIntf Assembly matrix
A i = K Tetra + K iIntf Compute tolerance for iterative driver , Solver: A i ∆u = ψ i for tolerance Update solution,
U = U + ∆u ∆u if ≤ utol and U
∆ψ i k Fext
ηi
ηi
≤ rtol then convergence
end while i. end do k. Box 1. Inexact Newton Algorithm k Note that in Fext we account for nodal forces, body forces and prescribed dis−
−
i placements and stresses U , σ . The total internal forces vector Fint is the sum of the
(
i tetrahedra element vector internal forces Fint
nal forces vector
(
i Fint
)
Intf
)
Tetra
plus the interface element inter-
. The total stiffness matrix is the sum of the continuum
i matrix K Tetra plus the interface matrix K Intf updated at every nonlinear iteration.
An approximate solution is obtained when the Inexact Newton termination criterion is satisfied, that is A i ∆u − ψ i ≤ η i ψ i
(12)
86
Carmen N. M. Paz et al.
Tolerance η i may be kept fixed throughout the nonlinear iterations or may be adaptively selected as the iterations progress towards the solution. Our choice for ηi follows the criteria suggested by Kelley. The iterative driver for the solution of the linearized set of equations is the element-by-element PCG. Several preconditioner options are available ranging from the simple diagonal up to incomplete factorizations. In this work, to stress the benefits of the inexact Newton method, we restricted ourselves to simple diagonal and nodal block diagonal.
Remarks 1) In the present implementation matrix vector products in EBE, PCG are computed as: N tetra
Ap = ∑
i =1
( K Tetra
)
N Intf
pi + ∑
j=1
(K Intf
pj
)
(13)
where, Ntetra is the number of tetrahedra, Nintf is the number of interface elements, K Tetra and K Intf are respectively element matrices for the tetrahedra and interface; p i and p j are the components of p restricted to the degrees of freedom of two element type. 2) Stiffness matrices for tetrahedra are computed and stored at the beginning of the analysis since they are elastic. 3) Stiffness matrix for interface elements are updated at every nonlinear iteration. 4) The arrays of element stiffness matrices are stored taking into account their symmetry; in the case of the element tetrahedra 78 coefficients are stored and for the interface element only 18 coefficients are stored, exploring the particular structure of the discrete gradient operator given in equation (10). 5) The mesh coloring algorithm of Hughes [10] was extended to block both solid and interface elements into disjoint groups thus enabling full vectorization an parallelization of the operations involved in equation (13).
5
Numerical Experiments
5.1
Direct tension test
The experimental results of concrete uniaxial tension tests published by (Li et al [12]) were used to validate the developments presented in this paper. The specimens are cylinders 101.6 mm in diameter and 203.2 mm high. This specimens had 25.4 mm notches at their midheight on both sides (figure 3). Concrete with maximum aggregate diameter of 9.525 mm was used. Its average tensile strength and Young’s modulus at the age of 28 days were: f ct = 4,72 MPa and E c = 42000 MPa .
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
87
Fig. 3. Uniaxial tension specimen geometry, dimensions, load, boundary conditions
The numerical experiments were controlled by a field of uniform displacements applied at the top of the test specimen in 30 incremental steps ∆u = 5.3 x 10 −6 . The boundary conditions restrain the degrees of freedom in the vertical at the bottom. The final mesh has 11,933 elements, where 4,775 are tetrahedra and 7,158 are interface elements (figure 4).
Fig. 4. Hidden line (a) and wire frame (b) representations of the computational mesh for the simulation of a direct tension test. It is also shown aside the resulting mesh for the interface elements considering
88
Carmen N. M. Paz et al.
3.5
σ
(MPa)
3
S a mple s 1 - 30 Me a n
2.5 2 1.5 1 0.5 δ
0 0
0.00005
0.0001
0.00015
0.0002
Fig. 5. Results for the complete Monte Carlo simulation
3.5
σ (Mpa) Numerical Experimental
3 2.5 2 1.5 1 0.5
δ
0 0
0.00005
0.0001
0.00015
0.0002
Fig. 6. Comparison of experimental and numerical results
Curves σ − δ for the several realizations of the Monte-Carlo simulation are given in figure 5. The comparison between experimental by Li et al [12] and the convergent Monte Carlo σ − δ curve is given in figure 6, figure 7 presents the crack configuration for a given sample at a stage corresponding to the softening branch of the σ − δ curve.
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
89
Fig. 7. Crack evolution for numerical simulation
5.2
Comparison of Inexact Newton Strategies
An assessment of the implemented inexact Newton strategies was carried out for one realization of the direct tension test simulation the selected preconditioner for this experiment was the diagonal nodal block. A fixed tolerance η = 10 −3 for PCG was selected for the first analysis requiring 52 inexact Newton iterations. For the second analysis, now employing the adaptive inexact Newton method, the tolerance range for PCG was set to 10 −3 ≤ η ≤ 10 −6 , requiring 54 inexact Newton iterations. Tolerances for the inexact Newton were iterations set to u tol = rtol = 10 −3 , in both cases. As can be seen in Figure (8), comparing the number of nonlinear iterations for both analyses, it remained fairly the same irrespective to the selection of the inexact Newton approach, either fixed or adaptive, indicating that the overall nonlinear character of the solution was not affected the adaptive approach. Figure (9) shows a plot of the number of PCG iterations per incremental step for both analyses. It is evident in this figure the reduction of PCG iterations due to the adaptive strategy. 5.3
Computational Performance
A detailed vector performance analysis is obtained by the summary of the PERFVIEW's Report presented in Table 1. The CPU time of the vectorized single processor run for CRAY T90 is 10.38 hours. This table list for a single CPU run, the Mflop/s rates for the three top routines. The routines Smatv-fint and Smatv-tetra are respectively responsible for the matrix-vector products for the interface and tetrahedra elements, routines are the computations kernels the routine PCG-block, implements the iterative solver the nodal block diagonal Preconditioned Conjugate Gradient.
90
Carmen N. M. Paz et al.
5
Fixed
Nonlinear iterations
Adaptive 4
3
2
1 0
5
10
15
20
Steps Fig. 8. Nonlinear iterations per incremental step for the direct tension test simulation
1,0E+07 Fixed Tolerance
PCG iterations
Adaptive 1,0E+06
1,0E+05
1,0E+04 1
3
5
7
9
11
13
15
17
19
Steps Fig. 9. PCG iterations per incremental step for the direct tension test simulation
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
91
Table 1 Performance Analysis -The top 3 subroutines
Routines smatv-intf smatv-tetra PCG-block Others
Single CPU (%) 52.80 17.32 25.90 3.98
Performance (Mflop/s) 613.7 554.0 82.4 -
Table 2 Summary of the ATEXPERT’s Report for the 5 dominant loops
Routines
% Parallel
Smatv-tetra Smatv-intf Fint-tetra Fint-intf A-Kintf
98.9 99.8 92.5 99.1 85.8
Dedicated Speedup 3.96 3.83 3.83 3.55 3.53
Actual Speedup 3.8 3.8 3.2 3.5 2.6
4
4
3
3
2
Amdahl's Law Dedicated System
1 0
Speed-up
Speed-up
The code achieved good vectorization on the CRAY T90 for a mesh with 11933 elements, comprising (7158 interface and 4775 tetrahedra elements). The top three subroutines consume the major CPU utilization in the whole analysis. The parallel performance is shown in table 2 and the figure 10 (a) and (b) as obtained from a summary ATEXPERT report, autotasking performance tool. The routines Fint-tetra and Fint-intf evaluate respectively the internal force vector of the interface elements and the tetrahedra elements. Routine A-Kintf computer update interface stiffness. According to ATEXPERT tool this program appears to be 99.2 percent parallel and 0.8 percent serial. Amdahl's Law predicts the program could expect to achieve a 3.9 times speedup on 4 cpus. A 3.8 speedup is predicted with 4 cpus on a dedicated system.
smatv-tetra smatv-intf
2
Fint-tetra Fint-intf
1
A-Kintf
0
0
1
2 CPUS
(a)
3
4
0
1
2
3
4
CPUS
(b)
Fig. 10 (a) Program Summary of the ATEXPERT’s Report (b) ATEXPERT’s Report -The top 6 subroutines on a Dedicated Speedup
92
6
Carmen N. M. Paz et al.
Concluding Remarks
This paper presented computational strategies used in an implementation of Rossi’s probabilistic model for the simulation of cracking in concrete structures. The resulting code achieved very good level, for parallel and vetor performance single CPU achieved 614 Mflop/s while parallel speed-up reached 3.8 in a 4 CPU system. The results emphasizes the suitability of the implemented code on a PVP system the CRAY T90 Extensive use of element-by-element techniques within the computational kernels comprised in the iterative solution drivers provided a natural way for achieving high flop rates and good parallel speed-up's. The use of an adaptive inexact Newton method showed to be effective in reducing overall linear iterations, by a factor of two in comparison to the fixed tolerance inexact Newton scheme.
Acknowledgments The authors are indebted to the Computer Graphics Technology Group TECGRAF/PUC-Rio, High Performance Computing Center NACAD/COPPE/UFRJ, and the Laboratory of Computational Methods in Engineering of the Program of Civil Engineering LAMCE /COPPE/UFRJ. CESUP/UFRGS is gratefully acknowledged for the computer time provided for the performance experiments. This work was partially supported by CAPES and CNPq's grant No 150039/018(NV).
References 1.
Rossi P., Richer S.: Numerical modeling of concrete cracking based on a stochastic approach. Materials and Structures, 20 (119), (1987) 334-337. 2. Rossi P., Wu X., Maou F. le, and Belloc A.: Scale effect on concrete in tension. Materials and Structures, 27 (172), (1994) 437-444. 3. Rossi P., Ulm F.-J., and Hachi F.: Compressive behavior of concrete: physical mechanisms and modeling. Journal of Engineering Mechanics ASCE, 122 (11), (1996), 1038-1043. 4. Rossi P. and Ulm F.-J.: Size effects in the biaxial behavior of concrete: physical mechanisms and modeling. Materials and Structures, 30 (198), (1997) 210-216. 5. Paz C. N. M., Development and Implementation Probabilistic Model for 2D and 3D Discrete Cracking Concrete in Parallel Computing, D.Sc. Thesis, COPPE/UFRJ, the Graduate Institute of Federal University of Rio de Janeiro, Brazil (2000) [in Portuguese]. 6. Press W. Teukolski H., S., Vetterling W.T. and Flannery B.: Numerical Recipes, Cambridge University Press (1992).
Parallel Implementation for Probabilistic Analysis of 3D Discrete Cracking in Concrete
7. 8. 9. 10.
11. 12. 13.
14.
93
Fairbairn E. M. R., Paz C. N. M., Ebecken N. F. F., and Ulm F-J., Use of neural network for fitting of probabilistic scaling model parameter, Int. J. Fracture, 95, (1999), 315-324. Fairbairn E. M. R., Ebecken N. F. F., Paz C. N. M., and Ulm F-J.: Determination of probabilistic parameters of concrete: solving the inverse problem by using artificial neural networks, Computers and Structures, 78, (2000) 497-503. Kelley C. T.: Iterative Methods for Linear and Nonlinear Equations. Frontiers in applied mathematics, SIAM Society for Industrial and Applied Mathematics, Philadelphia, (1995). Hughes T. J. R., Ferenez R. M. , Hallquist J. O.: Large-Scale Vectorized Implicit Calculation in Solid Mechanics on a CRAY X-MP/48 Utilizing EBE Preconditionated Conjugate Gradients Computer Methods in applied Mechanics and Engineering, 61, (1987) 2115-248. Papadrakakis M.: Solving Large-Scale Problems in Mechanics: The Development and Application of Computational Solution, Editor, M. Papadrakakis, John Wiley and Sons, (1993). Li Q. and Ansari F.: High Concrete in Uniaxial Tension, ACI Material J. 97(1), (2000) 49- 57. Coutinho A. L. G. A., Martins M. A. D., Alves J. L. D., Landau L., and Moraes A.: Edge-based finite element techniques for nonlinear solid mechanics problems, Int. J. for Numerical Methods in Engineering, 50 (9), (2001) 20502068. Fairbairn E. M. R., Debeux V.J.C., Paz C. N. M., and Ebecken N. F. F.: Application of probabilistic Approach to the Analysis of gravity Dam Centrifuge Test,
8 th ASCE Specialty Conference on Probabilistic Mechanics and Structural Reliability (2000) PMC 2000-261. 15. Kelley C. T.: Iterative methods for optimization. Frontiers in applied mathematics, SIAM Society for Industrial and Applied Mathematics, Philadelphia, (1999).
An A Posteriori Error Estimator for Adaptive Mesh Refinement Using Parallel In-Element Particle Tracking Methods Jing-Ru C. Cheng1 and Paul E. Plassmann2 1
Information Technology Laboratory US Army Engineer and Research Development Center Vicksburg, Mississippi 39180 [email protected] 2 Department of Computer Science and Engineering The Pennsylvania State University, University Park, PA 16802, USA [email protected]
Abstract. Particle tracking methods are a versatile computational technique central to the simulation of a wide range of scientific applications. In this paper we present an a posteriori error estimator for adaptive mesh refinement (AMR) using particle tracking methods. The approach uses a parallel computing framework, the “in-element” particle tracking method, based on the assumption that particle trajectories are computed by problem data localized to individual elements. Adaptive mesh refinement is used to control the mesh discretization errors along computed characteristics of the particle trajectories. Traditional a posteriori error estimators for AMR methods inherit flaws from the discrete solution of time-marching partial differential equations (PDEs)—particularly for advection/convection-dominated transport applications. To address this problem we introduce a new a posteriori error estimator based on particle tracking methods. We present experimental results that detail the performance of a parallel implementation of this particle method approach for a two-dimensional, time-marching convection-diffusion benchmark problem on an unstructured, adaptive mesh.
1
Introduction
1.1
Adaptive Mesh Refinement
A variety of numerical errors (e.g., spurious oscillation, numerical spreading, grid orientation, and peak clipping/valley elevating) have been described in the solution of advection/convection-dominated problems [3, 31, 30]. These errors can be partially mitigated through the implementation of higher-order finite element or finite difference discretizations, for example TVD and ENO methods [12, 29], because of the increased spatial and/or temporal resolution. However, instead of
This work was supported by NSF grants DGE–9987589 and ACI–9908057, DOE grant DG-FG02-99ER25373, and the Alfred P. Sloan Foundation.
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 94–107, 2003. c Springer-Verlag Berlin Heidelberg 2003
An A Posteriori Error Estimator for Adaptive Mesh Refinement
95
uniformly refining the computational mesh throughout the domain, an adaptive mesh refinement approach can be more effective in its use of computer resources for large-scale applications. Adaptive mesh refinement (AMR) improves the mesh resolutions in small areas where the numerical error is large, while remaining areas retain a coarse discretization because the computed solution in this part of domain is relatively accurate. A wide variety of AMR approaches have been explored in the literature [6, 24, 27, 8, 20, 21, 10, 13]. Jones and Plassmann [13] summarized a typical parallel finite-element AMR approach. Within this framework the mesh is adaptively refined when the maximum of computed local error on an element is greater than a prescribed error tolerance. The error estimator or indicator used is, therefore, crucial to the accuracy of the computed solution. A priori error estimates are, however, often only sufficient to guarantee the regularity of the solution [25]. To estimate the global error and to provide information for the adaptive mesh refinement algorithms discussed above we require a posteriori error estimates. Among all the partial differential equations, the mathematical analysis and numerical solution of convection-diffusion (or transport) equations has been intensively studied during the last decade [18, 23]. One often encounters numerical difficulties when solving the transport equations because of their parabolic and hyperbolic nature. Kay and Silvester [14] proposed a simple a posteriori error estimator based on the solution of local Poisson problems for the steady-state advection-dominated transport equations. They conclude that the method works effectively as long as streamline-diffusion stabilization (e.g., upstream winding) is incorporated. However, the solution of the unsteady transport equations are required in the majority of real-world applications for which this method would not be applicable. 1.2
Particle Tracking Applications
Particle tracking methods are used in many diverse applications. For example, in computer graphics the term “particle system” is used to describe the particle-based techniques for modeling, rendering, and animation. The Particle System Application Programmer Interface (API) was developed to provide a set of functions allowing applications to simulate the dynamics of particles [16]. This API design is aimed to add a variety of particle-based effects to interactive or non-interactive graphics applications. However, it was not intended for scientific simulation—although it has been implemented on both uniprocessor and multiprocessor shared memory systems. Parallel particle tracking methods have been employed to solve a variety of scientific and engineering simulation problems. Usually particle-based simulations are used in the solution of N-body problems. For these systems, the particles’ motion is determined from a force such as that resulting from gravitation, the Coulomb force, or the strong nuclear force in plasma models (such as those used in modeling fusion reactors). The bottleneck in these simulations is the computation of force on each particle, not the movement of particles. For these particle-particle (PP) methods the computational domain is not discretized—
96
Jing-Ru C. Cheng and Paul E. Plassmann
the particle motion is solved for by using a forward Euler method with a small time step chosen to ensure stability. On the other hand, the particle-mesh (PM) method uses not only a discretization in time but also a discretization of the spatial domain. For PM methods an algorithm is required to handle the assignment of “virtual” mass/charge values to the computational mesh. The PM approximation scheme can result in a computational efficient methods, but the accuracy is often not adequate for most applications. A third method used to handle particle-based models is the Tree Code (TC) method (such as Barnes-Hut [1]). This approach is again aimed at reducing the overall computation cost, but while retaining a specified level of accuracy. A quadtree in two dimensions or octree in three dimensions is built to count the total mass of all particles in a tree box. At the tree box center will be placed a virtual particle with the same amount of the the counted total mass. This method is similar to the PP method, except the tree implementation can reduce the overall complexity from O(N 2 ) in the PP approach to O(N log N ). All the above methods can be parallelized on distributed memory MIMD (Multiple Instruction Multiple Data) machines. However, the main focus of these methods is the calculation of the forces on the particles because computing the particle motion is straightforward when the restriction of a small time-step imposed. Another application is the Direct Simulation Monte Carlo (DSMC) method used to model the dynamics of dilute gas. Wong and Long [28] implemented a parallel DSMC algorithm using a forward Euler, explicit time marching scheme in the particle movement phase of the algorithm. They pointed out that two different levels of data parallelism (i.e., molecules and cells) cause parallel processing difficulties. Nance et al. [19] parallelized DSMC using the runtime library CHAOS [17] on a three-dimensional, uniformly discretized mesh. Robinson and Harvey [22] parallelized DSMC by use of a spatial mesh decomposition over the processors. This domain decomposition is based on a localized “load table” computed on each processor for its neighbors. Based on this load table cells are donated to these neighboring processors if they have lesser load. This approach was demonstrated for a two-dimensional driven cavity with approximately 10,000 uniform cells. Another field of application is the calculation of streamlines for a given vector field, for which the most common technique is the integration of particle paths numerically or analytically. Often, for the numerical approach, a high-order ordinary differential equation (ODE) solver such as a Runge-Kutta method or Adam’s method is required to maintain a high order of accuracy [15]. However, in two and three dimensions, the evaluation of velocity between grid points across processors required by the high-order ODE solver becomes too computationally expensive, and alternative methods have to be considered. Analytic solutions for streamlines within tetrahedra have been derived based on linear interpolation and, therefore, produces exact results for linear velocity fields [9]. However, these analytic solutions apply to only steady flow and are not applicable to unsteady, or time-dependent, flows.
An A Posteriori Error Estimator for Adaptive Mesh Refinement
1.3
97
The Parallel In-Element Particle Tracking Method
Cheng et al. [2] developed the sequential in-element particle tracking technique to accurately and efficiently trace fictitious particles in the velocity fields of real-world systems. Since in-element particle tracking methods are implemented with respect to the element basis, a natural approach is to develop a parallel framework based on specifying local, element-based operations. The key to the parallel algorithm is the correlated partitioning of the particle system and the unstructured element mesh. We partition particles to processors based on their element location—this approach ensures that the data required for the computation of the particle movement phase by the in-element method involves only data local to a processor. The correspondence between particles and elements is maintained through explicit references in the element and particle data structures. This correspondence is essential to ensure the correct reassignment of particles between processors. The reassignment occurs when particles move between elements owned by a processor and the ghost elements (elements with shared faces, edges, or vertices that are owned by another processor) [5]. In this paper, we present a new a posteriori error estimator to control adaptive mesh refinement of time-marching transport equations using parallel particle tracking methods. We present this research as follows. In Sect. 2, we define measure metrics of the error estimator to assess the quality of an a posteriori error estimator. In Sect. 3, we briefly describe the implementation of parallel inelement particle tracking technology. In Sect. 4, we present the new a posteriori error estimator for AMR methods after reviewing the state-of-the-art error estimators. A detailed illustration is provided to clarify how it can achieve our goals. In Sect. 5, experimental results are presented to demonstrate the improvement of our implementation. Both accuracy and performance aspects are included. In Sect. 6, we summarize these results and discuss planned future work.
2
Measure Metrics of the Error Estimator
To measure the quality of an a posteriori error estimator, an index is defined as the ratio of the estimated error and the true error and is called efficiency index, Ieff . According to [25, 26], Ieff is given as Ieff =
η , u − uh
(1)
where u − uh is the true error measured by the norm of the difference between the true solution and the discretized solution; and η is the estimated global error as shown in (2) 1/2 2 η= ηK , (2) K∈Γh
where ηK is the estimated error on the element K of the computational mesh −1 Γh , i.e., via the error estimator. If both Ieff and Ieff are bounded for all discretizations, the a posteriori error estimator is called efficient. Moreover, if they
98
Jing-Ru C. Cheng and Paul E. Plassmann
are bounded independently of the coefficients of the particular problem, then the efficient a posteriori error estimator is called robust for that class of problems [25]. To investigate the performance of the a posteriori error estimator with respect to adaptive mesh refinement, in Sect. 5 we compute the computational efficiency for the adaptive refinement algorithm incorporating this error estimator. Being compared with the exact solution, the adaptive mesh can be evaluated based on whether the refinement correctly takes place—that is in regions with higher discretization error. Moreover, the accuracy of the results computed on different estimator-controlled meshes is one way to evaluate the satisfactory of a posteriori error estimators. From the results obtained by [26] for steady-state solutions of convectiondiffusion equations, it is shown that the residual-based error estimator in the L2 – norm (ηres−L2 ) is reliable if the solution possess regular boundary layers. In addition, the ηres−L2 norm works well in conjunction with adaptive mesh refinement. The gradient indicator (gradind) ηgradind = ∇uh L2 ,K , the L2 norm of the gradient of uh on the element K, is widely used in software packages for adaptive mesh refinement because of its simplicity. However, with the gradient indicator it is not possible to obtain an estimate of the global error, and this approach is unsatisfactory when used in conjunction with adaptive mesh refinement for this class of problems.
3
A Parallel Particle Tracking Algorithm
Cheng et al. [2] developed the in-element particle tracking technique to accurately and efficiently trace fictitious particles in the velocity fields of real-world systems. Since in-element particle tracking methods are implemented with respect to the element basis, a natural approach is to develop a parallel framework based on specifying local, element-based operations. The key to the parallel algorithm is the correlated partitioning of the particle system and the unstructured element mesh [4]. We partition particles to processors based on their element location—this approach, as shown in Algorithm 3.1 ensures that the data required for the computation of the particle movement phase by the in-element method involves only data local to a processor. To balance the work load on each processor, which can change significantly at each time step because of the particle movement and adaptive mesh refinement, a load balanced parallel “in-element” particle tracking method has been developed [5].
4
A New A Posteriori Error Estimator
The use of traditional error estimators, such as the energy norm error estimator for the time-marching transport equations, could result in the selection of incorrect locations for adaptive refinement. This is because the discrete nature of the solution of the PDE causes discontinuities in time. Thus, the first iteration at each time step uses the mesh computed at the previous time step as the domain
An A Posteriori Error Estimator for Adaptive Mesh Refinement
99
Algorithm 3.1 The Parallel In-Element Particle Tracking Algorithm Let Pi be the set of particles on processor i Set the residual time tr to the time-step size n ← i Pi While (n > 0) do ni ← 0 Foreach (p ∈ Pi ) do While (p ∈ Pi and status(p) = finish) do Refine M to the prescribed number of subelements Track p subelement by subelement until time is exhausted or hit the boundary of the element M Compute tr , velocity, and identify the possible neighbor element M for continued tracking If tr > 0 do If owner(M ) = i do Pack p and remove p from Pi ni ← ni + 1 Endif Elseif tr = 0 do Interpolate concentration status(p) ← finish Endif Endwhile Endfor n = i ni Send and receive message, unpack messages to form new Pi Endwhile If backward tracking do Update concentration on each vertex Endif
discretization—unfortunately, this discretization may not be appropriate for the current time step. As might be expected, results obtained at the first iteration at the current time step using this mesh can be highly inaccurate if there has been significant change (for example, a change in a source/sink strength or boundary condition) between these two time steps. Based on this incorrect solution, we can repeatedly refine the mesh around the incorrectly computed moving front until it is within our error tolerance. Within this scenario, the traditional error estimator has failed to compute an accurate solution to the PDE system. To circumvent the failure of the traditional error estimator our aim is the development of a new a posteriori error estimator to address these problems. As shown in Algorithm 4.1, at time n, we use the particle tracking method to track all the vertices forward and backward in time. The value, Cjf , of the j-th forward particle is compared with the value, Cji , obtained from an interpolation derived from the mesh element vertices whose values originated from backward tracking
100
Jing-Ru C. Cheng and Paul E. Plassmann
and interpolation (we denote these values by C ∗ ). Once computed, we examine each element’s estimated error based on the first and second relative errors, which are obtained via a comparison with Cjf and with the maximum value f (CM ) throughout the entire domain, respectively. These two errors represent the local and global nature of the error. Only when both relative error criteria (3) | Cjf − Cji | /Cjf ≤ 1 f | Cjf − Cji | /CM ≤ 2
(3)
are violated, will the associated element containing this vertex be marked for refinement and be added to the set S n of elements to be refined. The elements in S n are refined based on the underlying AMR algorithm and generate the mesh T n which is used to solve the system using a traditional finite element method. We employ the following iterative procedure to find an optimal mesh for the current time step. The mesh is refined or coarsened based on whether the computed local error on an element is larger than the prescribed upper error tolerance or smaller than the lower error tolerance, respectively. Whenever the mesh is changed, we need to assemble the appropriate matrices and vectors and then solve a linear system based on this new mesh. After this new solution has been obtained, the local error estimate on each element according to the traditional error estimator, ηgradind,K = ∇uh L2 ,K ,
(4)
is computed. If the errors at any element are unacceptable, this set of elements, Sk , are refined to generate a new mesh Tk+1 . The values for the new vertices added to refine the elements in Sk are determined by interpolation. The problem is iteratively solved on the new discretized problem Tk+1 until neither refinement nor coarsening has been performed.
5
Experimental Results
In this section we present experimental results that demonstrate the improved accuracy of our proposed algorithm. We also discuss the computational overhead associated with this new method and demonstrate its parallel efficiency. Our experimental results are based on a representative two-dimensional transport problem. This problem is solved using the parallel particle tracking software incorporated within the SUMAA3d [11] programming environment. The parallel particle tracking incorporated within this software framework is used for the forward and backward tracking of each vertex. The particle tracking is required for the evaluation of the new local error estimates for the AMR method discussed in the previous section.
An A Posteriori Error Estimator for Adaptive Mesh Refinement
101
Algorithm 4.1 A parallel framework for the adaptive solution of PDEs Track each vertex backward and record the value C ∗ on each vertex Track each vertex (represented by a particle) forward and update the particle location Estimate the error on each element using the proposed new estimator Based on these error estimates, determine a set of elements, S n , to refine to form T n , including nonconforming elements k=0 while Tk is different than Tk−1 do Assemble matrix and vector on Tk Solve the associated matrix equation Estimate the error on each element using a traditional estimator Based on these error estimates, determine a set of elements, Sk , to be refined to form Tk+1 including nonconforming elements k =k+1 endwhile
5.1
Problem Description
The test problem we have used is one of benchmark problems from the Convection-Diffusion Forum [7]. This is an initial value problem with a small source transported by a shear flow with diffusion. In order to have an analytical solution, a two-dimensional uni-directional flow with velocity u0 in the x-direction with shear λ = du/dy is used. This advection inherent in this flow transports the initial source while diffusion, whose magnitude is determined by a coefficient D, disperses the source distribution. The governing equation for this problem is given by 2 ∂ C ∂C ∂2C ∂C + (u0 + λy) =D + (−∞ < x < ∞) . (5) ∂t ∂x ∂x2 ∂y 2 The initial and boundary conditions are specified as C(x, y, 0) = C0 (x, y) = M δ(x − x0 , y − y0 ) C(x, y, t) → 0 as |x| → ∞ or |y| → ∞ . The analytical solution of the system is C(x, y, t) =
M 1
4πDt(1 + λ2 t2 /12) 2
exp −
(x − x − 0.5λyt)2 y2 + 4πDt(1 + λ2 t2 /12) 4Dt
,
(6)
where x = x0 + u0 t .
(7)
Let the initial distribution be finite with the peak magnitude of 1 at location 1 (x, y0 ), i.e., M = 4πDt0 (1 + λ2 t2 /12) 2 . The simulation starts at time t = t0 .
102
Jing-Ru C. Cheng and Paul E. Plassmann
Table 1. Parameters of the problem Parameter Value x0 7,200 0 y0 2,400 t0 0.5 u0 D 10 λ 0.0005
The domain is defined to be the region (x ∈ [0, 24000], y ∈ [−3400, 3400]). The problem parameters used are specified in Table 1. This domain is initially discretized into 7,992 vertices and 15,957 triangular elements as shown in Fig. 1. The initial conditions are specified by (6). The traditional error estimator used in Algorithm 4.1 is the gradient indicator ηgradind , which is the L2 -norm of the gradient of uh as shown in (4) for element K. Results are shown at times 4800, 7200, and 9600, with the implicit time scheme using a time-step of size 50. The time-step size is adjusted to 50 because the mesh is generated based on the simple ηgradind error estimator—and the problem cannot be solved on this mesh by the traditional finite element method without this shorter (than the benchmark specified of 96) time step size. This restriction is crucial for the comparison of the accuracies of the two methods. The error tolerances used for the new error estimator given in (3) are 1 = 0.01 and 2 = 0.0001. This problem was run in parallel on an IBM SP2 at Argonne National Laboratory. 5.2
Computational Results
Fig. 2 and Fig. 3 illustrate the difference in the accuracy of these two methods. We show the computed solution using the finite element method for solving this time-marching transport problem on the mesh generated by the same AMR technique with and without incorporation of the proposed new error estimator. These results clearly demonstrate that the capture of the moving front and the profile similarity are improved with the use of our refinement criteria when compared to the analytical solution. In Fig. 3 we demonstrate this improvement by comparing the pointwise absolute error for these two approaches. From Fig. 4, we observe a significant speedup on the total running time and the time spent in particle tracking for the implementation of the presented a posteriori error estimator. Although the computation of the estimator does impose a non-negligible overhead, the tradeoff in using this method is that we obtain significantly more accurate results. In addition, the performance of this algorithm can also be shown to be scalable. This relative overhead drops dramatically when more processors are used. Fig. 4 also shows that a finer mesh is generated to model the critical region when using one iteration of the new error estimator. This approach causes more inner iterations for the AMR method in solving the transient problem using the same number of time steps. The plot for the efficiency index, on the
An A Posteriori Error Estimator for Adaptive Mesh Refinement
103
Fig. 1. The initial distribution in the source area (top) and the discretization of the domain (bottom)
right of Fig. 4, shows that the proposed error estimator is efficient based on the characterization given in Sect. 2.
6
Summary and Future Plans
In this paper, we have presented a new a posteriori error estimator based on a particle tracking method. With the incorporation of this error estimator, we have demonstrated that the accuracy of the computed solution of a representative convection-diffusion benchmark problem can be significantly improved. Although the overhead that the required particle tracking incurs cannot be neglected, this improvement in accuracy could not be achieved via the traditional a posteriori error estimator developed for steady state convection-diffusion equations. The intuitive reason for improvement is that the particle tracking method can accurately trace particle trajectories; thus, the new error estimator can correctly identify and refine the mesh at the critical region. Finally, our parallel implementation of the particle tracking method shows a significant speedup and computational efficiency. However, a more detailed study of the parallel performance and accuracy of the proposed error estimator under different flow fields and applications from other application areas is required. In particular, speedup and scalability studies on larger numbers of processors would be of great interest.
104
Jing-Ru C. Cheng and Paul E. Plassmann
Fig. 2. (a) At time 4800, (b) At time 7200, and (c) At time 9600, Left: the analytical solution (top) and the simulation result (bottom) with the incorporation of the new error estimator. Right: the analytical solution (top) and the simulation result (bottom) without the incorporation of the new error estimator
An A Posteriori Error Estimator for Adaptive Mesh Refinement
105
Fig. 3. The pointwise absolute error | u − uh | of the numerical results at time 4800 (left) and at time 9600 (right) using the numerical method with (top) and without (bottom) the incorporation of the new error estimator
Performance speedup
4
10
CPU time (sec)
total time particle tracking time
3
10
2
10
Efficiency index
2
10 0 10
1
2
10
10
number of processors
1
10
Problem size at each iteration 16000
Efficiency index
with the new estimator without the new estimator 14000
number of vertices
12000
10000
0
10
8000 −1
10 6000
4000 −2
2000
10 0
100
200
300
400
iteration no.
500
600
700
4
10
no. of vertices
Fig. 4. Left: The performance speedup (top) and total number of vertices generated by AMR with and without the incorporation of the new error estimator (bottom), Right: Efficiency index with respect to different discretized mesh
106
Jing-Ru C. Cheng and Paul E. Plassmann
References [1] J. Barnes and P. Hut. A hierarchical O(N log N ) force calculation algorithm. Nature, 324:446, Dec. 1986. 96 [2] H.-P. Cheng, J.-R. Cheng, and G. T. Yeh. A particle tracking technique for the Lagrangian-Eulerian finite-element method in multi-dimensions. International Journal for Numerical Methods in Engineering, 39:1115–1136, 1996. 97, 98 [3] J.-R. Cheng, H.-P. Cheng, and G. T. Yeh. A Lagrangian-Eulerian method with adaptively local zooming and peak/valley capturing approach to solve twodimensional advection-diffusion transport equations. International Journal for Numerical Methods in Engineering, 39:987–1016, 1996. 94 [4] J-R. C. Cheng and P. E. Plassmann. The accuracy and performance of parallel inelement particle tracking methods. In Proceedings of the Tenth SIAM Conference on Parallel Processing for Scientific Computing, pages 252–261. SIAM, 2001. 98 [5] J-R. C. Cheng and P. E. Plassmann. A parallel algorithm for the dynamic partitioning of particle-mesh computational systems. In Computational Science—ICCS 2002, The Springer Verlag Lecture Notes in Computer Science (LNCS 2331) series, Part III, pages 1020–1029. Springer Verlag, 2002. 97, 98 [6] L. P. Chew. Guaranteed-quality triangular meshes. Technical Report TR-89–983, Computer Science Department, Cornell University, 1989. 95 [7] Convection-Diffusion Forum. Specification of five convection-diffusion benchmark problems. 7th International Conference on Computational Methods in Water Resources, Massachusetts Institute of Technology, June 1988. 101 [8] L. Demkowicz, J. T. Oden, W. Rachowicz, and O. Hardy. Toward a universal h-p adaptive finite element strategy, part 1. Constrained approximation and data structures. Computer Methods in Applied Mechanics and Engineering, 77:79–112, 1989. 95 [9] D. Diachin and J. Herzog. Analytic streamline calculations for linear tetrahedra. In 13th AIAA Computational Fluid Dynamics Conference, pages 733–742, 1997. 96 [10] Joseph E. Flaherty, Pamela J. Paslow, Mark S. Shephard, and John D. Vasilakis. Adaptive Methods for Partial Differential Equations. Society for Industrial and Applied Mathematics, Philadelphia, 1989. 95 [11] Lori Freitag, Mark Jones, Carl Ollivier-Gooch, and Paul Plassmann. SUMAA3d Web page. http://www.mcs.anl.gov/sumaa3d/, Mathematics and Computer Science Division, Argonne National Laboratory, 1997. 100 [12] A. Harten. Eno schemes with subcell resolution. Journal of Computational Physics, 83:148–184, 1989. 94 [13] Mark T. Jones and Paul E. Plassmann. Adaptive refinement of unstructured finite-element meshes. Finite Elements in Analysis and Design, 25(1-2):41–60, March 1997. 95 [14] David Kay and David Silvester. The reliability of local error estimators for convection-diffusion equations. IMA Journal of Numerical Analysis, 21:107–122, 2001. 95 [15] A. V. Malevsky and S. J. Thomas. Parallel algorithms for semi-Lagrangian advection. International Journal for Numerical Methods in Fluids, 25:455–473, 1997. 96 [16] David K. McAllister. The design of an API for particle systems. Technical Report UNC-CH TR00-007, Department of Computer Science, University of North Carolina, 1999. 95
An A Posteriori Error Estimator for Adaptive Mesh Refinement
107
[17] Bongki Moon, Mustafa Uysal, and Joel Saltz. Index translation schemes for adaptive computations on distributed memory multicomputers. In Proceedings of the 9-th International Parallel Processing Symposium, pages 812–819, Santa Barbara, California, April 24-28, 1995. IEEE Computer Society Press. 96 [18] K. W. Morton. Numerical Solution of Convection-Diffusion Problems. Klumer Academic Publishers, 1995. 95 [19] Robert P. Nance, Richard G. Wilmoth, Bongki Moon, H. A. Hassan, and Joel Saltz. Parallel DSMC solution of three-dimensional flow over a finite flat plate. In Proceedings of the ASME 6-th Joint Thermophysics and Heat Transfer Conference, Colorado Springs, Colorado, June 20-23, 1994. AIAA. 96 [20] J. T. Oden, L. Demkowicz, W. Rachowicz, and T. A. Westermann. Toward a universal h-p adaptive finite element strategy, part 2. A posterior error estimation. Computer Methods in Applied Mechanics and Engineering, 77:113–180, 1989. 95 [21] W. Rachowicz, J. T. Oden, and L. Demkowicz. Toward a universal h-p adaptive finite element strategy, part 3. Design of h-p meshes. Computer Methods in Applied Mechanics and Engineering, 77:181–212, 1989. 95 [22] C. D. Robinsion and J. K. Harvey. A fully concurrent DSMC implementation with adaptive domain decomposition. In Proceedings of the Parallel CFD Meeting, 1997, 1997. 96 [23] H.-G. Roos, M. Stynes, and L. Tobiska. Numerical Methods for Singularity Perturbed Differential Equations. Springer, 1996. 95 [24] J. Ruppert. A new and simple algorithm for quality 2-dimensional mesh generation. In Proc. 4th ACM-SIAM Symp. on Disc. Algorithms, pages 83–92, 1993. 95 [25] R¨ udiger Verf¨ urth. A Review of A Posteriori Error Estimation and Adaptive MeshRefinement Techniques. John Wiley & Sons Ltd and B. G. Teubner, 1996. 95, 97, 98 [26] John Volker. A numerical study of a posteriori error estimators for convection– diffusion equations. Computer Methods in Applied Mechanics and Engineering, 190:757–781, 2000. 97, 98 [27] N. P. Weatherill, O. Hassan, D. L. Marcum, and M. J. Marchant. Grid generation by the Delaunay triangulation. Von Karman Institute for Fluid Dynamics, 19931994 Lecture Series, NASA Ames Research Center, January 1994. 95 [28] B. C. Wong and L. N. Long. A data-parallel implementation of the DSMC method on the connection machine. Computing Systems in Engineering Journal, 3(1-4), Dec. 1992. 96 [29] H. C. Yee, R. F. Warming, and A. Harten. Implicit total variation diminishing (TVD) schemes for steady-state calculations. Journal of Computational Physics, 57:327–360, 1985. 94 [30] G. T. Yeh, J.-R. Cheng, H.-P. Cheng, and C. H. Sung. An adaptive local grid refinement based on the exact peak capturing and oscillation free scheme to solve transport equations. International Journal of Computational Fluids Dynamics, 24:293–332, 1995. 94 [31] G. T. Yeh, J.-R. Cheng, and T. E. Short. An exact peak capturing and essentially oscillation-free scheme to so lve advection-dispersion-reactive transport equations. Water Resources Research, 28(11):2937–2951, 1992. 94
High Performance Data Mining Vipin Kumar, Mahesh V. Joshi, Eui-Hong (Sam) Han, Pang-Ning Tan, and Michael Steinbach University of Minnesota 4-192 EE/CSci Building, 200 Union Street SE Minneapolis, MN 55455, USA {kumar,mjoshi,han,ptan,steinbac}@cs.umn.edu
Abstract. Recent times have seen an explosive growth in the availability of various kinds of data. It has resulted in an unprecedented opportunity to develop automated data-driven techniques of extracting useful knowledge. Data mining, an important step in this process of knowledge discovery, consists of methods that discover interesting, non-trivial, and useful patterns hidden in the data [SAD+93, CHY96]. The field of data mining builds upon the ideas from diverse fields such as machine learning, pattern recognition, statistics, database systems, and data visualization. But, techniques developed in these traditional disciplines are often unsuitable due to some unique characteristics of today’s data-sets, such as their enormous sizes, high-dimensionality, and heterogeneity. There is a necessity to develop effective parallel algorithms for various data mining techniques. However, designing such algorithms is challenging, and the main focus of the paper is a description of the parallel formulations of two important data mining algorithms: discovery of association rules, and induction of decision trees for classification. We also briefly discuss an application of data mining to the analysis of large data sets collected by Earth observing satellites that need to be processed to better understand global scale changes in biosphere processes and patterns.
1
Introduction
Recent times have seen an explosive growth in the availability of various kinds of data. It has resulted in an unprecedented opportunity to develop automated data-driven techniques of extracting useful knowledge. Data mining, an important step in this process of knowledge discovery, consists of methods that discover interesting, non-trivial, and useful patterns hidden in the data [26, 6]. The field of data mining builds upon the ideas from diverse fields such as machine learning, pattern recognition, statistics, database systems, and data visualization. But,
This work was supported by NSF CCR-9972519, by NASA grant # NCC 2 1231, by Army Research Office contract DA/DAAG55-98-1-0441, by the DOE grant LLNL/DOE B347714, and by Army High Performance Computing Research Center cooperative agreement number DAAD19-01-2-0014. Access to computing facilities was provided by AHPCRC and the Minnesota Supercomputer Institute. Related papers are available via WWW at URL: http://www.cs.umn.edu/˜kumar.
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 111–125, 2003. c Springer-Verlag Berlin Heidelberg 2003
112
Vipin Kumar et al.
techniques developed in these traditional disciplines are often unsuitable due to some unique characteristics of today’s data-sets, such as their enormous sizes, high-dimensionality, and heterogeneity. To date, the primary driving force behind the research in data mining has been the development of algorithms for data-sets arising in various business, information retrieval, and financial applications. Businesses can use data mining to gain significant advantages in today’s competitive global marketplace. For example, retail industry is using data mining techniques to analyze buying patterns of customers, mail order business is using them for targeted marketing, telecommunication industry is using them for churn prediction and network alarm analysis, and credit card industry is using them for fraud detection. Also, recent growth of electronic commerce is generating wealths of online web data, which needs sophisticated data mining techniques. Due to the latest technological advances, very large data-sets are becoming available in many scientific disciplines as well. The rate of production of such data-sets far outstrips the ability to analyze them manually. For example, a computational simulation running on the state-of-the-art high performance computers can generate tera-bytes of data within a few hours, whereas human analyst may take several weeks or longer to analyze and discover useful information from these data-sets. Data mining techniques hold great promises for developing new sets of tools that can be used to automatically analyze the massive data-sets resulting from such simulations, and thus help engineers and scientists unravel the causal relationships in the underlying mechanisms of the dynamic physical processes. The huge size of the available data-sets and their high-dimensionality make large-scale data mining applications computationally very demanding, to an extent that high-performance parallel computing is fast becoming an essential component of the solution. Moreover, the quality of the data mining results often depends directly on the amount of computing resources available. In fact, data mining applications are poised to become the dominant consumers of supercomputing in the near future. There is a necessity to develop effective parallel algorithms for various data mining techniques. However, designing such algorithms is challenging. In the rest of this chapter, we present an overview of the parallel formulations of two important data mining algorithms: discovery of association rules, and induction of decision trees for classification. We also briefly discuss an application of data mining [13, 14, 10] to the analysis of large data sets collected by Earth observing satellites that need to be processed to better understand global scale changes in biosphere processes and patterns.
2
Parallel Algorithms for Discovering Associations
An important problem in data mining [6] is discovery of associations present in the data. Such problems arise in the data collected from scientific experiments, or monitoring of physical systems such as telecommunications networks, or from transactions at a supermarket. The problem was formulated originally in the
High Performance Data Mining
113
Table 1. Transactions from supermarket TID 1 2 3 4 5
Items Bread, Coke, Milk Beer, Bread Beer, Coke, Diaper, Milk Beer, Bread, Diaper, Milk Coke, Diaper, Milk
context of the transaction data at supermarket. This market basket data, as it is popularly known, consists of transactions made by each customer. Each transaction contains items bought by the customer (see Table 1). The goal is to see if occurrence of certain items in a transaction can be used to deduce occurrence of other items, or in other words, to find associative relationships between items. If indeed such interesting relationships are found, then they can be put to various profitable uses such as shelf management, inventory management, etc. Thus, association rules were born [2]. Simply put, given a set of items, association rules predict the occurrence of some other set of items with certain degree of confidence. The goal is to discover all such interesting rules. This problem is far from trivial because of the exponential number of ways in which items can be grouped together and different ways in which one can define interestingness of a rule. Hence, much research effort has been put into formulating efficient solutions to the problem. Let T be the set of transactions where each transaction is a subset of the itemset I. Let C be a subset of I, then we define the support count of C with respect to T to be: σ(C) = |{t|t ∈ T, C ⊆ t}|. Thus σ(C) is the number of transactions that contain C. An association rule is s,α an expression of the form X =⇒ Y , where X ⊆ I and Y ⊆ I. The support s of s,α the rule X =⇒ Y is defined as σ(X ∪ Y )/|T |, and the confidence α is defined as σ(X ∪ Y )/σ(X). For example, for transactions in Table 1, the support of rule {Diaper, Milk} =⇒ {Beer} is σ(Diaper, M ilk, Beer)/5 = 2/5 = 40%, whereas its confidence is. σ(Diaper, M ilk, Beer)/σ(Diaper, M ilk) = 2/3 = 66%. s,α The task of discovering an association rule is to find all rules X =⇒ Y , such that s is greater than or equal to a given minimum support threshold and α is greater than or equal to a given minimum confidence threshold. The association rule discovery is usually done in two phases. First phase finds all the frequent itemsets; i.e., sets satisfying the support threshold, and then they are post-processed in the second phase to find the high confidence rules. The former phase is computationally most expensive, and much research has been done in developing efficient algorithms for it. A comparative survey of all the existing techniques is given in [15]. A key feature of these algorithms lies in their method of controlling the exponential complexity of the total number of itemsets (2|I| ). Briefly, they all use the anti-monotone property of an itemset support, which
114
Vipin Kumar et al.
states that an itemset is frequent only if all of its sub-itemsets are frequent. Apriori algorithm [4] pioneered the use of this property to systematically search the exponential space of itemsets. In an iteration k, it generates all the candidate k-itemsets (of length k) such that all their (k − 1)-subsets are frequent. The number of occurrences of these candidates are then counted in the transaction database, to determine frequent k-itemsets. Efficient data structures are used to perform fast counting. Overall, the serial algorithms such as Apriori have been successful on a wide variety of transaction databases. However, many practical applications of association rules involve huge transaction databases which contain a large number of distinct items. In such situations, these algorithms running on single-processor machines may take unacceptably large times. As an example, in the Apriori algorithm, if the number of candidate itemsets becomes too large, then they might not all fit in the main memory, and multiple database passes would be required within each iteration, incurring expensive I/O cost. This implies that, even with the highly effective pruning method of Apriori, the task of finding all association rules can require a lot of computational and memory resources. This is true of most of the other serial algorithms as well, and it motivates the development of parallel formulations. Various parallel formulations have been developed so far. A comprehensive survey can be found in [15, 28]. These formulations are designed to effectively parallelize either or both of the computation phases: candidate generation and candidate counting. The candidate counting phase can be parallelized relatively easily by distributing the transaction database, and gathering local counts for the entire set of candidates stored on all the processors. The CD algorithm [3] is an example of this simple approach. It scales linearly with respect to the number of transactions; however, generation and storage of huge number of candidates on all the processors becomes a bottleneck, especially when highdimensional problems are solved for low support thresholds using large number of processors. Other parallel formulations, such as IDD [11], have been developed to solve these problems. Their key feature is to distribute the candidate itemsets to processors so as to extract the concurrency in candidate generation as well as counting phases. Various ways are employed in IDD to reduce the communication overhead, to exploit the total available memory, and to achieve reasonable load balance. IDD algorithm exhibits better scalability with respect to the number of candidates. Moreover, reduction of redundant work and ability to overlap counting computation with communication of transactions, improves its scalability with respect to number of transactions. However, it still faces problems when one desires to use large number of processors to solve the problem. As more processors are used, the number of candidates assigned to each processor decreases. This has two implications on IDD. First, with fewer number of candidates per processor, it is much more difficult to achieve load balance. Second, it results in less computation work per transaction at each processor. This reduces the overall efficiency. Further lack of asynchronous communication ability may worsen the situation.
High Performance Data Mining
115
Formulations that combine the approaches of replicating and distributing candidates so as to reduce the problems of each one, have been developed. An example is the HD algorithm of [11]. Briefly, it works as follows. Consider a P processor system in which the processors are split into G equal size groups, each containing P/G processors. In the HD algorithm, we execute the CD algorithm as if there were only P/G processors. That is, we partition the transactions of the database into P/G parts each of size N/(P/G), and assign the task of computing the counts of the candidate set Ck for each subset of the transactions to each one of these groups of processors. Within each group, these counts are computed using the IDD algorithm. The HD algorithm inherits all the good features of the IDD algorithm. It also provides good load balance and enough computation work by maintaining minimum number of candidates per processor. At the same time, the amount of data movement in this algorithm is cut down to 1/G of that of IDD. A detailed parallel runtime analysis of HD is given in [12]. It shows that HD is scalable with respect to both number of transactions and number of candidates. The analysis also proves the necessary conditions under which HD can outperform CD. Sequential Associations The concept of association rules can be generalized and made more useful by observing another fact about transactions. All transactions have a timestamp associated with them; i.e. the time at which the transaction occurred. If this information can be put to use, one can find relationships such as if a customer bought [The C Programming Language] book today, then he/she is likely to buy a [Using Perl] book in a few days time. The usefulness of this kind of rules gave birth to the problem of discovering sequential patterns or sequential associations. In general, a sequential pattern is a sequence of item-sets with various timing constraints imposed on the occurrences of items appearing in the pattern. For example, (A) (C,B) (D) encodes a relationship that event D occurs after an event-set (C,B), which in turn occurs after event A. Prediction of events or identification of sequential rules that characterize different parts of the data, are some example applications of sequential patterns. Such patterns are not only important because they represent more powerful and predictive relationships, but they are also important from the algorithmic point of view. Bringing in the sequential relationships increases the combinatorial complexity of the problem enormously. The reason is that, the maximum number of sequences having k events is O(mk 2k−1 ), where m is the total number of distinct events in the input data. In contrast, there are only (m k ) size-k item-sets possible while discovering non-sequential associations from m distinct items. Designing parallel algorithms for discovering sequential associations is equally important and challenging. In many situations, the techniques used in parallel algorithms for discovering standard non-sequential associations can be extended easily. However, different issues and challenges arise specifically due to the sequential nature and various ways in which interesting sequential associations can be defined. Details of various serial and parallel formulations and algorithms for finding such associations can be found in [17, 15].
116
3
Vipin Kumar et al.
Parallel Algorithms for Induction of Decision Tree Classifiers
Classification is an important data mining problem. The input to the problem is a data-set called the training set, which consists of a number of examples each having a number of attributes. The attributes are either continuous, when the attribute values are ordered, or categorical, when the attribute values are unordered. One of the categorical attributes is called the class label or the classifying attribute. The objective is to use the training set to build a model of the class label based on the other attributes such that the model can be used to classify new data not from the training data-set. Application domains include retail target marketing, fraud detection, and design of telecommunication service plans. Several classification models like neural networks [19], genetic algorithms [9], and decision trees [22] have been proposed. Decision trees are probably the most popular since they obtain reasonable accuracy [7] and they are relatively inexpensive to compute. Most of the existing induction–based algorithms like C4.5 [22], CDP [1], SLIQ [20], and SPRINT [23] use Hunt’s method [22] as the basic algorithm. Here is its recursive description for constructing a decision tree from a set T of training cases with classes denoted {C1 , C2 , . . . , Ck }. Case 1 T contains cases all belonging to a single class Cj . The decision tree for T is a leaf identifying class Cj . Case 2 T contains cases that belong to a mixture of classes. A test is chosen, based on a single attribute, that has one or more mutually exclusive outcomes {O1 , O2 , . . . , On }. Note that in many implementations, n is chosen to be 2 and this leads to a binary decision tree. T is partitioned into subsets T1 , T2 , . . . , Tn , where Ti contains all the cases in T that have outcome Oi of the chosen test. The decision tree for T consists of a decision node identifying the test, and one branch for each possible outcome. The same tree building machinery is applied recursively to each subset of training cases. Case 3 T contains no cases. The decision tree for T is a leaf, but the class to be associated with the leaf must be determined from information other than T . For example, C4.5 chooses this to be the most frequent class at the parent of this node. Figure 1 shows a training data set with four data attributes and two classes and its classification decision tree constructed using the Hunt’s method. In the case 2 of Hunt’s method, a test based on a single attribute is chosen for expanding the current node. The choice of an attribute is normally based on the entropy gains [22] of the attributes. The entropy of an attribute, calculated from class distribution information, depicts the classification power of the attribute by itself. The best attribute is selected as a test for the node expansion. Highly parallel algorithms for constructing classification decision trees are desirable for dealing with large data sets in reasonable amount of time. Classification decision tree construction algorithms have natural concurrency, as once a node is generated, all of its children in the classification tree can be generated
High Performance Data Mining Outlook sunny sunny sunny sunny sunny overcast overcast overcast overcast rain rain rain rain rain
Temp(F) 75 80 85 72 69 72 83 64 81 71 65 75 68 70
Humidity(%) 70 90 85 95 70 90 78 65 75 80 70 80 80 96
Windy? true true false false false true false true false true true false false false
117
Class Play Don’t Play Don’t Play Don’t Play Play Play Play Play Play Don’t Play Don’t Play Play Play Play
Outlook sunny
overcast
Humidity <= 75
Play
> 75
Dont’ Play
rain
Windy
Play true
false
Dont’ Play
Play
Fig. 1. A small training data set [Qui93] and its final classification decision tree
concurrently. Furthermore, the computation for generating successors of a classification tree node can also be decomposed by performing data decomposition on the training data. Nevertheless, parallelization of the algorithms for construction the classification tree is challenging for the following reasons. First, the shape of the tree is highly irregular and is determined only at runtime. Furthermore, the amount of work associated with each node also varies, and is data dependent. Hence any static allocation scheme is likely to suffer from major load imbalance. Second, even though the successors of a node can be processed concurrently, they all use the training data associated with the parent node. If this data is dynamically partitioned and allocated to different processors that perform computation for different nodes, then there is a high cost for data movements. If the data is not partitioned appropriately, then performance can be bad due to the loss of locality. Several parallel formulations of classification decision tree have been proposed recently [21, 8, 23, 5, 18, 16, 24]. In this section, we present two basic parallel formulations for the classification decision tree construction and a hybrid scheme that combines good features of both of these approaches described in [24]. Most of other parallel algorithms are similar in nature to these two basic algorithms, and their characteristics can be explained using these two basic algorithms. For
118
Vipin Kumar et al. Data Item
Proc 0
Proc 1
Proc 2
Proc 3
Proc 0
Proc 1
Proc 2
Proc 3
Class Distribution Information
Data Item
Proc 0
11 00
Proc 1
11 00
Proc 2
1 0
Proc 3 Proc 0
Proc 1
Proc 2
Proc 3
Proc 0
Proc 1
Proc 2
Proc 3
1 0
Class Distribution Information
(a) Synchronous Tree Construction
(b) Partitioned Tree Construction
Fig. 2. Synchronous Tree Construction Approach and Partitioned Tree Construction Approach these parallel formulations, we focus our presentation for discrete attributes only. The handling of continuous attributes is discussed separately. In all parallel formulations, we assume that N training cases are randomly distributed to P processors initially such that each processor has N/P cases. Synchronous Tree Construction Approach In this approach, all processors construct a decision tree synchronously by sending and receiving class distribution information of local data. Figure 2 (a) shows the overall picture. The root node has already been expanded and the current node is the leftmost child of the root (as shown in the top part of the figure). All the four processors cooperate to expand this node to have two child nodes. Next, the leftmost node of these child nodes is selected as the current node (in the bottom of the figure) and all four processors again cooperate to expand the node. Partitioned Tree Construction Approach In this approach, whenever feasible, different processors work on different parts of the classification tree. In particular, if more than one processors cooperate to expand a node, then these processors are partitioned to expand the successors of this node. Figure 2 (b) shows an example. First (at the top of the figure), all four processors cooperate to expand the root node just like they do in the synchronous tree construction approach. Next (in the middle of the figure), the set of four processors is partitioned in three parts. The leftmost child is assigned to processors 0 and 1, while the other nodes are assigned to processors 2 and 3, respectively. Now these sets of processors proceed independently to expand these assigned nodes. In particular, processors 2 and processor 3 proceed to expand their part of the tree using the
High Performance Data Mining
119
Computation Frontier at depth 3 Partition 1
Synchronous Tree Construction Approach
Partition 2
Partitioned Tree Construction Approach
Fig. 3. Hybrid Tree Construction Approach
serial algorithm. The group containing processors 0 and 1 splits the leftmost child node into three nodes. These three new nodes are partitioned in two parts (shown in the bottom of the figure); the leftmost node is assigned to processor 0, while the other two are assigned to processor 1. From now on, processors 0 and 1 also independently work on their respective subtrees. Hybrid Parallel Formulation The hybrid parallel formulation has elements of both schemes. The Synchronous Tree Construction Approach incurs high communication overhead as the frontier gets larger. The Partitioned Tree Construction Approach incurs cost of load balancing after each step. The hybrid scheme keeps continuing with the first approach as long as the communication cost incurred by the first formulation is not too high. Once this cost becomes high, the processors as well as the current frontier of the classification tree are partitioned into two parts. Figure 3 shows one example of this parallel formulation. At the classification tree frontier at depth 3, no partitioning has been done and all processors are working cooperatively on each node of the frontier. At the next frontier at depth 4, partitioning is triggered, and the nodes and processors are partitioned into two partitions. A key element of the algorithm is the criterion that triggers the partitioning of the current set of processors (and the corresponding frontier of the classification tree ). If partitioning is done too frequently, then the hybrid scheme will approximate the partitioned tree construction approach, and thus will incur too much data movement cost. If the partitioning is done too late, then it will suffer from high cost for communicating statistics generated for each node of the frontier, like the synchronized tree construction approach. In the hybrid algorithm, the splitting is performed when the accumulated cost of communication becomes equal to the cost of moving records and load balancing in the splitting phase. The size and shape of the classification tree varies a lot depending on the application domain and training data set. Some classification trees might be shallow and the others might be deep. Some classification trees could be skinny others could be bushy. Some classification trees might be uniform in depth while other trees might be skewed in one part of the tree. The hybrid approach adapts well to all types of classification trees. If the decision tree is skinny, the hybrid
120
Vipin Kumar et al.
approach will just stay with the Synchronous Tree Construction Approach. On the other hand, it will shift to the Partitioned Tree Construction Approach as soon as the tree becomes bushy. If the tree has a big variance in depth, the hybrid approach will perform dynamic load balancing with processor groups to reduce processor idling. Handling Continuous Attributes The approaches described above concentrated primarily on how the tree is constructed in parallel with respect to the issues of load balancing and reducing communication overhead. The discussion was simplified by the assumption of absence of continuous-valued attributes. Presence of continuous attributes can be handled in two ways. One is to perform intelligent discretization, either once in the beginning or at each node as the tree is being induced, and treat them as categorical attributes. Another, more popular approach is to use decisions of the form A < x and A ≥ x, directly on the values x of continuous attribute A. The decision value of x needs to be determined at each node. For efficient search of x, most algorithms require the attributes to be sorted on values, such that one linear scan can be done over all the values to evaluate the best decision. Among various different algorithms, the approach taken by SPRINT algorithm[23], which sorts each continuous attribute only once in the beginning, is proven to be efficient for large datasets. The sorted order is maintained throughout the induction process, thus avoiding the possibly excessive costs of re-sorting at each node. A separate list is kept for each of the attributes, in which the record identifier is associated with each sorted value. The key step in handling continuous attributes is the proper assignment of records to the children node after a splitting decision is made. Implementation of this offers the design challenge. SPRINT builds a mapping between a record identifier and the node to which it goes to based on the splitting decision. The mapping is implemented as a hash table and is probed to split the attribute lists in a consistent manner. Parallel formulation of the SPRINT algorithm falls under the category of synchronous tree construction design. The multiple sorted lists of continuous attributes are split in parallel by building the entire hash table on all the processors. However, with this simple-minded way of achieving a consistent split, the algorithm incurs a communication overhead of O(N ) per processor. Since, the serial runtime of the induction process is O(N ), SPRINT becomes unscalable with respect to runtime. It is unscalable in memory requirements also, because the total memory requirement per processor is O(N ), as the size of the hash table is of the same order as the size of the training dataset for the upper levels of the decision tree, and it resides on every processor. Another parallel algorithm, ScalParC [16], solves this scalability problem. It employs a distributed hash table to achieve a consistent split. The communication structure, used to construct and access this hash table, is motivated by the parallel sparse matrix-vector multiplication algorithms. It is shown in [16] that with the proper implementation of the parallel hashing, the overall communication overhead does not exceed O(N ), and the memory required does not exceed O(N/p) per processor. Thus, ScalParC is scalable in runtime as well as memory requirements.
High Performance Data Mining
4
121
Example Application: Data Mining for Earth Science Data
Data mining techniques have recently been used to find interesting spatio-temporal patterns from Earth Science data. This data consists of time series measurements for various Earth science and climate variables (e.g. soil moisture, temperature, and precipitation), along with additional data from existing ecosystem models (e.g., Net Primary Production). See figures 4 and 5. The ecological patterns of interest include associations, clusters, predictive models, and trends. To find association patterns we transformed these time series into transactions and then applied existing algorithms traditionally used for market-basket data. We found that association rules can uncover interesting patterns for Earth Scientists to investigate. For example, we found that high temperature was well correlated with high plant growth in the forest and cropland regions in the northern hemisphere. However, significant challenges for association analysis arise due to the spatio-temporal nature of the data and the need to incorporate domain knowledge to prune out uninteresting patterns. For further detail on this work, see [27]. To predict the effect of the oceans on land climate, Earth Scientists have developed ocean climate indices (OCIs), which are time series that summarize the behavior of selected areas of the Earth’s oceans. For example, the Southern Oscillation Index (SOI) is an OCI that is associated with El Nino. In the past, Earth scientists have used observation and, more recently, eigenvalue analysis
Fig. 4. Land and sea temperature
122
Vipin Kumar et al. NPP
.
Pressure Precipitation
NPP
.
Pressure
.
Precipitation
SST
SST
Latitude
grid cell
Longitude
Time
zone
Fig. 5. Illustration of Earth Science Data
techniques, such as principal components analysis (PCA) and singular value decomposition (SVD), to discover ocean climate indices. However, these techniques are only useful for finding a few of the strongest signals and, furthermore, impose a condition that all discovered signals must be orthogonal to each other. We have developed an alternative methodology for the discovery of OCIs that overcomes these limitations and is based on clusters that represent ocean regions with relatively homogeneous behavior [25]. The centroids of these clusters are time series that summarize the behavior of these ocean areas. We divide the cluster centroids into several categories: those that correspond to known OCIs, those that are variants of known OCIs, and those that represent potentially new OCIs. The centroids that correspond to known OCIs provide a validation of our methodology, while some variants of known OCIs may provide better predictive power for some land areas. Also, we have shown that, in some sense, our current cluster centroids are relatively complete, i.e., capture most of the possible candidate OCIs. For further details, the reader is referred to [25]. A number of aspects of Earth Science data and the previously described analyses require the use of high-performance computing. First, satellites are providing measurements of finer granularity. For instance, a 1◦ by 1◦ grid produces 64,800 data points, while a 0.1◦ by 0.1◦ grid produces 6,480,000 data points. Second, more frequent measurements, e.g., daily measurements, multiply monthly data by a factor of 30. Also, looking at weather instead of climate requires finer resolution to enable the detection of fast changing patterns, e.g., the movement of fronts Our current clustering analysis, while effective, requires O(n2 ) comparisons since it needs to evaluate the correlation of every ocean point with every land point. Furthermore, association rule algorithms can also be very compute intensive. Indeed, the computational complexity of these algorithms is potentially very much greater than O(n2 ). Finally, the amount of memory required for clustering and association rule algorithms can exceed the 4GB provided by traditional sequential servers.
High Performance Data Mining
5
123
Conclusion
This paper presented an overview of parallel algorithms for two of the commonly used data mining techniques: classification and associations. Key issues such as load balancing, attention to locality, extracting maximal concurrency, avoiding hot spots in contention, and minimizing parallelization overhead are just as central to these parallel formulations as they are to the traditional scientific parallel algorithms. In fact, in many cases, the underlying kernels are identical to well known algorithms, such as sparse matrix-vector product. To date, the parallel formulations of many decision-tree induction and association rule discovery algorithms are reasonably well-understood. Relatively less work has been done on the parallel algorithms for other data mining techniques such as clustering, rule-based classification algorithms, deviation detection, and regression. Some possible areas of further research include parallelization of many emerging new and improved serial data mining algorithms, further analysis and refinements of existing algorithms for scalability and efficiency, designs targeted for shared memory and distributed shared memory machines equipped with symmetric multiprocessors, and efficient integration of parallel algorithms with parallel database systems. High-performance data mining algorithms and tools are needed for mining large-scale data sets that arise in a variety of applications. This paper presented a possible application, i.e., large data sets collected by Earth observing satellites that need to be processed to better understand global scale changes in biosphere processes and patterns. Other examples of important applications of data mining include understanding gene functions in the field of genomics, the categorization of stars and galaxies in the field of astrophysics, and using data obtained through monitoring network traffic to detect illegal network activities. The key technical challenges in mining these data sets include (i) high volume, dimensionality and heterogeneity; (ii) the spatio-temporal aspects of the data; (iii) possibly skewed class distributions; (iv) the distributed nature of the data; (v) the complexity in converting raw collected data into high level features. High performance data mining is essential to analyze the growing data and provide analysts with automated tools that facilitate some of the steps needed for hypothesis generation and evaluation.
References [1] R. Agrawal, T. Imielinski, and A. Swami. Database mining: A performance perspective. IEEE Transactions on Knowledge and Data Eng., 5(6):914–925, December 1993. 116 [2] R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of items in large databases. In Proc. of 1993 ACM-SIGMOD Int. Conf. on Management of Data, Washington, D. C., 1993. 113 [3] R. Agrawal and J. C. Shafer. Parallel mining of association rules. IEEE Transactions on Knowledge and Data Eng., 8(6):962–969, December 1996. 114
124
Vipin Kumar et al.
[4] R. Agrawal and R. Srikant. Fast algorithms for mining association rules. In Proc. of the 20th VLDB Conference, pages 487–499, Santiago, Chile, 1994. 114 [5] J. Chattratichat, J. Darlington, M. Ghanem, Y. Guo, H. Huning, M. Kohler, J. Sutiwaraphun, H. W. To, and D. Yang. Large scale data mining: Challenges and responses. In Proc. of the Third Int’l Conference on Knowledge Discovery and Data Mining, 1997. 117 [6] M. S. Chen, J. Han, and P. S. Yu. Data mining: An overview from database perspective. IEEE Transactions on Knowledge and Data Eng., 8(6):866–883, December 1996. 111, 112 [7] D. J. Spiegelhalter D. Michie and C. C. Taylor. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994. 116 [8] S. Goil, S. Aluru, and S. Ranka. Concatenated parallelism: A technique for efficient parallel divide and conquer. In Proc. of the Symposium of Parallel and Distributed Computing (SPDP’96), 1996. 117 [9] D. E. Goldberg. Genetic Algorithms in Search, Optimizations and Machine Learning. Morgan-Kaufman, 1989. 116 [10] R. Grossman, C. Kamath, P. Kegelmeyer, V. Kumar, and R. Namburu. Data Mining for Scientific and Engineering Applications. Kluwer Academic Publishers, 2001. 112 [11] E.-H. Han, G. Karypis, and V. Kumar. Scalable parallel data mining for association rules. In Proc. of 1997 ACM-SIGMOD Int. Conf. on Management of Data, Tucson, Arizona, 1997. 114, 115 [12] E. H. Han, G. Karypis, and V. Kumar. Scalable parallel data mining for association rules. IEEE Transactions on Knowledge and Data Eng., 12(3), May/June 2000. 115 [13] J. Han and M. Kamber. Data Mining: Concepts and Techniques. MorganKaufman, 2000. 112 [14] D. Hand, H. Mannila, and P. Smyth. Principles of Data Mining. MIT Press, 2001. 112 [15] M. V. Joshi, E.-H. Han, G. Karypis, and V. Kumar. Efficient parallel algorithms for mining associations. In M. J. Zaki and C.-T. Ho, editors, Lecture Notes in Computer Science: Lecture Notes in Artificial Intelligence (LNCS/LNAI), volume 1759. Springer-Verlag, 2000. 113, 114, 115 [16] M. V. Joshi, G. Karypis, and V. Kumar. ScalParC: A new scalable and efficient parallel classification algorithm for mining large datasets. In Proc. of the International Parallel Processing Symposium, 1998. 117, 120 [17] M. V. Joshi, G. Karypis, and V. Kumar. Universal formulation of sequential patterns. Technical Report TR 99-021, Department of Computer Science, University of Minnesota, Minneapolis, 1999. 115 [18] R. Kufrin. Decision trees on parallel processors. In J. Geller, H. Kitano, and C. B. Suttner, editors, Parallel Processing for Artificial Intelligence 3. Elsevier Science, 1997. 117 [19] R. Lippmann. An introduction to computing with neural nets. IEEE ASSP Magazine, 4(22), April 1987. 116 [20] M. Mehta, R. Agrawal, and J. Rissanen. SLIQ: A fast scalable classifier for data mining. In Proc. of the Fifth Int’l Conference on Extending Database Technology, Avignon, France, 1996. 116 [21] R. A. Pearson. A coarse grained parallel induction heuristic. In H. Kitano, V. Kumar, and C. B. Suttner, editors, Parallel Processing for Artificial Intelligence 2, pages 207–226. Elsevier Science, 1994. 117
High Performance Data Mining
125
[22] J. Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993. 116 [23] J. Shafer, R. Agrawal, and M. Mehta. SPRINT: A scalable parallel classifier for data mining. In Proc. of the 22nd VLDB Conference, 1996. 116, 117, 120 [24] A. Srivastava, E.-H. Han, V. Kumar, and V. Singh. Parallel formulations of decision-tree classification algorithms. Data Mining and Knowledge Discovery: An International Journal, 3(3):237–261, September 1999. 117 [25] M. Steinbach, P. Tan, V. Kumar, S. Klooster, and C. Potter. Temporal data mining for the discovery and analysis of ocean climate indices. In KDD Workshop on Temporal Data Mining(KDD’2002), Edmonton, Alberta, Canada, 2001. 122 [26] M. Stonebraker, R. Agrawal, U. Dayal, E. J. Neuhold, and A. Reuter. DBMS research at a crossroads: The vienna update. In Proc. of the 19th VLDB Conference, pages 688–692, Dublin, Ireland, 1993. 111 [27] P. Tan, M. Steinbach, V. Kumar, S. Klooster, C. Potter, and A. Torregrosa. Finding spatio-temporal patterns in earth science data. In KDD Workshop on Temporal Data Mining(KDD’2001), San Francisco, California, 2001. 121 [28] M. J. Zaki. Parallel and distributed association mining: A survey. IEEE Concurrency (Special Issue on Data Mining), December 1999. 114
Data Mining for Data Classification Based on the KNN-Fuzzy Method Supported by Genetic Algorithm José L. A. Rosa and Nelson F. F. Ebecken COPPE, Universidade Federal do Rio de Janeiro, Brazil
Abstract. This paper presents a classification method based on the KNN-Fuzzy classification algorithm, supported by Genetic Algorithm. It discusses how to consider data clustering according to the Fuzzy logic and its consequences in the area of Data Mining. Analyses are made upon the results obtained in the classification of several data bases in order to demonstrate the proposed theory.
1
Introduction
Data classification is one of the most used data mining tasks. ANDERBERGS [1] defines classification as being the process or act to associate a new item or comment to a category. As example, a person can be classified, according some attributes: sex (female or male), nationality (country where it was born), naturalness (state where it was born), instruction degree (illiterate or not), height (low, high). Most of the knowledge discovery techniques are based on mathematical statistics and Machine Learning fundamentals [2]. Many classification methods are used and some of them are described in [3]. A memory based reasoning strategy was adopted in this paper. Basically it needs: the set of stored cases, a distance metric to compute distances between cases and the number of nearest neighbors to retrieve. This corresponds to a classical nonparametric architecture. The basic idea is very straight-forward. For training, all input-to-output pairs in the training set are stored into a database. When an estimation or a classification is needed on a new input pattern, the answer is based on the nearest training patterns in the database. The method requires no training time other than the time required to preprocess and store the entire training set. It is very memory intensive since the entire training set is stored. Classification/estimation is slow since the distance between input pattern and all patterns in the training set must be computed. This method typically performs better for lower dimensional problems (less than 10), as theoretical and empirical results suggest that the amount of training data needed in higher dimensions is greater than that required in other models. One of the difficulties that arises when utilizing this technique is that each of the labeled samples is given equal importance in deciding the class memberships of the J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 126-133, 2003. Springer-Verlag Berlin Heidelberg 2003
Data Mining for Data Classification Based on the KNN-Fuzzy Method
127
pattern to be classified and there is no indication of its "strength" of memberships in that class. This problem can be addressed by incorporating fuzzy set theory. The basis of the fuzzy algorithm is to assign membership as a function of the vector's distance from its nearest neighbors and those neighbors' memberships in the possible classes. The main questions now are: -
How many nearest neighbors (K) must be considered? (empirical formulas, K should be less than the square root of the total number of training patterns) How weight the distance to calculate each neighbor's contribution to the membership value?
In this paper those questions are answered by using a genetic algorithm. A parallel version was implemented, including all the commented improvements to build a more precise and efficient classification methodology. Section 2 presents the KNN-Fuzzy Method and the genetic algorithm approach used to optimize it. The parallelization schema and its consequences, and the available computational environment are discussed in section 3. The studied case is analyzed with the Amdahl's law point of view in section 4. Some conclusions and comments are presented in section 5.
2
KNN-Fuzzy Method
KNN (K-Nearest Neighbors) consists of the identification of groups of individuals with similar features and posterior grouping. The calculation of the distance of the unknown sample in relation to the known samples is made by: (1) Assuming i known samples with two classes, 0 and 1, the KNN algorithm can be summarized as indicated below. BEGIN READ i known samples (ks) READ unknown sample (us) Compute each distance from us to ksi ASSIGN both distance and class to a distance vector SORT distance vector by distance SELECT K vector distance samples Compare each class from the K selected samples IF (class is equal 0) Increment class0 ELSE Increment class1 ENDIF END
128
José L. A. Rosa and Nelson F. F. Ebecken
Each distance is related to the corresponding class in a vector of distances. After the calculation of all the distances, the K smaller distances are selected; the unknown sample then is classified as being of the same present class. As shown in [2], the value of K must be that one that satisfies the function max that defines the maximum value of neighbors (samples) Kmax, who are near the unknown sample, in relation to all the samples of the considered set, K tot: (2) In the classification task, it must be stressed that a variable can have more contribution in the definition of the class than others. A possible solution may be to consider the weight of the variables in accordance with its degree of importance for the classification process. According to [1], the balance of the variables transforms the original space in a representation space that can be wider or with reduced number of dimensions, which gives better classification accuracy. In accordance with the work presented in [4], the basis of the KNN-Fuzzy classification algorithm is to associate the relevancy of a vector to its next Kneighbors and those members in possible classes. Let W = { Z1, Z2..., Zc } be the set of c prototype representing the c classes. Let µ (x) the membership (to be computed) associated to vector x. As seen in (3): (3)
the associate memberships of x, are influenced by inverse of the distances of the neighbors and its c membership’s classes. The inverse of the distance serves as weight, taking in consideration how close or not the vector to be classified is from the classified vector. The variable m determines how heavily is the weight attributed at a distance of each neighbor to the value of membership. If the value of m is equal two, then the contribution of each point of the neighborhood is considered reciprocal to the longdistance of the point to be classified. As m increases, it diminishes the contribution of each neighbor; however, the reduction of m (m greater than one), implies more strong contribution. Assuming i known samples with two classes, 0 and 1 the KNN-Fuzzy algorithm can be summarized as follow. BEGIN READ i known samples (ks) READ unknown sample (us) Compute each distance from us to ksi ASSIGN both distance and class to a distance vector SORT distance vector by distance (column 1) SELECT K vector distance samples Compute µ(x) using (3) END
Data Mining for Data Classification Based on the KNN-Fuzzy Method 2.1
129
The Proposed Methodology
In this paper, a standard genetic algorithm scheme is employed to evolve the KNNFuzzy method in order to achieve the best classification precision The binary coding [5] was used to generate the chromosome (gene). The representation corresponds to 2n chromosomes, where n is the amount of genes that each chromosome possesses. The set of all the patterns that the chromosome can assume forms its space of search. In the present study the K, m and weight variables are taken into account. To evolve the population the genetic operators promote modifications that improve the average performance of the population to each generation, using information on the previous generations. The new population starts with the selection of the individuals of the ascending population with bigger probability to compose the following generation. In a population of n individuals, each one with fitness f, a chromosome c has it probability p of selection of this individual given by: (4) There are many selection methods, as described in [6]. In this work the method of stochastic selection via tournament was used. Among these chromosomes, the one with bigger aptitude is selected for the following population. The process then is repeated until it fills the descending population. Once selected the individuals of the descending population, the operators of crossover and mutation are applied. The crossover operator makes swap on fragments between pairs of chromosomes, whose strings are cut in a random position producing, for each individual, two amounts of alelos that will have to be reallocated. Mutation inverts one or more bits of an individual in a probability generally low, improves the diversity of the chromosomes in the population and destroys the information contained in the chromosome. In this way the values of K, m and the weights of the variables were obtained to achieve the best classification precision.
3
Parallel Implementation
The main issues under consideration in the study of parallel implementation of conceived sequential algorithms are the partitioning algorithm schema and the target machine. 3.1
Partitioning Algorithm Schema
For the partitioning algorithm schema the study considers the use of the data parallelism approach that keeps a copy of the entire labeled samples, with the internal variables and functions, in each processing node partitioning training data set among nodes. This approach ensures that all values needed during the training phase like K,
130
José L. A. Rosa and Nelson F. F. Ebecken
m and variable weights, are locally available, reducing the number of messages passing among nodes and the algorithm synchronization. In fact, all nodes perform only one communication after each complete presentation of a training data subset. In distributed memory machines this approach seems to be very efficient leading to great gain of performance. This implementation of data parallelism uses a control node that performs genetic operations, realizes chromosome encoding and decoding, broadcasts the values of K, m and weights of the variables, and summarize the interested value (total of right classes). All nodes start with same internal parameters but with different subsets of training data. During training phase of KNN-Fuzzy method, each node passes its training data subset producing partial errors that must be combined with partial errors produced by other nodes, generating an overall error. This error is used to update the weights of the variables in each node, until the procedure reaches the minimal acceptable overall error. It can be pointed out that each node only broadcasts its partial error to the control node at this time. All other calculations involve local data and can be made without synchronization. This drastically reduces the communications between nodes. The application uses few and simple MPI standard primitives and can be portable for several machines. 3.2
Target Machine
The machine available in this work was an academic pc-cluster of the NACAD/COPPE/UFRJ High Performance Computing Laboratory. This machine contains 8 nodes (Intel Pentium III with 256 MB memory) each connected by a Fast Ethernet network). The Linux operating system and LAM MPI [7] programming environment supports the parallel execution of jobs for that environment. LAM features a full implementation of the MPI communication standard among parallel process. The access obtained for the execution of the application in this environment was shared, i.e., all programs share all nodes during execution. It could be emphasized here that time tests were made with unbalanced work on nodes.
4
The Studied Case
The data set has been extracted from a real world insurance database that contains relationships among clients, contracts and tariffs. The former objective of this application is modeling classification criteria for two predefined classes and performance assessment. Some characteristics and values have been protected by nondisclosure policies. Despite of the relation's complexity, the attributes and its values were grouped in one single table for this study. This table contains 80 attributes and 147478 registers for the predefined classes. The data set had been pre-processed, clearing some aspects not suitable for data mining methods. The resultant set contains 64 attributes and 130143 lines of registers. In table 1 are presented the elapsed time spent in a limited execution of the application with different number of nodes for each job submission, the ideal (IdSp) and obtained speedup (Sp) and efficiency (Ep).
Data Mining for Data Classification Based on the KNN-Fuzzy Method
131
The application in each node reads the data files (known , training and test sets), according to the "fagknnp.cfg" configuration file, which can be altered depending on desired data set partitioning. The application of Amdahl's Law for performance evaluation shows a 99,6% of parallelization coefficient for the developed algorithm. This is explained by the large time expended by KNN-Fuzzy procedure in the overall application time (in a serial evaluation the total computation time is equal 105057 s and the KNN-Fuzzy procedure evaluation time is equal 104634 s). Table 1. Jobs submission report
nodes 2
Time (s) 57804
IdSp 1,991980
Sp 1,817469
Ep 0,995990
3
38287
2,976035
2,743934
0,992012
4
30173
3,952260
3,481821
0,988065
5
23945
4,920749
4,387430
0,984150
6
19905
5,881592
5,277920
0,980265
7
17775
6,834881
5,910380
0,976412
8
15880
7,780703
6,615680
0,972588
Fig. 1. Representation of the ideal and obtained speedup, according to Amdahl's Law
Fig. 2. Obtained efficiency
132
José L. A. Rosa and Nelson F. F. Ebecken
5
Classification Task
To demonstrate the efficiency of the genetic algorithm in the optimization of database classification, were used three small well known databases: Iris, Pima and Erudit. Table 2 shows the descriptions of each one of the analyzed databases: Table 2. Databases descriptions
Database
Samples
Variables
Classes
150 768
4 8 27
3 2 2
Iris Plant Database Pima Indians Diabetes Erudit
In a first step, the databases had been analyzed without the aid of the genetic algorithm, getting itself, therefore, results on this side of the waited one. In a second step, the method KNN-Fuzzy was optimized by a genetic algorithm to find the ideal values of the classification parameters: K, m and weights of the variable. Table 3 presents the obtained classification results. Table 3. Comparison between standard and GA strategies
Database
Iris Pima Erudit
Standard
GA
k
m
%
k
m
%
90 384 130
2 3 2
97.0 79.0 66.0
30 56 114
0.025781 0.0202148 0.004297
100 89.0 94.4
As it can be observed, a significant improvement in the correct classification rate was achieved in all studied cases.
6
Conclusions
The volume of data existing in real world data warehouse with complex relations can only be handled by methods implemented in high performance computers with efficient parallel methodology. The case used in this work has these characteristics and could only be handled by parallel application. The use of a standard library of communication primitives allows portability of the application for several machines, bringing great vantages for the methodology. The KNN-Fuzzy method, optimized by a genetic algorithm, has the ability of produce a very precise classification model comparing with other paradigms used in classification problems. The obtained performance of the parallel implementation has shown very good results.
Data Mining for Data Classification Based on the KNN-Fuzzy Method
133
For classification tasks, the KNN-fuzzy method showed to be very useful. It improves the accuracy and results in high classification precision, not obtained with crisp implementations. The genetic algorithm optimization has great importance in the improvement of the accuracy, finding optimal values of K and m. The generation of the population of chromosomes in a random form showed, in practical, to be more attractive than the exploitation of previous populations. Depending on the generation of the population, values close or not the global minimum can occur, resulting in a variation in the processing efforts.
References 1.
ANDERBERG, MICHAEL R. Cluster Analysis for Applications. Academic Press, 1973. 2. MITCHIE, D., SPIEGELHALTER, D. J., TAYLOR, C. C. Machine Learning, Neural and Statistical Classification. Ellis Horwood, 1994. 3. MANLY, B. F. J. Multivariate Statistical Methods. A Primer , Chapman & Hall, London, 1986. 4. KELLER, J. M. , GRAY, M. R. , GIVENS JR. , J. A. , A Fuzzy K-Nearest Neighbor Algorithm, IEEE Transactions on Systems, Man, and Cybernetics, v. SMC-15, n. 4, pp. 258-260, July/August 1985. 5. HEATH, F. G. , Origins of Binary Code, Scientific American, v.227, n.2, pp.7683, August 1972. 6. ESPÍNDOLA, ROGÉRIO P. Optimizing Classification Fuzzy Rules by Genetic Algorithms, M.Sc thesys, COPPE/PEC, May, 1999. 7. LAM / MPI Parallel Computing home page. Internet: http://www.lam-mpi.org , 2002. 8. Databases from UCI Machine Learning Repository obtained in http://www.ics.uci.edu/~mlearn/ MLRepository.html 9. Databases from European Network in Uncertainty Techniques Developments for Use in Information Technology - ERUDIT, obtained in http://www.erudit.de/erudit/index.htm 10. KDD-Sisyphus. KDD Sisyphus Workshop – Data Preparation, Preprocessing and Reasoning for Real-World Data Mining Applications. Internet: http://research.swisslife.ch/kdd-sisyphus, 1998.
Lignin Biosynthesis and Degradation – a Major Challenge for Computational Chemistry Bo Durbeej1,2, Yan-Ni Wang1,3, and Leif A. Eriksson1 1
Division of Structural and Computational Biophysics Department of Biochemistry, Box 576, Uppsala University 751 23 Uppsala, Sweden 2 Department of Quantum Chemistry, Box 518, Uppsala University 751 20 Uppsala, Sweden 3 currently at: Advanced Biomedical Computing Center SAIC Frederick, P.O. Box B, National Cancer Institute at Frederick Frederick, MD 21702-1201, USA
Abstract. In the present chapter we review a series of computational studies related to lignin biosynthesis and degradation, with the aim to understand at a molecular level processes crucial to paper and pulp industries. Due to the complexity of the problem, a wide variety of computational approaches are employed, each with its own merits. From the theoretical studies we are able to draw conclusions regarding the behavior of lignol monomers and their corresponding dehydrogenated radicals in aqueous solution and in lipid bilayers, and reaction mechanisms, conformations and relative stabilities of lignol dimers.
1
Introduction
Despite the global economic significance of pulp and paper industries, computational chemistry techniques have to date been very little employed in order to increase our understanding of processes pertaining to biosynthesis and degradation of wood fibers and their constituents. The main reason for this is the complex constitution of wood fibers, putting very high demands on the computational method to be used both with respect to applicability and size of the systems that in the end can be explored at high accuracy. To this should also be added the limited amount of available high-resolution (i.e. atomic level) structural data, which makes construction of computational models a difficult task. A wood fiber cell wall is composed of bundles of cellulose polymers, so-called microrfibrils, which are synthesized by trans-membrane enzymes arranged in a ‘rosette-like’ fashion, Fig. 1 [1]. Each microfibril normally contains 20-40 cellulose polymers, has a cross-section diameter of 20-30 µm, and is coated with a layer of branched chains of hemicelluloses [2, 3]. The vacant regions between the microfibrils are filled with very low-structured lignin polymers, providing rigidity (i.e. mechanical J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 137-165, 2003. Springer-Verlag Berlin Heidelberg 2003
138
Bo Durbeej et al.
strength) to the plant. In addition, lignin reduces the water permeability (by increasing the hydrophobicity) of the cell walls, and participates in the plant’s defense as a physicochemical barrier against pathogens [4]. The wood cell wall as such also has an overall complex structure. In the simplest possible picture, the cell walls formed in higher plants can be thought of as comprising four major morphological regions, differing in the relative concentration of cellulose, hemicellulose and lignin as well as the relative orientation of the cellulose microfibrils (Fig. 2) [5].
Fig. 1. Schematic model of cellulose synthesizing enzymes in a wood fiber cell wall (from ref [1])
Fig. 2. Schematic model of the different morphological regions of a wood fiber cell wall. P stands for the primary cell wall, and S1, S2, and S3 stand for the outer, middle, and inner layers of the secondary cell wall, respectively (reprinted by courtesy of the Wood Ultrastructure Research Center (WURC) – www-wurc.slu.se)
Lignin Biosynthesis and Degradation – a Major Challenge
139
In this review, we will try to illustrate how different computational chemistry techniques – ranging from classical molecular dynamics simulations to quantum chemical methods – can be employed to provide valuable new information on, in particular, various aspects of lignin biosynthesis. We will present results from recent studies performed in our group focusing on the initial stages of this process. Besides playing a crucial role in wood anatomy and physiology, lignin is also important from an economical point of view. In paper making, it is considered an undesirable component – yielding paper with poor printing quality that readily fragments and turns yellow with age [6] – that has to be removed by alkaline pulping and chemical bleaching. Even though the present work, as noted above, primarily concerns lignin biosynthesis, we hope it will be evident how computational studies of this type also can benefit industry focussing on lignin degradation. OH
OH
OH
γ β α 1 2
6
3
5
MeO
OMe
4
OMe OH
OH
OH γ
p - Coumaryl alcohol
Coniferyl alcohol
Sinapyl alcohol
Fig. 3. The three different hydroxycinnamyl alcohols (p-coumaryl, coniferyl and sinapyl alcohol) that form the building blocks of lignin biopolymers
CH2OH
CH2OH
CH2OH
CH
CH
CH
.CH
CH
CH
CH
CH
CH
CH
.
CH2OH
CH2OH
.
.
OMe
OMe
OMe
OMe
OMe
O
.
O
O
O
O
RO4
R5
R1
Rβ
R3
Fig. 4. Main mesomeric forms of a phenoxy radical produced by oxidation of a coniferyl alcohol by peroxidase
140
Bo Durbeej et al.
The biosynthesis of the lignin polymer is initiated by enzymatic dehydrogenation of hydroxycinnamyl alcohols (monolignols) of three different type : p-coumaryl alcohol, coniferyl alcohol, and sinapyl alcohol (Fig. 3) [7, 8]. These monolignols are oxidized by peroxidases and/or laccases to form phenoxy radicals that have five different resonance structures (Fig. 4). It has been proposed that the phenoxy radicals couple non-enzymatically to produce quinone methides (QMs) as reactive intermediates [7, 8]. The dimeric QMs are then converted into various dilignols by addition of water, or by intra-molecular nucleophilic attack by primary alcohols or quinone groups. The mechanism by which lignin polymerization proceeds has not yet been fully understood, and several proposals exist (ref [7], and references therein). All dilignols are phenols, and can therefore also undergo enzymatic dehydrogenation to form dilignol radicals. Hence, in one model of lignin growth, dilignol radicals are proposed to couple with phenoxy radicals to produce phenolic trilignols which, in turn, are oxidized enzymatically etc. This model of molecular growth is thus based on repeating dehydrogenations and successive radical-radical additions. In a second growth mechanism, enzymatic dehydrogenation is restricted to monolignols, and the polymer is formed by successive non-radical addition of phenols to the QMs. In a third growth mechanism, the polymer evolves from polymerization of QMs. It is also a matter of dispute whether, or to what extent, the polymerization is controlled enzymatically. It has, for example, been demonstrated that an auxiliary protein affects the stereospecificity of in vitro phenoxy radical coupling reactions [9]. The high variability in the molecular structure of lignin is largely due to the different resonance structures of the phenoxy radicals. This leads to a large number of possible coupling reactions and, thus, to a disordered polymer. In theory, twenty-five different combinations of intermonomeric linkages are possible. The most frequent linkages in both softwood and hardwood are β-O4, β-5, β-β, β-1, 5-5, and 5-O4 (ref [10], and references therein; cf. Fig. 5). The relative distribution of the linkages is however not clearly established, since different analytical methods give different results [11, 12]. The complex constitution of lignin makes it difficult to solubilize the whole polymer and hampers chemical and physical characterization. It is furthermore difficult to extract chemically unmodified lignin from wood [13]. As a result, current experimental techniques can yield only limited information on lignin structure and mechanistic details of lignin polymerization. NMR data has been obtained for lignin model compounds [14, 15], and crystallographic structures of β-O4 linked dilignols – the major type of intermonomeric linkage in both softwood and hardwood [16] – have been solved [17, 18]. The difficulties associated with experimental studies have paved the way for molecular modelling as an alternative approach to provide an enhanced understanding of lignin chemistry. Lignin structure has been the subject of several computational studies using both molecular mechanics force field methods [19, 20, 21], and quantum chemical methods [19, 22, 23, 24]. Lignin polymerization has also been investigated by means of computer modelling [25, 26, 27, 28, 29, 30]. The models developed by these researchers account for many experimental findings – the Monte Carlo simulation strategy of Roussel and Lim [29, 30], e.g., reproduce both the distribution of intermonomeric couplings and the number of bonds per monomer rather well – and
Lignin Biosynthesis and Degradation – a Major Challenge
141
CH2OH CH
CH2OH
CH
CH
OH OMe
CH
B CH2OH HC R1
OMe
B
HC
O
C
O
CH2OH
OMe O
HC
R2
H2C
CH
HC
CH
HC
CH2
O A
A
OMe
OMe
OMe
OH
OH Guaiacylglycerol-β-coniferyl ether
OH Pinoresinol
Dehydrodiconiferyl alcohol
R1 = H and R2 = OH: erythro β-O4 R1 = OH and R2 = H: threo β-O4
β-5
β-β
CH2OH OMe
CH2OH HC R1
C
OH R2
B
CH2OH
CH2OH
CH
CH2OH
CH
CH
CH
CH
CH
CH
CH
A MeO
A OMe OH Diarylpropane R1 = H and R2 = OH: erythro β-1 R1 = OH and R2 = H: threo β-1
B
O
MeO
OMe
OMe OH
OH
OH
Dehydro-bis-coniferyl alcohol 5-5
Diphenylether 5-O4
Fig. 5. Dimeric lignin structures containing the principal linkage modes between coniferyl alcohol monolignols. The acronyms reflect the sites of initial radical-radical coupling, e.g. the β-O4 dilignol is formed by coupling of the mesomeric forms Rβ and RO4 of Figure 4
enable investigations of comparatively large systems. None of these models, however, have included a three-dimensional representation of the atomic structure of lignin. Even though a subsequent model devised by Jurasek [31, 32] attempts to make use of realistic lignin structures, and indeed provides valuable new insight into the process of lignification, the knowledge of the mechanistic details at the atomic level remains
142
Bo Durbeej et al.
limited. For example, no estimates of the energetics of lignin polymerization have evolved from the abovementioned studies. In this context, quantum chemical modelling can be employed to provide new information. While the computational effort associated with such an approach inevitably puts restrictions on the size of the system to be studied, these methods offer a possibility to shed light on the atomic as well as the electronic details of lignin polymerization, and can be used to make predictions as to whether a certain mechanism is energetically favourable. Below, some results from our laboratory [33] on the energetics of different reaction mechanisms for the formation of β-O4 linked dimeric lignin structures will be presented. Another aspect of lignin polymerization that has attracted the attention of molecular modelers is whether the initial dimerization of monolignols is thermodynamically controlled, i.e. whether the observed distribution of intermonomeric linkages to a great extent can be correlated with the stabilities of the corresponding dilignols. This so-called thermodynamic control hypothesis has been explored on the basis of semiempirically calculated heats of formation of various dilignols [34], and has recently been revisited in a study making use of high-level quantum chemical methods [24]. Besides thermodynamic control, the electron distribution of phenoxy radicals and steric repulsion effects (between phenoxy radicals) during the initial radical-radical coupling have been suggested as possible intrinsic lignin properties having an impact on the observed ratios of the various linkages. In an early study, Mårtensson and Karlsson [35] applied resonance stabilization principles to deduce that the unpaired electron of a phenoxy radical should exist primarily at the side chain β position, at the phenolic oxygen, and at the 1, 3, and 5 positions on the aromatic ring, respectively (Fig. 4). By assuming that radical-radical addition reactions involving the 1 and 3 positions should be sterically hindered (i.e. kinetically controlled) by the 3-hydroxypropen-1-yl- and the methoxy group, respectively, these researchers furthermore argued that the phenoxy radical electron distribution would account for the fact that β5, β-β, β-O4, and 5-5 are the major oxidative dimerization products (dilignols) observed in experimental studies. This hypothesis, which below will be referred to as the spin distribution hypothesis, was later tested by Elder and Worley [36], and subsequently revisited by Elder and Ede [34]. Both studies made use of semiempirical computational chemistry methods to calculate spin densities in a free coniferyl alcohol radical. Even though the calculations showed that the positions through which dimerization occurs actually have the most distinct radical character, no direct correlation between calculated spin densities and linkage ratios could be pointed out. Below, the first evaluation of the spin distribution hypothesis that goes beyond a semiempirical level of approximation will be presented [37]. The effect of the biological environment on the polymerization process is another important aspect of lignin biosynthesis that needs to be further investigated. The various monolignols are synthesized intracellularly, and thus have to cross the plasma membrane to enter the cell wall. However, it still remains unclear how this transport occurs, and how the monolignols aggregate in various biological environments. As a first step towards gaining insight into these issues, we will below present some preliminary results from molecular dynamics simulations of monolignols in aqueous solution and in a lipid bilayer [38, 39].
Lignin Biosynthesis and Degradation – a Major Challenge
143
The structural complexity of lignin makes lignin degradation in industrial processes a difficult task. Since certain intermonomeric linkages are stronger than others, a better understanding of the chemistry of the different linkages would shed light on how these may be weakened (i.e. more susceptible to cleavage) through chemical modification. Such information could be utilized to optimize the conditions for the pulping and bleaching processes. In this context, it should also be noted that enzymatic and photochemical bleaching have been proposed as alternatives to the traditional bleaching chemicals chlorine (Cl2), chlorine dioxide (ClO2), hydrogen peroxide (H2O2) and similar [40, 41, 42]. The present chapter is organized as follows. In the next section, the computational techniques used in the different studies are outlined. In sections 3 and 4, results from molecular dynamics simulations of monolignols and their corresponding dehydrogenated radicals in aqueous solution and in a lipid bilayer are presented. In section 5, structural features of the most common lignin dimerization products are discussed, and the thermodynamic control and spin distribution hypotheses are evaluated. In section 6, finally, an investigation of the energetics of different reaction mechanisms for the formation of β-O4 linked dimeric lignin structures is presented. The systems explored range in size from those containing thirty or so atoms to those with several thousands of atoms, and the aim with this review is to illustrate that a combination of different methodologies – each with its own advantages, bottlenecks, and computational requirements – is required if one wishes to study a complex biochemical problem such as lignin biosynthesis using the tools of computational chemistry.
2
Computational Approaches
When embarking on a computational project, the choice of computational scheme will depend upon the nature of the chemical problem at hand, and the applicability of the method in terms of accuracy and computational cost. If, for example, we are interested in studying a biochemical system consisting of, say, 10000 atoms, the method of choice is to run a classical molecular dynamics (MD) simulation of the system. The result of such a simulation is a trajectory, which shows how the atomic positions vary in time. If the simulation is carried out long enough, the system will eventually reach an equilibrium state with respect to certain thermodynamic and structural properties. By averaging over an equilibrium trajectory, many macroscopic physical properties can subsequently be calculated. In a system consisting of an enzyme and its corresponding substrate, or a solute in a liquid, this methodology offers a possibility to calculate, e.g., the binding constant of the substrate to the enzyme and the radial distribution function between specific solute/solvent atom pairs, respectively. MD simulations also provide a means to calculate non-equilibrium macroscopic properties such as diffusion constants of molecules within biological membranes. A key problem within this branch of molecular modeling is that of choosing an adequate force field by which the different bonded and non-bonded interactions are described through potential energy functions. These functions depend on the atomic positions only, i.e. electrons are not considered explicitly. This means that MD
144
Bo Durbeej et al.
simulations are applicable only to systems that remain in their electronic ground state, and furthermore that a proper treatment of chemical reactions is beyond reach. In the work presented in section 3, we have performed MD simulations in order to investigate the behavior of the three different monolignols, and their corresponding phenoxy radicals, in aqueous solution. The force field by Cornell et al. [43] was applied to all atoms except to those of the unsaturated hydrocarbon tail, which were described using the MM3 force field [44]. The TIP3P water solvent model was used throughout all simulations [45]. Force field charges for the monolignol atoms were obtained from initial quantum chemical calculations (level of theory given in section 3 below). In order to gain insight into the trans-membrane transport of lignin precursors, MD simulations of p-coumaryl alcohol – the smallest monolignol – in a dipalmitoylphosphatidylcholine (DPPC) membrane were also carried out; the results of which are presented in section 4. In this case, the monolignol was described using the AMBER force field [46], whereas a refined version thereof [47] previously employed in MD simulations of membranes was applied to the DPPC model [48]. Again, the TIP3P water model was used and monolignol charges were obtained from initial quantum chemical calculations. All MD simulations described here were performed using the AMBER 5.0.1 simulation package [49], and further computational details thereof are given in the respective sections below. In cases where we are interested in calculating electronic and spectroscopic properties of a certain chemical compound, or would like to study processes that involve the breaking and formation of covalent bonds (i.e. chemical reactions), a quantum mechanical description in which electrons are treated explicitly becomes mandatory. Including electronic interactions dramatically increases the complexity of the equations that in the end need to be solved. As a result, the number of atoms that can be included in a quantum chemical calculation typically lies in the 10-100 atoms range, depending on the methodology chosen. The more accurate the method, the more demanding the calculation will be. This implies that quantum chemical calculations generally are performed on small model systems, and that solvation effects are either neglected or taken into account through the use of so-called dielectric continuum models [50, 51, 52]. Based on existing benchmark studies, the quantum chemical method best-suited for the system under study can be selected and subsequently applied to provide information on explicit orbital interactions, estimates of bond strengths, charge distributions, spectroscopic properties, and similar; information that might be difficult to obtain experimentally. By using the highly robust geometry optimization algorithms of today, which have been implemented in several commercially available quantum chemical computer codes, it is furthermore possible to locate stationary points (i.e. minima and transition states) along the reaction coordinate of a chemical reaction. Thereby, the effects of different substituents on energy barriers and reaction energies can be calculated, and shortlived intermediates can be identified; a cumbersome, if not impossible, task from an experimental point of view. The success of quantum chemical modeling (in terms of providing results in good agreement with experimental data) is partly a result of many chemical phenomena being highly localized. It is thereby possible to construct a simplified computational
Lignin Biosynthesis and Degradation – a Major Challenge
145
model that both captures the important features of the macroscopic system using only a small number of relevant chemical groups, and enables the calculations to be performed at a necessary high level of theory. Of the many quantum chemical methods available today, those based on density functional theory (DFT) [53, 54] have over the past decade become increasingly more popular, and have been extensively used in fields as diverse as solid state physics, atmospheric chemistry, and biochemistry. A key theorem in DFT says that all groundstate properties of a many-body system, such as the total electronic energy Etot, can be obtained from the ground-state electron density ρ(r) of that particular system. We hence wish to find an approach where we can use use ρ(r), which depends on 3 spatial coordinates (the value of the density at point r), rather than the wave function, which depends on 3N spatial coordinates (with N being the number of electrons and assuming fixed nuclear positions), as the basic variable for the description of the many-body system under study, and express the total electronic energy as a functional of the electron density; Etot = E[ρ(r)]. In combination with a variational principle for the electron density, and a computational methodology known as the Kohn-Sham self-consistent field approach, a computational scheme has been developed that is considerably less complicated than those arising within wave-function formalism. In addition to being applicable to larger systems, DFT methods already from the start take electron correlation effects into account through the so-called exchangecorrelation potential, whereas conventional ab initio wave-function methods include these effects through perturbation series expansions or increasing orders of excited state configurations added to the ‘zeroth-order’ Hartree-Fock solution. Thus, DFT methods can be used to perform accurate calculations at a comparatively low computational cost. For more details on DFT we refer to any of the recent textbooks on the subject [55, 56, 57]. In the quantum chemical studies presented in sections 5 and 6, we have exclusively employed the B3LYP hybrid density functional [58, 59, 60] as implemented in the Gaussian 98 program [61]. This exchange-correlation functional consists of a mixture of local and gradient corrected density functionals, and also includes a fraction of exact (Hartree-Fock) exchange. Bulk solvation effects of the surrounding medium are accounted for using the self-consistent polarized continuum model (PCM) by Tomasi and coworkers [50]. In this model, the surrounding medium is treated as a macroscopic continuum (with a dielectric constant ε), and the molecule under study is thought of as filling a cavity in this continuous medium. In order to mimic an aliphatic environment, ε was chosen to be equal to 4. Further computational details are given in the respective sections below.
3
MD Simulations of Monolignols and the Corresponding Phenoxy Radicals in Aqueous Solution
In order to better understand the hydrogen bonding and other interactions between the monolignols and their local surrounding, we undertook a detailed study of the three different hydroxycinnamyl alcohols and their corresponding radical forms in aqueous solution. The geometries of the different monolignols were first optimized using the
146
Bo Durbeej et al.
B3LYP functional in conjunction with the 6-31G(d,p) basis set. The optimized structures were then placed into rectangular water boxes with periodic boundary conditions, followed by initial energy minimization to remove the strain between the solute and surrounding water molecules. After equilibration simulations, the temperature was raised stepwise, and finally kept at 300 K using the Berendsen coupling algorithm [62]. A single temperature scaling factor was used for all atoms. The pressure was set to 1 bar (ca. 1 atm.) with isotropic position scaling and 500 ps relaxation time. The density of the equilibrated system was about 1.0 g/cm3. A 4 ns NPT ensemble production run was finally performed with 2 fs step size. A 9 Å atom based cut-off was used to separate short-range and long-range interactions, and Ewald summation [63] was employed to calculate long-range electrostatic interactions. The non-bonded pair list was updated every 10th time step. For the corresponding phenoxy radicals, similar energy minimization and equilibration simulations were carried out, except that the computationally faster Particle Mesh Ewald (PME) method [64, 65] was used to calculate long range electrostatic interactions. Finally 1 ns periodic boundary NPT production simulations were performed to analyze the behaviour of the radicals in solution. Lignin biosynthesis is a process involving dehydrogenated precursors, and thus the interaction between the hydroxyl groups and its environment is of key importance. In the current simulations information about the intermolecular solute-solvent atom pairs were analysed in terms of radial distributions functions (RDF) – data that provide information about the solvation shells and hydrogen bonding interaction. In Fig. 6 and 7 we show the RDF between the phenolic hydrogen and the oxygens of the surrounding water molecules, and between the phenolic oxygen of the corresponding dehydrogenated radicals and hydrogen of water, respectively. The simulations of the parent compounds (Fig. 6) reveal that for p-coumaryl alcohol there is a strong hydrogen bond formed to the hydroxyl hydrogen, ca 1.8 Å long and with an average number of coordinated solvation oxygen atoms of 1.2. For both coniferyl and sinapyl alcohol much weaker hydrogen bonds are found (at 1.9 and 2.0 Å average distance, respectively). These results clearly reflect that the neighbouring methoxy groups influence the interactions between lignin and the surrounding significantly by hindering the water molecules to approach the phenolic end, hence reducing the possibility of hydrogen bonding. The same results were found when analysing the O4 - - Owater RDF (not shown). The second solvation shell is seen as a broad shoulder around 3.4 Å, and is similar for all three monolignols. Turning to the dehydrogenated radicals (Fig. 7), the first peak of the O4 - - Hwater RDF display the same trends as for the parent compounds, albeit not quite as pronounced. For p-coumaryl alcohol the first hydrogen bonding shell appears at 1.8 Å (0.8 hydrogens), whereas for coniferyl and sinapyl the corresponding data are 1.9 Å (0.7 hydrogens) and 2.0 Å (0.6 hydrogens). Again steric hindrance is excerted by the increasing number of rotating methoxy groups in the series p-coumaryl (no OMe group), coniferyl (1 OMe) and sinapyl (2 OMe) alcohol. Similar analyses were made for the γ-hydroxy groups (of the ‘tail’ of the compounds). In this case all three systems, radical as well as non-radical, display identical RDF. The first peak in the Hγ - - Owater RDF appears at 1.9 Å, with magnitude 1.0, and with the second solvation shell at 3.4 Å. Common for all systems investigated is that all the hydrogen bonds are linear.
Lignin Biosynthesis and Degradation – a Major Challenge
147
Fig. 6. Radial distribution functions between cinnamyl alcohol hydroxyl group H4 and oxygen of surrounding water molecules. Dashed line: p-coumaryl alcohol; solid line: coniferyl alcohol; dot-dashed line: sinapyl alcohol
Fig. 7. Radial distribution functions between cinnamyl alcoholyl radical O4 and hydrogen of surrounding water molecules. Dashed line: p-coumaryl alcohol; solid line: coniferyl alcohol; dot-dashed line: sinapyl alcohol
148
Bo Durbeej et al.
From the simulations, information regarding diffusion of the monolignols in aqueous solution can also be extracted through mean square displacements (MSD). According to the Einstein relation [63], the MSD becomes linear with time, and the diffusion constant can be calculated from the slope of the MSD vs time. In the present simulations, the diffusion constants of p-coumaryl, coniferyl and sinapyl alcohol were estimated to about 2.5 x 10-9 m2s-1, 2.4 x 10-9 m2s-1, and 1.7 x 10-9 m2s-1, respectively, indicating that the p-coumaryl alcohol diffuses slightly faster than the other two, and that all diffusion constants are of similar magnitude as that of water itself. Similar results were obtained for the corresponding radicals. It hence appears that the methoxy substituents hinders the diffusion more than the presence of strong hydrogen bonding. Further simulations are under way with increased concentration of monolignols, to explore the possibility of aggregation.
4
MD Simulations of p-Coumaryl Alcohol in a Lipid Bilayer
Once the monolignols are synthesized in the cytoplasm, these must be transported across the cell membrane. The most likely transport mechanism is that of “free” diffusion through the lipid bilayer, although aggregation and more active transport mechanisms may also be plausible. In the present work we present MD simulation data for passive motion of p-coumaryl alcohol in a lipid bilayer. The initial lipid model was obtained by extracting a single configuration from a previous wellequilibrated simulation [48]. The details of the model and the simulations have been described elsewhere [66]. The membrane system consisted of 64 dipalmitoylphosphatidylcholine (DPPC) molecules located in the two halves of a planar bilayer and surrounded by a total of 1472 water molecules on both sides of the bilayer. This corresponds to 23 water molecules per lipid, which is within the range found experimentally for the number of water molecules at saturation. The initial edge sizes of the membrane were 45.6 Å, 45.6 Å, and 60.6 Å for X, Y, and Z direction respectively, leading to the initial value for the area per lipid to 0.65 nm2 (X and Y stand for the membrane layer plane and Z for the normal of the membrane). The pcoumaryl alcohol was placed randomly in the center of the pre-equilibrated membrane. The system was first carefully equilibrated using an NPT ensemble by scaling the temperature of the lipids and solute separately from the water, employing the Berendsen algorithm [62]. After the equilibration processes, a 1500 ps production simulation at constant pressure (P=1 atm) and temperature (T=323 K) with periodic boundary conditions was performed, with time steps of 2 fs. The PME method [64, 65] was used to calculate the Lennard-Jones and electrostatic interactions. A 10 Å atom-based cutoff was used to separate short-range and long-range interactions. The non-bonded pair list was updated every 10 time steps. Figure 8 displays a snapshop from the initial stage of the simulation.
Lignin Biosynthesis and Degradation – a Major Challenge
149
Fig. 8. Snapshop from the initial stage of the p-coumaryl – DPPC simulation. The monolignol is seen in ‘ball-and-stick’ image at the center of the bilayer
The reorientation of the lignol was analysed through the relative orientation of the C1-C4 axis and the axis formed by the mass centers of the two lipid bilayers. The angle between the two axes as a function of time is shown in Fig. 9. From the analysis it is clearly seen that the lignol very rapidly rotates/reorients to a situation where it is parallel to the norm of the bilayer (angle φ ≈ 180o), with the phenoxyl head pointing towards the lipid-water interface. Rotation of the entire molecule and its constituent groups were also investigated. It was concluded that the lignol rotates very slowly with time, that the molecule as such is rigid but that the two OH groups rotate freely. Diffusion of the p-coumaryl alcohol inside the lipid bilayer was analysed in terms of the Einstein relation [63], and showed that the rate of diffusion was essentially constant at 3 – 4x10-7 cm2/s during the entire 1500 ps simulation. The only exception was a segment in the time interval 250-500 ps from the start, when the monolignol rapidly diffused in the direction of the norm of the bilayer at a rate ca 10 times higher than during the rest of the simulation. According to previous experimental and theoretical studies the diffusion constants in lipid bilayers of solutes of similar size as here are on the order of 10-6 cm2/s, whereas larger, more complex solutes diffuse in lipid bilayers at rates of the order 10-7 – 10-8 cm2/s [67, 68, 69]. The present data is hence consistent with previous observations.
150
Bo Durbeej et al.
Fig. 9. Angle between the monolignol C1-C4 axis and the norm of the bilayer as function of time
Fig. 10. Diffusion movement of the p-coumaryl alcohol in DPPC bilayer, in (left) Y-Z direction and (right) X-Z direction, in 50ps intervals. Simulations start at the top of the figures. X and Y denote the plane and Z the norm of the bilayer
Lignin Biosynthesis and Degradation – a Major Challenge
151
The diffusion was also mapped in terms of the relative position of O4 (the phenoxylic oxygen) in the XZ and YZ planes every 50 ps of the simulation, Fig. 10. These maps display that the rapid diffusion movement is correlated with a fast motion in the Z-direction, i.e. along the norm of the bilayer. The movement corresponds to ca 15 Å distance, which places the monolignol in the polar head-group region of the phospholipids. The present data shows that the lignin monomers move much faster in the hydrophobic (inner) region of the bilayer than in the head group region with Dcenter/Dhead ≈ 10. The barrier to trans-membrane diffusion of monolignols thus appears to be the polar head group region – again consistently with other simulations of small solutes in lipid bilayers. Further studies of larger (di-, tri-) lignols, and of a higher concentration monolignols within the bilayer are currently under way. 4.1
Structural Features of Lignin Dimerization Products
Once the monolignols have entered the cell wall region, polymerization will take place. We have here investigated two different hypotheses for the initial stages of dimerisation, the thermodynamic and spin control hypetheses, and begin with an outline of the possible dimers that can be formed from coniferyl alcohol. Six different coniferyl alcohol dimerization products were considered (β-O4, β-5, β-β, β-1, 5-5, and 5-O4, see Fig. 5) in the study of lignin structure [24]. Standard bond lengths and angles were used to construct an initial conformational data set consisting of one unique starting geometry for 5-5 and 5-O4, threo and erythro β-O4 and β-1 diastereomers (Fig. 5), cis and trans β-5 stereoisomers, and several β-β stereoisomers with the Cβ hydrogens arranged either cis or trans. These geometries were then subjected to a systematic conformational search with respect to key dihedral angles, using the semiempirical PM3 method [70, 71]. The lowest-energy conformers from each search were identified and used as input in subsequent B3LYP-DFT calculations. The geometries were first optimized with the small 3-21G basis set, whereby only a minority of the conformers obtained at the PM3 level survived. The remaining structures were refined through geometry optimizations using the larger 6-31G(d,p) basis set. In order to reduce the computational effort, only those conformers within each dimerization product having a relative energy within 10 kcal mol-1 of the most stable one at the B3LYP/3-21G level were considered in these calculations. Altogether, geometry optimizations of thirty-two conformers (eight β-O4 conformers, five β-5, six β-β, five β-1, four 5-5, and four 5-O4 conformers, respectively) were carried out. Zero-point vibrational energy (ZPE) corrections were calculated with the 6-31G(d,p) basis set, also providing a control that the stationary points obtained were in fact true minima. Finally, better relative energies were evaluated using the larger 6311G(2df,p) basis set and the PCM solvent approximation. Figures 11 and 12 display the lowest-energy conformer of each dilignol obtained from the above calculations, and Tables 1 and 2 list key geometric parameters. A common feature of the lowest-energy conformers of β-O4, β-5, β-1, and 5-5 is the presence of intra-molecular hydrogen bonding that significantly stabilizes each of these structures. In the preferred erythro β-O4 conformer, the A-ring (see fig. 5) γhydroxyl hydrogen bonds to the B-ring 3-methoxy, and the A-ring α-hydroxyl is
152
Bo Durbeej et al.
C5 β O4
α
O4
β
β-5
β -O4
β β
β−β
C1
β
β-1
Fig. 11. Geometry-optimized structures of β-O4, β-5, β-β, and β-1 dilignols. Numerical values are listed in Tables 1 and 2
directed towards the B-ring ether oxygen. The most stable threo β-O4 conformer (not shown), in turn, has its A-ring α-hydroxyl hydrogen-bonded to the B-ring 3-methoxy and its A-ring γ-hydroxyl directed towards the B-ring ether oxygen. This conformer lies 6.5 kcal/mol higher in energy than the preferred erythro conformer, a result that confirms earlier experimental work showing that the erythro β-O4 is more stable than threo β-O4 [72].
Lignin Biosynthesis and Degradation – a Major Challenge
153
In the lowest-energy conformer of the β-5 (trans β-5) and β-1 dilignols, the intramolecular hydrogen bonding pattern is such that the A-ring γ-hydroxyl either acts as a hydrogen-bond donor to the B-ring γ-hydroxyl (β-5), or as a hydrogen-bond acceptor from the A-ring α-hydroxyl (β-1). The most stable 5-5 conformer, in turn, has the 4hydroxyl of one of the aromtic rings hydrogen-bonded to the 4-hydroxyl of the other ring. The lowest-energy β-β conformer has its Cβ hydrogens arranged in a cis conformation, and the methoxy and hydroxyl of the two 3-methoxy-4-hydroxyphenyl groups have similar orientations, being aligned in the plane of the corresponding aromatic ring (the C3-C4-O4-H4 and C2-C3-O3-CMe dihedrals are close to zero) with the hydroxyl directed towards and interacting with the lone electron pairs of the neighbouring methoxy oxygen. The A-ring 3-methoxy and 4-hydroxyl groups of the most stable conformer of β-O4, β-5, and β-1, as well as the B-ring 3-methoxy and 4hydroxyl groups of β-1 and 5-04, are aligned in a similar fashion. Given that the calculations yielded a couple of 5-O4 conformers with the B-ring 4-hydroxyl hydrogen-bonded to the A-ring 3-methoxy, it is somewhat surprising that the most stable 5-O4 conformer actually lacks an intra-molecular hydrogen bond.
C5 O4
C5
C5
5-5
5-O4
Fig. 12. Geometry-optimized structures of 5-5 and 5-O4 dilignols. Numerical values are listed in Tables 1 and 2
154
Bo Durbeej et al.
Table 1. Selected dihedral angles (in degrees) and zero-point corrected relative energies (in kcal mol-1) for the most stable conformer of β-O4, β-5, β-β, β-1, 5-5, and 5-O4
Relative Energy Dihedral Angles Cα-Cβ-C1′-C2′ Cα-Cβ-O4′-C4′ Cβ-Cα-C1-C2 Cβ-O4′-C4′- C5′ C3-C4-O4-H4 C3′-C4′-O4′-H4′ C2-C3-O3-CMe C2′-C3′-O3′CMe′ C3-C4-O4-C5′ C4-C5-C5′-C4′ C4-O4-C5′-C4′ Hα-Cα-Cβ-Hβ Hβ-Cβ-Cβ′-Hβ′
β-O4
β-5
β-β
β-1
5-5
5-O4
(11.6)
2.6
0.0
-
6.6
22.6
−126.1 −115.1 66.3 −0.7 −2.0 0.3
−105.5 77.5
−132.8, 56.8
86.8
0.1
−0.4, 0.5
0.3 0.5
−2.0, 1.7
2.0 1.5 4.1 7.6
144.2, 0.6 −0.7, −0.1
−1.3 110.2 −1.0 165.0
48.0 −145.4
−19.0
−78.2
−180.0
The dihedral angles are defined according to the atom identification scheme given in Figure 3 with primed atoms/methyl groups referring to B-ring atoms/methyl groups as defined in Figure 5. Since a distinction between the aromatic rings of β-β and 5-5 cannot be made, two values are reported for these. Values to the left in the corresponding column are associated with a certain ring, whereas values to the right are associated with the other ring. The relative energy of β-1 is omitted since β-1 has a different stochiometry. Table 2. Geometric parameters (in Å and degrees, respectively) for intra-molecular hydrogen bonds present in the most stable conformer of β-O4, β-5, β-1, and 5-5
Hydrogen Bond H⋅⋅⋅O Distance Bond Angle
β-O4
β-5
β-1
5-5
Oγ-Hγ⋅⋅⋅O3′ 1.99 163.8
Oγ-Hγ⋅⋅⋅Oγ ′ 2.34 160.9
Oα-Hα⋅⋅⋅Oγ 1.87 146.0
O4-H4⋅⋅⋅O4’ 1.78 154.0
The bond angles are defined according to the atom identification scheme given in Figure 3 with primed atoms referring to B-ring atoms as defined in Figure 5. 4.2
The Thermodynamic Control Hypothesis
In order to assess the validity of the thermodynamic control hypothesis as an explanation for the observed distribution of intermonomeric linkages, we have in Table 1 also listed the relative energies of the different dilignols. The energy of water – evaluated at the same level of theory – has been added to the energies of β-β, β-5, 55, and 5-O4, respectively. This is of course not entirely correct, but allows for an estimate of the stabilities of those dilignols relative to that of β-O4, as formation of the latter involves addition of a water molecule to the corresponding quinone methide. Even though factors such as type of solvent [73, 74, 75] and pH [73] certainly come into play, the major oxidative dimerization products in aqueous solution are β-5
Lignin Biosynthesis and Degradation – a Major Challenge
155
(50%), β-β (30%), β-O4 (20%), and, in small amounts, 5-5 [76]. This is in sharp contrast to the occurrence in natural lignin, in which β-O4 couplings are predominant [10]. From the data presented in Table 2, it is thus interesting to note that 5-O4 is the least stable dilignol, and that β-5 and β-β are the two most stable structures. The data is, however, somewhat inconclusive in that 5-5 lies 5.0 kcal mol-1 lower in energy than β-O4, which clearly contradicts the thermodynamic control hypothesis. The calculated energies furthermore suggest that β-β rather than β-5 is the predominant dimerization product. If the relative concentrations of the various dilignols follow a strict Boltzmann distribution, i.e., that it is only the relative product energies that control the formation, then the computed data listed in Table 2 would yield β-β (98.8%) and β-5 (1.2%) as the only products observed at room temperature. It is hence obvious that our results do not support the thermodynamic control hypothesis from a quantitative point of view. We nevertheless claim that one should not entirely dismiss this hypothesis on the basis of a comparison between our theoretically estimated relative concentrations and experimental data since small errors (~1-2 kcal mol-1) in the calculated relative energies of β-5 and β-β could have a dramatic effect on the estimated relative concentrations. Elder and Ede have on the basis of calculated heats of formation of various dilignols argued that their results suggest dimerization of monolignols to be thermodynamically controlled [34]. Their conclusion was later questioned by Houtman, who pointed at the fact that the 12 kcal mol-1 difference in heats of formation between the most stable (β-β) and the second most stable (β-5) dilignol would predict the relative concentration of β-β to be 100% at room temperature [75]. The semiempirical calculations performed by Elder and Ede [34] provided relative energies that differ quite substantially from those reported here, but the proposed order of thermodynamic stability – which at a qualitative level correlates with experimental data – agrees with our results. 4.3
The Spin Distribution Hypothesis
As pointed out in the introduction, all dilignols are phenols that – as the polymerization of lignin proceeds – also might undergo enzymatic dehydrogenation. The evaluation of the spin distribution hypothesis was therefore extended to include computations of spin densities in β-O4, β-5, β-β, β-1, 5-5, and 5-O4 dehydrogenated dilignols (DHDLs) [33]. Starting from the B3LYP/6-31G(d,p) geometry-optimized structures shown in Figures 11 and 12, an initial set of DHDL geometries was created by removal of a phenolic hydrogen from the corresponding closed-shell conformers. The resulting open-shell structures were then optimized at the same level, followed by computation of spin densities using the extended 6-311++G(2df,p) basis set and standard Mulliken population analysis. The β-β, 5-5, and β-1 dilignols all have two phenolic hydrogens. Therefore, abstraction of either of the two phenolic hydrogens was taken into account by initial geometry optimizations at the B3LYP/3-21G level. Only the most stable B3LYP/321G hydrogen abstraction product was then considered at the higher level of theory. The calculated spin densities are presented in Table 3. We note that our calculations on the coniferyl alcohol radical assign negative spins to the C2, C4, and C6 positions. The odd-alternant pattern with negative spins at those particular
156
Bo Durbeej et al.
positions is a well-characterized feature of phenyl radicals (cf. Refs [77, 78]). The calculations by Elder and Worley [36] showed that the atomic sites having most of the unpaired spin actually are those through which polymerization occurs, and that the spin densities of the atomic sites within the aromatic ring are higher than that of the phenolic oxygen. Our study verifies that the unpaired spin of the coniferyl alcohol radical exists primarily on atoms involved in bond formation during polymerization, but assigns the highest spin to the O4 position (i.e. the phenolic oxygen). This result agrees with previous findings for the tyrosyl radical, and is supported by experimental studies of similar systems (e.g. Ref [79], and references therein). The formation of dilignols in aqueous solution at room temperature typically yields 50% β-5, 30% β-β, 20% β-O4, and small amounts of 5-5 couplings [76]. The calculations on the coniferyl alcohol radical provide spin densities that, apart from distinguishing the atomic sites through which polymerization occur, correlate rather weakly with this distribution. Given the predominance of β-O4 couplings in natural lignin [10], it is however noteworthy that dehydrogenation of dilignols, in turn, also seems to produce radicals with most of the spin residing at the phenolic oxygen. For all DHDLs except 5-5, the calculations assign most of the spin to the O4 position. This result might partly explain the large percentage of β-O4 couplings in natural lignin. Table 3. Calculated spin distributions for coniferyl alcohol radical and DHDLs
Atom C1 C2 C3 C4 O4 C5 C6 Cα Cβ Cγ
Coniferyl
β-O4
β-5
β-β
β-1
5-5
O4-5
0.25 −0.10 0.22 −0.00 0.32 0.18 −0.08 −0.10 0.21 −0.01
0.22 −0.11 0.25 0.01 0.37 0.16 −0.04 0.01 0.01 0.01
0.29 −0.11 0.23 −0.02 0.36 0.22 −0.07 −0.00 0.00 −0.00
0.29 −0.12 0.24 −0.03 0.36 0.24 −0.09 0.01 0.02 0.01
0.30 −0.11 0.26 −0.04 0.36 0.22 −0.08 0.00 0.01 0.01
0.30 −0.10 0.23 −0.06 0.29 0.24 −0.10 −0.12 0.23 −0.01
0.28 −0.11 0.24 −0.09 0.30 0.23 −0.10 −0.11 0.23 0.00
The atoms are numbered according to the scheme given in Figure 3. The spin densities of the DHDLs are those of the atomic sites at the monolignol subunit from which the phenolic hydrogen has been abstracted. Without exception, the spin densities of the atomic sites on the second subunit are negligible.
5
Formation of the β-O4 Dimeric Linkage
As mentioned above, the distribution of dimeric linkages found in natural lignin differs considerably from that observed when letting lignin react in aqueous solution. The large fraction of unpaired spin at the O4 position of the dehydrogenated radicals may be one explanation for this effect, but also other mechanistic and environmental
Lignin Biosynthesis and Degradation – a Major Challenge
157
features can come into play. Our investigation of the mechanistic details of the formation of β-O4 linked dimeric lignin structures involves two alternative chemical reactions, as shown in Figure 13 [37]. The first reaction corresponds to the formation of a β-O4 linked QM from two coniferyl alcohol radicals. As such, this scheme represents a prototype lignin polymerization mechanism based on successive radicalradical additions. The second reaction corresponds to the addition of a coniferyl alcohol to its phenoxy radical, yielding a radical hereafter denoted QMR. This particular reaction constitutes an alternative ‘competing’ pathway to the formation of β-O4 linked dimeric lignin structures, and will below be assessed through a comparison between the calculated energetics of the two pathways. In order to reduce the computational effort, the 3-hydroxy-propen-1-yl tail of the RO4 radical was replaced by a hydrogen atom in the modeling of the reactions described in Figure 13. Given that the structural and electronic properties of the full β-O4 dilignols are similar to those of models lacking this particular side chain [24], we believe that this simplification is justified. In order to locate a transition state structure for a radical-radical addition reaction, whereby an energy barrier can be obtained, one should ideally employ a multiconfigurational ab initio method such as CASSCF [80] to account for non-dynamical electron correlation effects. It should therefore be noted that the B3LYP hybrid density functional method, while requiring considerably less of computational resources and thus being applicable to larger systems, analogously with most ab initio methods such as HF, MP2, QCISD and similar is a single-reference method. An approximation to the energy barrier for the formation of a β-O4 linked QM was obtained by constrained triplet state geometry optimizations in which the Cβ-O4 distance between the two radicals was held fixed at different values ranging from 2.52 to 1.72 Å. All other geometrical parameters were allowed to relax during the optimizations. For each of these geometries, the corresponding B3LYP/6-31G(d,p) singlet state energy – which should be higher than the triplet energy at large Cβ-O4 distances and lower at small distances – was subsequently evaluated, and the geometry at which the singlet and triplet surfaces intersect was identified (Cβ-O4 distance 2.25 Å). In addition, singlet and triplet B3LYP/6-311G(2df,p) and B3LYP/6311++G(2df,p) energy curves were computed using the B3LYP/6-31G(d,p) triplet state geometries. From the corresponding intersection points, an approximate energy barrier with respect to the isolated reactants could be obtained. The results from the above calculations are listed in Table 4 together with the corresponding reaction energies. Since we have not explicitly located a transition state structure, ZPE corrections have not been added to the presented energies. We observe that the energy barrier for the formation of QM is low (~2-5 kcal/mol), and that the reaction energy is negative by more than 20 kcal/mol. These results show that the process is both fast and thermodynamically favourable, and in agreement with results from studies of other radical-radical additions.
158
Bo Durbeej et al. H2COH
Formation of β-O4 linked QM
CH CH
H2COH
.
CH
CH
CH
HC
H2COH
H2COH
HC
OMe O
CH
+ OMe
OMe
OMe
O
.O
O
Rβ
RO4
QM H2COH CH
Formation of β-O4 linked QMR
H2COH
H2COH
CH
CH
CH
CH
CH
H2COH HC
OMe O
. CH
+
OH coniferyl alcohol
OMe
OMe
OMe
.O RO4
OH QMR
Fig. 13. Formation of β-O4 linked dimeric lignin structures
Lignin Biosynthesis and Degradation – a Major Challenge
159
Table 4. Energy barriers (∆E†) and reaction energies (∆E) for the formation of β-O4 linked QM (kcal/mol)
Level B3LYP/6-311G(2df,p) B3LYP/6311G(2df,p)/PCM B3LYP/6-311++G(2df,p)
∆E† 1.6-3.1 3.3-5.2
∆E −24.9 −23.0
3.6-5.2
−23.0
All calculations carried out using B3LYP/6-31G(d,p) optimized geometries. The energy barrier is calculated as ∆E† = E(S/T) − E(Rβ) − E(RO4), where E(S/T) is the energy at which the singlet and triplet energy curves intersect; the reaction energy is calculated as ∆E = E(QM) − E(Rβ) − E(RO4). The uncertainties in the calculated energy barriers reflect the uncertainties in E(S/T). PCM indicates that the polarized continuum model has been used.
RO4
O4
O4
Cβ coniferyl alcohol
Cβ
Transition state
Product
Fig. 14. Geometry-optimized structures of transition state and product for the formation of β-
O4 linked QMR. Numerical values are listed in Table 5
The geometry-optimized structure of the transition state for the formation of a βO4 linked QMR is shown in Figure 14 together with the corresponding product structure (both obtained at the B3LYP/6-31G(d,p) level). Relevant geometric parameters are given in Table 5, which also lists total atomic spin densities calculated from Mulliken population analysis at different levels of theory using the B3LYP/631G(d,p) geometries. Energy barriers and reaction energies are presented in Table 6. We observe that the Cβ-O4′ and C4′-O4′ bond lengths of the transition state equal 1.95 and 1.32 Å, respectively (cf. 1.48 and 1.38 Å in the QMR product). The C4′-O4′ bond thus displays partial double-bond character. This feature is the result of RO4 not being the only possible resonance structure of a coniferyl alcohol radical (Fig. 4). As the reaction proceeds, however, the unpaired spin is transferred from the RO4 moiety to the coniferyl alcohol moiety. The different resonance structures thereby lose significance, which provides an explanation for the fact that the C4′-O4′ bond length
160
Bo Durbeej et al.
of the product is 0.06 Å larger than that of the transition state. As expected, the CαCβ bond is stretched during the course of the reaction. The two structures furthermore differ in that the product is stabilized by an intra-molecular hydrogen bond between the γ-hydroxyl and the 3′-methoxy. This particular hydrogen bond is also present in the lowest-energy erythro conformer of β-O4 dilignols [24]. Table 5. Geometric parameters (in Å and degrees, respectively) for transition state (TS) and product (PROD) for the formation of β-O4 linked QMR
Geometric parameter Cβ-O4′ C4′-O4′ Cβ-C4′ Oγ-O3′ Hγ-O3′ Oγ-Hγ-O3′ Cα-Cβ Spin distribution Cα C6 C4 C2 C1 O4′ C1′
TS 1.95 1.32 2.87 2.93 2.20 131.0 1.40 I 0.41 0.14 0.13 0.09 −0.06 0.27 0.10
I/PCM 0.42 0.14 0.14 0.09 −0.06 0.26 0.09
PROD 1.48 1.38 2.47 2.86 1.97 151.6 1.49 II 0.43 0.16 0.15 0.07 −0.06 0.25 0.10
I 0.66 0.22 0.20 0.16 −0.12 0.03 −0.00
I/PCM 0.66 0.22 0.20 0.16 −0.12 0.03 −0.00
II 0.66 0.24 0.21 0.12 −0.11 0.02 −0.00
Primed atoms refer to atomic sites originally at RO4. I and II stand for B3LYP calculations using the 6-311G(2df,p) and 6-311++G(2df,p) basis sets, respectively. PCM indicates that the polarized continuum model has been used.
Table 6. ZPE-corrected (and uncorrected) energy barriers (∆E†) and reaction energies (∆E) for the formation of β-O4 linked QMR (kcal/mol)
∆E† 7.6 (6.7) 11.5 (10.6) 9.7 (8.8)
Level B3LYP/6-311G(2df,p) B3LYP/6-311G(2df,p)/PCM B3LYP/6-311++G(2df,p) All calculations carried optimized geometries.
out
using
∆E −1.1 (−3.5) 2.8 (0.4) 0.9 (−1.5)
B3LYP/6-31G(d,p)
Lignin Biosynthesis and Degradation – a Major Challenge
161
H2COH
H2COH
H2COH
CH
CH CH
CH
CH CH H2COH
H2COH
H2COH
CH
HC
CH
OMe
OMe
HC
MeO
O
O
CH
O
CH
+ OMe
OMe
OMe OH
O
OH coniferyl alcohol
β,γγ-bis-coniferyl ether guaiacylglycerol-β
QM
H2COH
H2COH
H2COH
CH
CH
CH
CH
CH CH
H2COH
H2COH
H2COH
CH
HC
CH
. CH
OMe
OMe O
HC
MeO O
O
CH
+ OMe
OMe
OMe
.O
OH
RO4
QMR
OH β,γγ-bis-coniferyl ether guaiacylglycerol-β
Fig. 15. Formation of guaiacylglycerol-β,γ-bis-coniferyl ether according to two different schemes
The calculated energy barrier for the formation of QMR equals 11.5 kcal/mol at the B3LYP/6-311G(2df,p)/PCM level (10.6 kcal/mol without ZPE corrections), and the corresponding reaction energy is slightly endothermic. From both a kinetic and thermodynamic point of view, this reaction is hence less favourable than the previous radical-radical addition (∆E† = 3.3-5.2 and ∆E = –23.0 kcal/mol at the B3LYP/6311G(2df,p)/ PCM level). This holds true also at the other levels of theory employed. Even though the difference in energy barrier between the two reactions according to transition state theory translates into reaction rates differing by several orders of magnitude, we believe that the possibility that QMRs are actually formed should not be discarded since ~8-12 kcal/mol clearly also constitutes a surmountable energy barrier under standard conditions of temperature and pressure. The fact that the
162
Bo Durbeej et al.
formation of QMR appears to be a more or less thermoneutral process further supports this view. From the calculated spin densities of the transition state structure, we note that – even though considerable amount of unpaired spin is assigned to the O4′ position – most of the spin resides at the coniferyl alcohol moiety. The spin distribution of QMR, in turn, shows that essentially all spin has been transferred to the coniferyl alcohol moiety upon completion of the reaction. This distribution reflects the fact that QMR has four different resonance structures with the unpaired electron assigned to either of the Cα, C2, C4, and C6 positions. Given the above spin distribution, it can thus be debated whether QMRs play an important role in lignin polymerization. Hypothetically, by considering the formation of the guaiacylglycerol-β,γ-bis-coniferyl ether trilignol from coniferyl alcohol and a β-O4 linked QM (Fig. 15) – which constitutes the simplest example of how lignin growth proceeds through the addition of phenols to QMs [7] – it however seems rather plausible that an alternative pathway to the formation of this trilignol would be through the addition of a coniferyl alcohol radical to a QMR (Fig. 15). This hypothesis is supported by the fact that the α-carbon of QMR carries the major part of the unpaired spin, and the results from the first section indicating that radical-radical additions within the context of lignin polymerization are favourable both in terms of energy barriers and reaction energies.
6
Concluding Remarks
In the present review we have tried to present an overview of the types of results that can be obtained using a range of different theoretical chemical approaches to address a complex (bio)chemical problem – namely that of lignin biosynthesis. From MD simulations of lignin monomers and their corresponding radicals in aqueous solution, it was concluded that the methoxy groups hinders hydrogen bonding interaction with the solvent at the phenolic end, and reduces the diffusion rate of the species. The ‘tail end’ of the molecules were however unaffected by the different substituent patterns and radical/non-radical form of the system. In a first attempt to address the question about the transport process of monolignols across the cell membrane, the behaviour of p-coumaryl alcohol was simulated in a DPPC + water environment. The data from these MD simulations revealed that the monolignol diffuses rapidly in the interior of the membrane, but will have difficulty in passing the polar head group region – in analogy with results on similar systems. The simulations also revealed a clear orientational preferrence of the lignol, with the phenol ring towards the head group region and the tail end towards the center of the bilayer. Using quantum chemical methods we then explored the relative stabilities of various dilignols, and tested the two dominating hypotheses regarding the observed product distribution in aqueous solution. It is concluded that, albeit the relative stabilities provide a good qualitative explanation for the observed distribution, the thermodynamic control hypothesis does not explain all details. Looking at the spin distributions of the dehydrogenated radicals, on the other hand, the odd-alternant pattern again yields increased radical character (and, hence, reactivity) at those
Lignin Biosynthesis and Degradation – a Major Challenge
163
positions known to be the sites of cross-link formation. The main fraction of the unpaired spin is however located on the phenolic oxygen (O4), and the spin control hypothesis can also not alone explain the observed product ratio in aqueous solution. Instead it appears that the elevated level of unpaired spin at O4 may be a contributing factor for the predominance of β-O4 cross-links in natural lignin. In the final section, we investigated two different mechanisms for the formation of the β-O4 dimers; a radical – radical and a radical – non-radical addition mechanism. The radical – radical addition mechanism was found to have a barrier of only a few kcal/mol, and be exothermic by 23 kcal/mol, whereas the radical – non-radical mechanism excerts a ca 10 kcal/mol high barrier to reaction and is essentially thermoneutral. Based on this data we conclude that the first mechanism is the most likely, albeit the second is also plausible under standard conditions. We ended the review with an outline of how the corresponding mechanisms may come into play in the formation of tri- and higher lignols.
Acknowledgements The Swedish Science Research Council (VR), the Carl Trygger Foundation and the Wood Ultrastructure Research Center (WURC) are gratefully acknowledged for financial support. We also acknowledge Dr. David van der Spoel and Prof. Aatto Laaksonen for valuable discussions.
References [1]
Newman, R. in Microfibril Angle In Wood. IAWA/IUFRO International Workshop on the Significance of Microfibril Angle to Wood Quality. Ed. B.G. Butterfield, Westport, New Zealand, p. 81, 1998. [2] Heyn, A.N.J., J. Ultrastruct. Res.1969, 26, 52. [3] Higuchi, T., Wood Sci. Technol. 1990, 24, 23. [4] Baucher, M. B. Monties, M. Van Montagu and W. Boerjan, Critical Reviews in Plant Sciences. 1998, 17, 125. [5] Cotê, W.A., Cellular Ultrastructure of Woody Plants. Syracuse Univ. Press, New York, Syracuse, 1965. [6] Weir, N.A., J. Arct and A. Ceccarelli, Polymer Degradation and Stability 1995, 47, 289, and references therein. [7] Freudenberg, K., Science 1965, 148, 595. [8] Higuchi, T., in Biosynthesis and Biodegradation of Wood Components. Ed. T. Higuchi. Academic Press, Orlando. p 141, 1985. [9] Davin, L.B., H.-B. Wang, A.L. Crowell, D.L. Bedgar, D.M. Martin, S. Sarkanen and N.G. Lewis, Science 1997, 275, 362. [10] Monties, B., in Methods in Plant Biochemistry: Plant Phenolics, Vol. 1. Eds. P.M. Dey, J.B. Harborne. Academic Press, New York. p 113, 1989. [11] Monties, B., in The Biochemistry of Plant Phenolics (Ann. Proc. Phytochem. Soc. Eur. Vol 25), p 161, 1985.
164
Bo Durbeej et al.
[12] Monties, B., Polym. Degrad. Stabil., 1998, 59, 53. [13] Sjöström, E., Wood Chemistry – Fundamentals and Applications, 2nd ed. Acad. Press, San Diego, CA, 1993. [14] Lundquist, K., in Methods in Lignin Chemistry. Eds. S.Y. Lin, C.W. Dence. Springer-Verlag, Berlin. p 242, 1992. [15] Robert, D., in Methods in Lignin Chemistry. Eds. S.Y. Lin, C.W. Dence. Springer-Verlag, Berlin. p 250, 1992. [16] Sakakibara, A., Wood Sci. Technol. 1980, 14, 89. [17] Stomberg, R. and K. Lundquist, Acta Chem. Scand. 1986, A40, 705. [18] Stomberg, R., M. Hauteville and K. Lundqvist, Acta Chem. Scand. 1988, B42, 697. [19] Elder, T.J., M.L. McKee and S.D. Worley, Holzforschung 1988, 42, 233. [20] Faulon, J.-L. and P.G. Hatcher, Energy and Fuels 1994, 8, 402. [21] Simon, J.P. and K.-E.L. Eriksson, Holzforschung 1995, 49, 429. [22] Simon, J.P. and K.-E.L. Eriksson, J. Mol. Struct. 1996, 384, 1. [23] Simon, J.P. and K.-E.L. Eriksson, Holzforschung 1998, 52, 287. [24] Durbeej, B. and L.A. Eriksson, Holzforschung, in press, 2002. [25] Glasser, W.G. and H.R. Glasser, Macromolecules 1974, 7, 17. [26] Glasser, W.G. and H.R. Glasser , Paperi Ja Puu 1981, 2, 71. [27] Glasser, W.G., H.R. Glasser and N. Morohoshi. Macromolecules 1981, 14, 253. [28] Lange, H., B. Wagner, J.F. Yan, E.W. Kaler and J.L. McCarthy, in Seventh Intl. Symposium on Wood and Pulping Chemistry, Vol 1 .Beijing. p 111, 1993. [29] Roussel, M.R. and C. Lim, Macromolecules 1995, 28, 370. [30] Roussel, M.R. and C. Lim, J. Comp. Chem. 1995, 16, 1181. [31] Jurasek, L., J. Pulp Paper Sci. 1995, 21, J274. [32] Jurasek, L., J. Pulp Paper Sci. 1996, 22, J376. [33] Durbeej, B. and L.A. Eriksson, in preparation. [34] Elder, T.J. and R.M. Ede, in Proceedings of the 8th International Symposium on Wood and Pulping Chemistry, Vol. 1. Gummerus Kirjapaino Oy, Helsinki. p 115, 1995. [35] Mårtensson, O. and G. Karlsson, Arkiv för Kemi 1968, 31, 5. [36] Elder, T.J. and S.D. Worley, Wood Sci. Technol. 1984, 18, 307. [37] Durbeej, B. and L.A. Eriksson, Holzforschung, in press, 2002.. [38] Wang, Y.-N. and L.A. Eriksson, submitted for publication. [39] Wang, Y.-N. And L. A. Eriksson, in preparation. [40] Onysko, K.A., Biotech. Adv. 1993, 11, 179. [41] Reid, I.D. and M.G. Paice, FEMS Microbiol. Rev. 1994, 13, 369. [42] Viikari, L., A. Kantelinen, J. Sundqvist and M. Linko, M., FEMS Microbiol. Rev. 1994, 13, 335. [43] Cornell, W.D., P. Cieplak, C.I. Bayly, I.R. Gould, K.M. Merz, D.M. Ferguson, D.C. Spellmeyer, T. Fox, J.W. Caldwell and P.A. Kollman, J. Am. Chem. Soc., 1995, 117, 5179. [44] Allinger, N.L., Y.H. Yuh, J.-H. Lii, J. Am. Chem. Soc. 1989, 111, 855. [45] Jorgensen, W.L., J. Chandrasekhar, J.D. Madura, R.W. Impey and M.L. Kelwin, J. Chem. Phys. 1983, 79, 926. [46] Weiner, S.J., P.A. Kollman, D.A. Case, U.C. Singh, C. Ghio, G. Alagona, S. Profeta and P. Weiner, J. Am. Chem. Soc. 1984, 106, 765.
Lignin Biosynthesis and Degradation – a Major Challenge
[47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80]
165
Smondyrev, A.M. and M.L. Berkowitz, J. Comp. Chem. 1999, 20, 531. Söderhäll, J.A. and A. Laaksonen, J. Phys. Chem. B 2001, 105, 9308. AMBER 5. Case, D.A., et al., University of California, San Francisco, 1987. Miertus, S., E. Scrocco and J. Tomasi, J. Chem Phys. 1981, 55, 117. Miertus, S. and J. Tomasi, Chem Phys. 1982, 65, 239. Cossi, M., V. Barone, R. Cammi, R. and J. Tomasi, Chem. Phys. Lett. 1996, 255, 327. Hohenberg, P. and W. Kohn, Phys. Rev. B 1964, 136, 864. Kohn, W. and L.J. Sham, Phys. Rev. A 1965, 140, 1133. Parr, R.G. and W. Yang, Density Functional Theory of Atoms and Molecules, Oxford Univ. Press: N.Y., 1989. Gross, E.K.U. and R.M. Dreizler, Density Functional Theory, Springer, 1992. Koch, W. and M.C. Holthausen, A Chemist’s Guide to Density Functional Theory, Wiley-VCH: Weinheim, 2001. Becke, A.D., J. Chem. Phys. 1993, 98, 5648. Lee, C., W. Yang and R.G. Parr, Phys. Rev. 1988, B37, 785. Stevens, P.J., F.J. Devlin, C.F. Chabalowski and M.J. Frisch, J. Phys. Chem. 1994, 98, 11623. Gaussian 98, rev A7-A11. Frisch, M.J. et al. Gaussian Inc., Pittsburgh, PA 1998. Berendsen, H.J.C., J.P.M. Postma, W.F. Gunsteren, A. DiNola and J.R. Haak, J. Chem. Phys. 1984, 81, 3684. Allen, M.P. and D.J. Tildesley, Computer Simulation of Liquids. Oxford Science, Oxford, 1987. Darden, T., D. York and L.G. Pedersen, J. Chem. Phys. 1993, 98, 10089. Pedersen, L.G., J. Chem. Phys. 1995, 103, 3668. Berger, O., O. Edholm and F. Jähnig, Biophys. J. 1997, 72, 2002. Alper, H.E. and T.R. Stouch, J. Phys. Chem. 1995, 99, 5724. Mason, R.P. and D.W. Chester, Biophys. J. 1989, 56, 1193. Wu, E.-S., K. Jacobson and D. Papahadjopoulos, Biochemistry. 1977, 16, 3936. Stewart, J.J.P, J. Comp. Chem. 1989, 10, 209. Stewart, J.J.P, J. Comp. Chem. 1989, 10, 221. Miksche, G., Acta Chem. Scand. 1972, 26, 4137. Chioccara, F., S. Poli, B. Rindone, T. Pilati, G. Brunow, P. Pietikäinen and H. Setälä, Acta Chem. Scand. 1993, 47, 610. Terashima, N. and R.H. Atalla, in Proceedings of the 8th International Symposium on Wood and Pulping Chemistry, Vol. 1. Gummerus Kirjapaino Oy, Helsinki. p 69, 1995. Houtman, C.J., Holzforschung 1999, 53, 585. Tanahashi, M., H. Takeuchi and T. Higuchi, Wood Res. 1976, 61, 44. Himo, F., A. Gräslund and L.A. Eriksson, Biophys. J. 1997, 72, 1556. Himo, F., G.T. Babcock and L.A. Eriksson, Chem. Phys. Lett. 1999, 313, 374. Gräslund, A. and M. Sahlin, Annu. Rev. Biophys. Biomol. Struct. 1996, 25, 259. Roos, B., in Advanced Chemistry and Physics Vol. 69 (Ab Initio Methods in Quantum Chemistry, Part II). Ed. K.P. Lawley. Wiley, New York. p 399, 1987.
High Performance Computing in Electron Microscope Tomography of Complex Biological Structures Jos´e J. Fern´ andez1 , Albert F. Lawrence2, Javier Roca1 , Inmaculada Garc´ıa1, Mark H. Ellisman2 , and Jos´e M. Carazo3 1
2
Dpt. Arquitectura de Computadores, Universidad de Almer´ıa 04120 Almer´ıa, Spain National Center for Microscopy and Imaging, University of California San Diego, La Jolla, CA 92093-0608, USA 3 Biocomputing Unit. Centro Nacional de Biotecnolog´ıa Universidad Aut´ onoma, 28049 Madrid, Spain [email protected], [email protected], [email protected] [email protected], [email protected], [email protected]
Abstract. Series expansion reconstruction methods using smooth basis functions are evaluated in the framework of electron tomography of complex biological specimens, with a special emphasis upon the computational perspective. Efficient iterative algorithms that are characterized by a fast convergence rate have been used to tackle the image reconstruction problem. The use of smooth basis functions provides the reconstruction algorithms with an implicit regularization mechanism, very appropriate for noisy conditions. High Performance Computing (HPC) techniques have been applied so as to face the computational requirements demanded by the reconstruction of large volumes. An efficient domain decomposition scheme has been devised that leads to a parallel approach with capabilities of interprocessor communication latency hiding. Comparisons with Weighted BackProjection (WBP), the standard method in the field, are presented in terms of computational demands as well as in reconstruction quality under highly noisy conditions. The combination of efficient iterative algorithms and HPC techniques have proved to be well suited for the reconstruction of large biological specimens in electron tomography, yielding solutions in reasonable computational times. Keywords: Electron Tomography; High Performance Computing; Iterative Reconstruction Algorithms; Parallel Computing
1
Introduction
Electron microscopy is central to the study of many structural problems in disciplines related to Biosciences. In concrete, electron microscopy together with sophisticated image processing and three-dimensional (3D) reconstruction techniques allow to obtain quantitative structural information about the 3D strucJ.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 166–180, 2003. c Springer-Verlag Berlin Heidelberg 2003
High Performance Computing in Electron Microscope Tomography
167
ture of biological specimens [1, 2]. Obtaining this information is essential for interpreting the biological function of the specimens. In contrast to the conventional instrument, high-voltage electron microscopes (HVEMs) are able to image relatively thick specimens that contain substantial 3D structure. Electron Tomography makes it possible the determination and the analysis of their complex 3D structure from a set of projection HVEM images acquired from different orientations by tilting the specimen, typically following single-tilt axis [2] or, lately, double-tilt axis [3], geometries. Rigorous structural analysis require that the tomographic image acquisition and the subsequent reconstructions should have sufficient resolution for a proper interpretation and measurement of the structural features. As a consequence, Electron Tomography of large complex biological specimens requires large projection HVEM images (typically from 256×256 up to 1024×1024 pixels) and provides large volumes, which involves an intensive use of computational resources and considerable processing time. High Performance Computing (HPC) then arises as the appropriate tool to face those high computational demands by using parallel computing on supercomputers or networks of workstations. As far as the reconstruction methods are concerned, Weighted BackProjection (WBP) [1] may be considered as the standard method in electron tomography of large specimens. The relevance of WBP in this field mainly stems from the computational simplicity of the method (O(N 3 ), where N 3 is the number of voxels of the volume). However, WBP exhibits the strong disadvantages that (i) the results may be heavily affected by the limited tilt capabilities of the HVEM, and (ii) WBP does not implicitly take into account the noise conditions. As a consequence, a posteriori regularization techniques (such as low-pass filtering) may be needed to attenuate the effects of the noise. Series expansion reconstruction methods constitute one of the main alternatives to WBP to image reconstruction. Despite their potential advantages [4, 5], these methods still have not been extensively used in the field of electron tomography due to their high computational costs (O(2KN 3 V 2 ), where K is the number of iterations, V is the extension of the volume elements, and N is the same as above). One the main advantages of these methods is the flexibility to represent the volume elements by means of basis functions different from the traditional voxels. During the nineties [6, 7], spherically symmetric volume elements (blobs) have been thoroughly investigated and, as a consequence, the conclusion that blobs yield better reconstructions than voxels has been drawn in Medicine [8, 9] and Electron Microscopy [4, 5]. This work addresses the application of series expansion methods using blobs as basis functions in electron tomography of complex biological specimens from a computational throughput perspective. The aim is to show that blob-based series expansion methods may be so efficient that may constitute real alternatives to WBP as far as the computation time is concerned. In that sense, this work contributes (i) the use of very recent series expansion methods [10, 11] (component averaging methods) characterized by a really fast convergence, achieving leastsquares solutions in a very few iterations, and (ii) the use of HPC techniques,
168
Jos´e J. Fern´ andez et al.
which allows to determine the 3D structure of large volumes in reasonable computation time with either reconstruction method, taking advantage of parallel computer architectures. On the other hand, the use of blob basis functions provides the series expansion methods with an implicit regularization mechanism which makes them better suited for noisy conditions than WBP. This work has been structured into the following: Section 2 reviews iterative reconstruction methods. Section 3 is devoted to the HPC approach devised for this application. Section 4 then presents the experimental results, in terms of computation time and speedup rate, that have been obtained. Also the experimental results from the application of the approach to real biological data are shown, for quality comparison purposes. Finally, the results are discussed in Section 5.
2 2.1
Iterative Image Reconstruction Methods Series Expansion Methods
Series expansion reconstruction methods assume that the 3D object or function f to be reconstructed can be approximated by a linear combination of a finite set of known and fixed basis functions bj f (r, φ1 , φ2 ) ≈
J
xj bj (r, φ1 , φ2 )
(1)
j=1
(where (r, φ1 , φ2 ) are spherical coordinates) and that the aim is to estimate the unknowns xj . These methods also assume an image formation model where the measurements depend linearly on the object in such a way that: J yi ≈ li,j xj (2) j=1
where yi denotes the ith measurement of f and li,j the value of the ith projection of the jth basis function. Under those assumptions, the image reconstruction problem can be modeled as the inverse problem of estimating the xj ’s from the yi ’s by solving the system of approximate equations. Algebraic Reconstruction Techniques (ART) constitute a well known family of iterative algorithms to solve such systems [12]. Series expansion methods exhibit some advantages over WBP: First, the flexibility in the spatial relationships between the object to be reconstructed and the measurements taken, and second, the possibility of incorporate spatial constraints. The main drawbacks of series expansion methods are (i) their high computational demands and (ii) the need of parameter optimization to properly tune the model.
High Performance Computing in Electron Microscope Tomography
2.2
169
Component Averaging Methods
Component averaging methods have recently arisen [10, 11] as efficient iterative algorithms for solving large and sparse systems of linear equations. In essence, these methods have been derived from ART methods [12], with the important novelty of a weighting related to the sparsity of the system which allows a convergence rate that may be far superior to the ART methods, specially at the early iteration steps. Assuming that the whole set of equations in the linear system (Eq. (2)) may be subdivided into B blocks each of size S, a generalized version of component averaging methods can be described via its iterative step from the k-th estimate to the (k + 1)-th estimate by: = xkj + λk xk+1 j
S yi − Jv=1 li,v xkv li,j J b 2 s=1 v=1 sv (li,v )
(3)
where: λk denotes the relaxation parameter. b = (k mod B), and denotes the index of the block. i = bS + s, and represents the i-th equation of the whole system, and sbv denotes the number of times that the component xv of the volume contributes with nonzero value to the equations in the b-th block. The processing of all the equations in one of the blocks produces a new estimate. And one iteration of the algorithm involves processing all the blocks. This technique produces iterates which converge to a weighted least squares solution of the system of equations provided that the relaxation parameters are within a certain range and the system is consistent [11]. The efficiency of component averaging methods stems from the explicit use of the sparsity of the system, represented by the sbv term in Eq. (3). Component averaging methods, as ART methods, can be classified into the following categories as a function of the number of blocks involved: Sequential. This method cycles through the equation one-by-one producing consecutive estimates (S = 1). This method exactly matches the well-known row-action ART method [12] and is characterized by a fast convergence as long as relaxation factors are optimized. Simultaneous. This method, known simply as “component averaging” (CAV) [10], uses only one block (B = 1), considering all equations in the system in every iterative step. This method is inherently parallel, in the sense that every equation can be processed independently from the others. However, it is characterized, in general, by a slow convergence rate. Block-Iterative. Block-Iterative Component Averaging methods (BICAV) represent the general case. In essence, these methods sequentially cycle blockby-block, and every block is processed in a simultaneous way. BICAV exhibits an initial convergence rate significantly superior to CAV and on a par with row-action ART methods, provided that the block size and relaxation parameter are optimized. BICAV methods (with S > 1) methods also exhibit an inherent nature for parallelization.
170
2.3
Jos´e J. Fern´ andez et al.
Basis Functions
In the field of image reconstruction, the choice of the set of basis functions to represent the object to be reconstructed greatly influences the result of the algorithm [6]. Spherically symmetric volume elements (blobs) with smooth transition to zero have been thoroughly investigated [6, 7] as alternatives to voxels for image representation, concluding that the properties of blobs make them well suited for representing natural structures of all physical sizes. The use of blobs basis functions provides the reconstruction algorithm with an implicit regularization mechanism. In that sense, blobs are specially suited for working under noisy conditions, yielding smoother reconstructions where artifacts and noise are reduced with relatively unimpaired resolution. Specifically in electron tomography, the potential of blob-based iterative reconstruction algorithms with respect to WBP were already highlighted by means of an objective task-oriented comparison methodology [4]. Blobs are a generalization of a well-known class of window functions in digital signal processing called Kaiser-Bessel [6]. Blobs are spatially limited and, in practice, can also be considered band-limited. The shape of the blob is controlled by three parameters: the radius, the differentiability order and the density drop-off. The appropriate selection of them is highly important. The blob full width at half maximum (FWHM), determined by the parameters, is chosen on the basis of a compromise between the resolution desired and the data noise suppression: narrower blobs provide better resolution, wider blobs allow better noise suppression. For a detailed description of blobs, refer to [6]. The basis functions in Eq. (1) are shifted versions of the selected blob arranged on a simple cubic grid, which is the same grid as the one used with voxels, with the important difference that blobs have an overlapping nature. Due to this overlapping property, the arrangement of blobs covers the space with a pseudocontinuous density distribution (see Fig. 1) very suitable for image representation. On the contrary, voxels are only capable of modeling the structure by non-overlapping density cubes which involve certain spatial discontinuities.
3
High Performance Computing in Electron Tomography
Parallel Computing has been widely investigated for many years as a means of providing high-performance computational facilities for large-scale and grandchallenge applications. In the field of electron tomography of large specimens, parallel computing has already been applied to face the computational demands required to reconstruct huge volumes by means of WBP as well as voxel-based iterative methods [2]. The use of voxels as basis functions makes the single-tilt axis reconstruction algorithms relatively straightforward to implement on massively parallel supercomputers. The reconstruction of each of the one-voxel-thick slices orthogonal to the tilt axis of the volume is assigned to an individual node on the parallel computer. In that sense, this is an example of an embarrassingly parallel application (see [13] for a revision of the most important concepts
High Performance Computing in Electron Microscope Tomography
171
Fig. 1. Image representation with blobs. Left: the single-tilt axis acquisition geometry is sketched. The slices are those one-voxel-thick planes orthogonal to the tilt axis. Center: depicts the slices by means of columns, and the circle represents a generic blob which, in general, extends beyond the slice where it is located. Right: Blobs create a pseudo-continuous 3D density distribution representing the structure in parallel computing) that is no longer valid for blob-based methodologies, as will be shown below. For the blob case, more effort is needed for a proper data decomposition and distribution across the nodes. 3.1
Data Decomposition
The single-tilt axis data acquisition geometries typically used in electron tomography allow the application of the Single-Program Multiple-Data (SPMD) computational model for parallel computing. In the SPMD model, all the nodes in the parallel computer execute the same program, but for a data subdomain. Single-tilt axis geometries allow a data decomposition consisting of dividing the whole volume to be reconstructed into slabs of slices orthogonal to the tilt axis. The SPMD model then involves that the task to be carried out by every node is to reconstruct its own slab. Those slabs of slices would be independent if voxel basis functions were used. However, due to their overlapping nature, the use of blobs as basis functions makes the slices, and consequently the slabs, inter-dependent (see Fig. 1). Therefore, the nodes in the parallel computer have to receive a slab composed of its corresponding subdomain together with additional redundant slices from the neighbor nodes. The number of redundant slices depends on the blob extension. Fig. 2(a) shows an scheme of the data decomposition.
172
Jos´e J. Fern´ andez et al.
(a)
(b)
Fig. 2. Data decomposition. (a) Every node in the parallel system receives a slab including unique slices (light-gray), and additional redundant slices (dark-gray) according to the blob extension. (b) Classification of the slices in the slab The slices in the slab received by a given node are classified into the following categories (see scheme in Fig. 2(b)): Halo These slices are only needed by the node to reconstruct some of its other slices. They are the redundant slices mentioned above, and are located at the extremes of the slab. Halo slices are coming from neighbor nodes, where they are reconstructed. Unique These slices are to be reconstructed by the node. In the reconstruction process, information from neighbor slices is used. These slices are further divided into the following subcategories: Edge Slices that require information from the halo slices coming from the neighbor node. Own Slices that do not require any information from halo slices. As a result, these slices are independent from those in the neighbor nodes. It should be noted that edge slices in a slab are halo slices in a neighbor slab. 3.2
The Parallel Iterative Reconstruction Method
In this work, the BICAV (with S > 1) and CAV methods have been parallelized following the SPMD model and the data decomposition just described. The row-action version of component averaging methods has been discarded since the BICAV method yields a convergence rate on-a-par with it but with better warranties of speedup due to its inherent parallel nature. Conceptually, BICAV and CAV reconstruction algorithms may be decomposed into three subsequent stages: (i) Computation of the forward-projection of the model; (ii) Computation of the error between the experimental and the calculated projections; and (iii) refinement of the model by means of backprojection
High Performance Computing in Electron Microscope Tomography
173
Node i
Volume Initialization
Forward Projection
Halo Unique
Volume Refinement
Halo
s l i c e s
Edge Own Edge
Compute error with experimental projections
Communications
Node i+1
Communication of differences
Halo Unique
s l i c e s
Error BackProjection Communication of reconstructed slices
New Block
Halo
Edge Own Edge
New Iteration
(a)
(b)
Fig. 3. The Parallel Iterative Reconstruction Method. (a) Flow chart of the iterative reconstruction algorithm, including communication/synchronization points. (b) Communications in the parallel algorithm
of the error. Those stages can be easily identified in Eq. (3). Those reconstruction algorithms iteratively pass through those stages for every block of equations and for every iteration, as sketched in Fig. 3(a). Initially, the model may be set up to zero, constant value or even from the result of another reconstruction method. The interdependence among neighbor slices due to the blob extension implies that, in order to compute either the forward-projection or the error backprojection for a given slab, there has to be a proper exchange of information between neighbor nodes. Specifically, updated halo slices are required for a correct forward-projection of the edge slices. On the other hand, updated halo error differences are needed for a proper error backprojection of the edge slices. The need of communication between neighbor nodes for a mutual update of halo slices is clear. The flow chart in Fig. 3(a) shows a scheme of the iterative algorithm, pointing out the communication points: Just before and after the error backprojection. Fig. 3(b) also shows another scheme depicting the slices involved in the communications: halo slices are updated with edge slices from the neighbor node. Our parallel SPMD approach then allows all the nodes in the parallel computer to independently progress with their own slab of slices. Notwithstanding this fact, there are two implicit synchronization points in every pass of the algorithm in which the nodes have to wait for the neighbors. Those implicit synchronization points are the communication points just described. In any parallelization project where communication between nodes is involved, latency hiding becomes an issue. That term stands for overlapping communication and computation so as to keep the processor busy while waiting the communications to be completed. In this work, an approach that further exploits the data decomposition has been devised for latency hiding. In essence,
174
Jos´e J. Fern´ andez et al.
Fig. 4. Latency hiding: Overlapping communication and computation. The boxes represent stages of the iterative algorithm, and the discontinuous lines denote the transmission of the data already processed. The latency hiding is applied for both communication points in the parallel algorithm: Just before and after the error backprojection stage
the approach is based on ordering the way the slices are processed between communication points. Fig. 4 sketches this approach. First of all, the left edge slices are processed, and they are sent as soon as they are ready. The communication of left edge slices is then overlapped with the processing of right slices. Similarly, the communication of right edge slices is overlapped with the processing of own slices. This strategy is applied to both communication points, just before and after the error backprojection stage. On the other hand, the ordered way the nodes communicate with each other also makes this parallel approach deadlock-free.
4
Results
The experiments that have been carried out had a two-fold aim. First, the effective speedup and the computation times that our parallel approach yields were computed so as to evaluate its efficiency. Second, we intended to do a comparison between WBP and the component averaging methods in terms of the quality of the reconstructions when applied to real data under extremely noisy conditions.
High Performance Computing in Electron Microscope Tomography
175
For both purposes, we tested WBP, CAV and BICAV algorithms under different parameters that proved to be representative [6, 7, 9, 10, 11]: – block sizes for component averaging methods: 1, 10, . . . , 70. We have considered that the block size should be multiple of the size of the projection images so that all the pixels of the same image belong to the same block. – blob parameters. We have tested different blob radii: 1.5, 2.0, 2.25, 2.795. Those values represent a spectrum of different resolution properties. Those four blobs have different FWHMs: 1.20, 1.33, 2.15 and 2.45, respectively, relative to the sampling interval. Consequently, those blobs have a direct influence on the maximum resolution attainable in the reconstructions. – different values for the relaxation factor. The software developed for this application is based on C programming language and the MPI (Message-Passing Interface) library. The portability of the software has been tested in a number of computer platforms. 4.1
Evaluation of the Parallel Approach
The performance results that will be described and analyzed here were measured in the cluster of workstations at the NCMIR (National Center for Microscopy and Imaging Research). Such a cluster consists of 30 computers (uni- and dualPentium III) which involve 50 processors, and a total of 17 GBytes of RAM memory distributed across them. The computers in the cluster are switched by a fast ethernet network. The efficiency of any parallelization approach is usually evaluated in terms of the speedup [13], defined as the ratio between the computation time required for the application to be executed in a single-processor environment and the corresponding time in the parallel system. Fig. 5(a) shows the speedup obtained for the component averaging methods in reconstructing a 512×512×512 volume from 70 projection images using blob radius of 2.0 (standard blob) and 2.795 (wide blob). Specifically, we have computed the speedup for the extreme cases of BICAV: BICAV using 70 blocks and CAV (equivalent to BICAV using only one block). In Fig. 5(a) only one curve for CAV is presented because the speedup curves that have been obtained for both the standard and the wide blob present negligible differences (lower than 0.01%). In general, the speedup curves in Fig. 5(a) exhibit a global behavior that approaches the linear speedup. Still, CAV yields better speedup rates than BICAV. The difference between CAV and BICAV in this experiment is that the amount of communications in BICAV is scaled, with respect to CAV, by the number of blocks (70 in this case). In theory, the approach for overlapping communication and computation that we have developed here should provide latency hiding. Consequently, in principle, there should be no significant differences in the speedup rates. We speculate that the causes underlying those differences are related to the communication startup time involved in any communication step and which is not avoidable for any latency hiding approach. As the number
176
Jos´e J. Fern´ andez et al.
of processors in the system increases, the time needed for all the processors to start the communication is longer, possibly due to the network overhead. This hypothesis is justified by the fact that the reconstruction algorithm implemented here works synchronously. Finally, it is clearly appreciated in Fig. 5(a) that there seems to be an inflexion point in all the curves when using around 20 processors. From 20 processors on, all the speedup curves follows a slope lower than the one exhibited for less processors, progressively moving away from the linear speedup. This behavior still has to be further investigated. On the other hand, in order to compare component averaging methods with WBP in terms of the computational burden, we have measured the time required for every iteration, using different numbers of blocks for the BICAV methods. We have taken such measures for different reconstruction sizes (256×256×256, 512×512×512, 1024×1024×200), different number of tilt angles (61 or 70) and different blob sizes, obtaining similar relative behaviors. Fig. 5(b) shows the results obtained for a 512×512×512 reconstruction from 70 projections, using the standard blob (radius 2.0, bars in light-gray) and wide blob (radius 2.795, bars in dark-gray). In that figure, it is clearly appreciated that WBP is the lowest resources-consuming method, since it requires 235 seconds for the whole reconstruction. However, one iteration of the component averaging methods only requires a little bit more time (324-358 seconds and 438-496 seconds for blob with radius, respectively, 2.0 and 2.795). Taking into account that these iterative methods are really efficient and yield suitable solutions after a few iterations (in a range usually between 1 and 10), these computation times make component averaging methods real alternatives to WBP, as opposed to the traditional iterative ART methods. Fig. 5(b) shows that, in iterative methods, the time per iteration is slightly larger as the number of blocks increases (from 324/438 seconds in CAV to 358/496 seconds in BICAV with 70 blocks, using a blob with radius 2.0/2.795, respectively). As happened with the speedup, we speculate that this behavior may be caused by the communication startup times. 4.2
Application to Electron Tomography of Real Mitochondria Data
In this work, we have applied the BICAV and WBP reconstruction methods to real mitochondria data obtained from HVEM and prepared using photooxidation procedures, which are characterized by the low contrast and the extremely low signal to noise ratio (approaching SNR = 1.0) exhibited by the images. Seventy projection images (with tilt angles in the range [−70, +68]) were combined to obtain the reconstructions. The projection images were 1024×480, and the volume 1024×480×256. We have tested the algorithms under different conditions, but here only the most significant results are presented. A montage showing one z-section of the volume reconstructed with the different methods is presented in Fig. 6. Fig. 6(a) shows the result coming from WBP. Fig. 6(b) shows the reconstruction obtained from BICAV with 70 blocks
High Performance Computing in Electron Microscope Tomography
177
60 linear speedup
50
CAV 40
BICAV BICAV (wide blob)
30
20
10
0
0
10
20
30
40
(a)
50
60
(b)
Fig. 5. Evaluation of the parallel approach. The measures were obtained from reconstruction processes of a 512×512×512 volume from 70 projection images using blob radius of 2.0 (standard blob) and 2.795 (wide blob). (a) Speedup exhibited by the component averaging methods (CAV and BICAV with 70 blocks). (b) Computation times for component averaging methods and WBP
after 4 iterations with a relaxation factor of 1.0 and a blob with radius 2.0. We have also made tests with CAV, but the solutions are still blurred after 30-50 iterations, due to its relatively slow convergence rate compared to BICAV.
(a)
(b)
Fig. 6. Comparison of WBP and component averaging methods in an experimental application. One of the slices along the Z axis is shown. (a) Result from WBP. (b) Result from BICAV with 70 blocks after 4 iterations
Fig. 6 clearly shows that blob-based BICAV yields a solution much cleaner than WBP and, moreover, at the same resolution level. The excellent behavior exhibited by BICAV under so noisy situations comes from the regularization nature of blobs. However, WBP provides a “noisy” solution due to the high noise level in the experimental projection images. We have also tested wider blobs in the BICAV methods, and the results prove to be smoother than those presented.
178
Jos´e J. Fern´ andez et al.
However, using wider blobs involves further loss of resolution, which may not be desirable from the point of view of the biologist who wants to measure structural features in the reconstructions. Regarding the computation times, the reconstructions were done in the cluster of workstations at the NCMIR, using 50 processors. For the results shown in Fig. 6, WBP took around 200 seconds of computation time to obtain the solution, whereas BICAV took around 1300 seconds (which involves around 325 seconds per iteration).
5
Discussion and Conclusion
In this work we have analyzed the application of blob-based series expansion methods in electron tomography of complex biological specimens, with a special emphasis on the computational perspective. First, we have made use of efficient iterative methods to tackle the problem of image reconstruction. On the other hand, high performance computing (HPC) techniques have been applied so as to face the high computational demands and take advantage of parallel systems. A parallel approach for the iterative algorithms has been devised, exhibiting a speedup nearly linear with the number of processors in the system. This approach has also proved to be very efficient to deal with the communications among the processors. The results indicate that the latency due to communications is almost completely hidden by overlapping computation and communication. This parallel strategy allows the iterative methods to take between 5 and 8 computation minutes per iteration in the reconstruction of a 512×512×512 volume. The parallel approach is also exploited for WBP in such a way that a result of the same size is obtained after nearly 4 computation minutes. In this way, High Performance Computing is making it possible to afford “grand challenge” applications (e.g., reconstructions of 2048×2048×2048) currently unapproachable by uni-processor systems due to the computational resources requirements. The new iterative reconstruction methods we have applied here are very efficient, providing least squares solutions in a very few of iterations. Specifically, BICAV with a large number of blocks yields good reconstructions in a number of iterations in the range [1,10], depending on the parameters. This implies that BICAV could produce a solution after a computation time of 5-8 minutes for a 512×512×512 volume, compared to 4 minutes needed by WBP. On the other hand, the use of blobs has a two-fold benefit. First, blobs are better than voxels from the computational point of view, since they allow the use of look-up tables to speed up the forward- and backward- projection stages. Second, blobs provide the reconstruction algorithms with an implicit regularization mechanism which makes them well suited for noisy environments. As a consequence, the solutions yielded by blob-based iterative methods are smoother than those by WBP, but with relatively unimpaired resolution. In particular, under extremely noisy conditions, this type of algorithms clearly outperform WBP. In those situations, WBP would require a strong low-pass filtering postreconstruction stage which would limit the maximum resolution attainable.
High Performance Computing in Electron Microscope Tomography
179
Regularized iterative reconstruction methods have additional and interesting implications in data post-processing, particularly the segmentation of the structural components from the reconstruction. The results yielded by regularized methods usually present smoothly varying densities that are specially propitious for automatic segmentation techniques. Finally, this work has shown that the combination of efficient iterative methods and HPC is well suited to tackle the reconstruction of large biological specimens in electron tomography, yielding solutions in reasonable computation times.
Acknowledgments The authors wish to thank Dr. G. Perkins who kindly provided the real mitochondria data and contributed with valuable comments on the reconstructions. The authors also wish to thank C.O.S Sorzano and J.G. Donaire for fruitful discussions during this work. This work has been partially supported through grants from the Spanish CICYT TIC99-0361, TIC2002-00228, BIO98-0761 and BIO2001-1237, from the Spanish MECD PR2001-0110, from the Commission for Cultural, Educational and Scientific Exchange between USA and Spain, grant #99109. This work was also supported by the NIH/National Center for Research Resources through grants P41-RR08605 and P41-RR04050. Some of this work was performed using NSF-NPACI facilities under grant CISE NSF-ASC 97-5249.
References [1] J. Frank, editor. Electron Tomography. Three-Dimensional Imaging with the Transmission Electron Microscope. Plenum Press, 1992. 167 [2] G. A. Perkins, C. W. Renken, J. Y. Song, T. G. Frey, S. J. Young, S. Lamont, M. E. Martone, S. Lindsey, and M. H. Ellisman. Electron tomography of large, multicomponent biological structures. J. Struc. Biol., 120:219–227, 1997. 167, 170 [3] D. N. Mastronarde. Dual-axis tomography: An approach with alignment methods that preserve resolution. J. Struct. Biol., 120:343–352, 1997. 167 [4] R. Marabini, E. Rietzel, E. Schroeder, G. T. Herman, and J. M. Carazo. Threedimensional reconstruction from reduced sets of very noisy images acquired following a single-axis tilt schema. J. Struct. Biol., 120:363–371, 1997. 167, 170 [5] R. Marabini, G. T. Herman, and J. M. Carazo. 3D reconstruction in electron microscopy using art with smooth spherically symmetric volume elements (blobs). Ultramicroscopy, 72:53–56, 1998. 167 [6] R. M. Lewitt. Alternatives to voxels for image representation in iterative reconstruction algorithms. Phys. Med. Biol., 37:705–716, 1992. 167, 170, 175 [7] S. Matej, R. M. Lewitt, and G. T. Herman. Practical considerations for 3-d image reconstruction using spherically symmetric volume elements. IEEE Trans. Med. Imag., 15:68–78, 1996. 167, 170, 175 [8] S. Matej, G. T. Herman, T. K. Narayan, S. S. Furuie, R. M. Lewitt, and P. E. Kinahan. Evaluation of task-oriented performance of several fully 3d pet reconstruction algorithms. Phys. Med. Biol., 39:355–367, 1994. 167
180
Jos´e J. Fern´ andez et al.
[9] P. E. Kinahan, S. Matej, J. S. Karp, G. T. Herman, and R. M. Lewitt. A comparison of transform and iterative reconstruction techniques for a volume-imaging pet scanner with a large axial acceptance angle. IEEE Trans. Nucl. Sci., 42:2281– 2287, 1995. 167, 175 [10] Y. Censor, D. Gordon, and R. Gordon. Component averaging: An efficient iterative parallel algorithm for large and sparse unstructured problems. Parallel Computing, 27:777–808, 2001. 167, 169, 175 [11] Y. Censor, D. Gordon, and R. Gordon. BICAV: A block-iterative, parallel algorithm for sparse systems with pixel-related weighting. IEEE Trans. Med. Imag., 20:1050–1060, 2001. 167, 169, 175 [12] G. T. Herman. Medical Imaging. Systems, Techniques and Applications. C. Leondes (Ed.), chapter Algebraic Reconstruction Techniques in Medical Imaging, pages 1–42. Gordon and Breach Science, 1998. 168, 169 [13] B. Wilkinson and M. Allen. Parallel Programming. Prentice Hall, 1999. 170, 175
Visualization of RNA Pseudoknot Structures Wootaek Kim1 , Yujin Lee2 , and Kyungsook Han2 1
2
Department of Automation Engineering, Inha University Department of Computer Science and Engineering, Inha University Inchon 402-751, South Korea [email protected]
Abstract. RNA pseudoknots are not only important structural elements for forming the tertiary structure, but also responsible for several functions of RNA molecules such as frame-shifting, read-through, and the initiation of translation. There exists no automatic method for drawing RNA pseudoknot structures, and thus representing RNA pseudoknots currently relies on significant amount of manual work. This paper describes the first algorithm for automatically generating a drawing of H-type pseudoknots with RNA secondary structures. Two basic criteria were adopted when designing the algorithm: (1) overlapping of structural elements should be minimized to increase the readability of the drawing, and (2) the whole RNA structure containing pseudoknots as well as the pseudoknots themselves should be recognized quickly and clearly. Experimental results show that this algorithm generates uniform and clear drawings with no edge crossing for all H-type RNA pseudoknots. The algorithm is currently being extended to handle other types of pseudoknots.
1
Introduction
Visualization of complex molecular structures helps people understand the structures, and is a key component of support tools for many applications in biosciences. This paper describes a new algorithm for automatically producing a clear and aesthetically appealing drawing of RNA pseudoknot structures. A pseudoknot is a tertiary structural element of RNA formed when bases of a single-stranded loop pair with complementary bases outside the loop. Pseudoknots are not only widely occurring structural motifs in all kinds of viral RNA molecules, but also responsible for several important functions of RNA. For example, pseudoknot structures present in coding regions can stimulate ribosomal frame-shifting and translational read-through during elongation. Pseudoknots in non-coding regions can initiate translation either by being part of the so-called internal ribosomal entry site (IRES) in the 5 non-coding region or by forming translational enhancers in the 3 non-coding region [1]. Although 14 types of topologically distinct pseudoknots are possible according to this broad definition [2], most of the pseudoknots are of the so-called H-type, where H stands for hairpin loop, in which bases in a hairpin loop pair with complementary bases outside the hairpin loop (see Figure 1). J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 181–194, 2003. c Springer-Verlag Berlin Heidelberg 2003
Wootaek Kim et al.
loop 1
3’
(C) loop 2
(D)
loop 1
loop 2
stem 1
(B) stem 2
(A)
stem 2
182
5’
stem 2
loop 3
stem 1
loop 3
stem 1
stem 1
stem 2
loop 2
loop 1 loop 3
3’ 5’
3’
5’
5’
3’
Fig. 1. Schematic representation of H-type pseudoknots, adapted from [3]. (A) General configuration. (B) Loop 1 is eliminated. (C) Loop 2 is eliminated. (D) Loop 3 is eliminated. The pseudoknot shown in (C) is the one most abundant in natural RNA There currently exists no automatic method for drawing RNA pseudoknot structures. Several computer programs have been developed for drawing RNA structures (for example, [4, 5, 6, 7], but all of them are intended for drawing RNA secondary structures only. Therefore, representing RNA pseudoknots heavily relies on manual work. In a sense of graph theory, a drawing of RNA pseudoknots is a graph (and possibly non-planar graph) while that of RNA secondary structures is a tree. Thus drawing RNA pseudoknot structures is computationally much harder than RNA secondary structures. One of the difficulties in drawing RNA structures is overlapping of structural elements, which reduces the readability of the drawing. In most drawing programs of RNA secondary structures, computational load is increased because of the work associated with removing overlap of structural elements, performed either by an iterative process of the programs or with user intervention. We have developed an algorithm for generating overlap-free drawings of Htype pseudoknot structures, and implemented the algorithm in a working program called PseudoViewer. This is the first algorithm for automatically drawing RNA pseudoknot structures. We adopted two basic criteria when designing the algorithm: (1) Overlapping of structural elements should be minimized to increase the readability of the drawing, and (2) Not only pseudoknots themselves but also the whole RNA structure containing pseudoknots should be recognized quickly and clearly. The rest of this paper describes the algorithm and its experimental results.
2
Preliminaries
An RNA molecule is made up of ribonucleotides linked together in a chain. Each ribonucleotide comprises one of four different bases, adenine (A), cytosine (C), guanine (G), and uracil (U). Single-stranded RNA often folds back onto itself in structures stabilized by hydrogen bonds between A and U and between G and C. The so-called G-U wobble pair is also frequently found although it may have a weaker bond than the A-U and G-C pairs.
Visualization of RNA Pseudoknot Structures
183
Root
stem angle upSLoopNum stemPosOfUpSLoop totStemsOfUpSLoop pairVector basesVector
Base
RLoop isPKLoop upStemNum upRLoopNum radius startAngle angle farFromUpRLoop sortedVector treeDepth basesVector
PKLoop sideCount sidesLength[ ] radius treeDepth basesVector Pseudoknot PKLength angle basesVector
Fig. 2. The relationship of the classes representing the structural elements. RLoop stands for a regular loop, and PKLoop stands for a pseudoknot loop
In RNA secondary structures, there are two types of structural elements: – Stem (also called helix): double-stranded part, which is a contiguous region of base pairs. – Regular loop: single-stranded part such as a hairpin loop, internal loop, bulge loop, multiple loop, or dangling end. Since our algorithm draws pseudoknot structures, we consider additional structural elements as well: – Pseudoknot: structural element formed by pairing of bases in a regular loop with complementary bases outside the loop (see Figure 1 for an example). – Pseudoknot loop: loop that contains a pseudoknot as well as single-stranded part (see Figure 9 for an example). Figure 2 shows the relationship of the classes representing the above structural elements. Every class has a variable called basesVector, which contains the bases of the class.
3
Drawing RNA Pseudoknots
As mentioned earlier, a drawing of RNA pseudoknots is a graph with inner cycles within a pseudoknot as well as possible outer cycles formed between a pseudoknot and other structural elements. What we call a ”PK loop” represents the outer cycle. Given RNA pseudoknots and secondary structures, we represent the whole structure as a tree rather than as a graph (see Figure 3 for an example of the abstract tree). This is possible by representing both regular loops and PK loops as nodes of the tree. Edges of the tree represent stems of the secondary structure. The root node of the tree is the loop with the smallest starting base number. If
184
Wootaek Kim et al.
1-10/ 340~350
0 20-30/ 120-130
2 40~50/ 60-70
140-150/ 320-330
1
160-170/ 280-290 180-190/ 200-210
80~90/ 100-110
2
1
3
2
3
280-290/ 260-270
220-230/ 240-250
3
Fig. 3. An abstract tree for the whole structure. The label in each loop represents the level (i.e., depth) of the loop in the abstract tree. The labels in each stem represent the starting and ending base numbers of the stem. For example, the stem with label 1-10/340-350 indicates the stem is formed by pairing bases 1-10 with bases 340-350 there is a dangling end, we add artificial bases to pair the first and last bases. The artificial bases introduced in this step are not actually shown in the final drawing. A pseudoknot itself is not represented in the abstract tree; it is part of a node of the tree. By hiding the inner cycles as well as the outer cycles in the nodes of the abstract tree, we can represent the whole, top-level RNA structure as a tree, making the drawing process simple. Loops of the tree are placed and drawn in increasing order of their depth values (the root node has the smallest depth value). The member treeDepth of classes RLoop and PKLoop represents the depth of the loop in the abstract tree. The outline of both a regular loop and a PK loop is drawn in a circle shape. The algorithm of PseudoViewer is outlined as follows: 1. Stems, regular loops, pseudoknots and PK loops are identified from the input structure data. 2. An abstract tree is constructed for representing the entire structure. 3. For each node and edge of the tree, its size and shape is determined. 4. Starting with the root node, each node and edge of the tree is positioned level by level by both translation and rotation. Following principles are used in drawing RNA pseudoknot structures. – All angles are measured with respect to the positive y-axis. – Units of angles are radians rather than degrees, unless specified otherwise. – Modulus operator (%) is used for computing angle values since angle values should be in the range [0, 2π]. – In the drawing, unfilled circles represent the bases of a stem while filled circles represent the bases in the single-stranded part.
Visualization of RNA Pseudoknot Structures
185
|x2-x1| (x2,y2)
|y2-y1| PKLength
C
C
G
G
C
G
C
U
G
U
A
3'
G
A
G
A
G
C
A
G
C
A
A
U
U
G
5'
G
G
C
C
G
(x1,y1)
Fig. 4. Drawing of a pseudoknot
3.1
Pseudoknots
PseudoViewer takes as input an ASCII file in pairing format, which is widely used for representing pseudoknots [9]. The paring format describes pseudoknots as well as secondary structures in the following style. In this example, a pseudoknot is formed by base pairing G56 CGGU U61 with A74 GCCGC79 . 50 60 70 $ CGAGGGGCGGUUGGCCUCGUAAAAAGCCGC % (((((:[[[[[[::))))):::::]]]]]]
Given this input, PseudoViewer generates a drawing of a pseudoknot as shown in Figure 4. The data member PKLength of class Pseudoknot represents the diagonal length of a bounding box of the pseudoknot and is computed by P KLength = (xmax − xmin )2 + (ymax − ymin )2 (1) xmin , ymin : minimum x and y coordinates of the bounding box xmax , ymax : maximum x and y coordinates of the bounding box 3.2
Regular Loops
The relation of a regular loop and stems connected to the loop can be classified into 4 types, as shown in Figure 5. The outline of a regular loop is drawn in a circle shape. In Figure 6, the distance between the centers of adjacent bases of a regular loop is 2x. Since
186
Wootaek Kim et al.
(A)
(B)
(C)
(D)
Fig. 5. The relation of a regular loop and stems connected to the loop. (A) Two stems are directly connected to each other. (B) There exists at least one intervening base between two stems. ( C) There exists only one stem connected to a loop. (D) All stems are directly connected to each other with no intervening base between them
th e
ta
x
delta
r a b
(xc , yc )
(xi , yi)
Fig. 6. The radius of a regular loop and the angle between adjacent stems
a regular loop is considered to consist of several isosceles triangles, the vertical angle can be computed using equation 2, where nlb is the number of bases on the regular loop. 1 2π π a = · = (2) 2 nlb nlb The radius of a regular loop can be easily computed using equation 3. x (3) sin a Once we determine the radius of a regular loop, we can calculate the angle between adjacent stems and the angle θi between the i-th base of a regular loop and the positive y-axis. Let nlb be the number of bases on a regular loop and nb be the number of intervening bases between adjacent stems. Then, r=
Visualization of RNA Pseudoknot Structures
δ = 2a(nb + 2)
187
(4)
θi = (2i + 1)a, i = 0, 1, 2, . . . , nlb − 1
(5)
Using the angles computed by equations 2-5, the bases of a regular loop can be positioned as follows. xi = −r sin θi + xc
(6)
yi = r cos θi + yc
(7)
xi , yi : x and y coordinates of the i-th base of a regular loop xc , yc : x and y coordinates of the center of a regular loop θi : angle between the i-th base of a regular loop and the positive y-axis r: radius of a regular loop In Figure 7, an arrow points to the base with the smallest number of a stem. In the first regular loop, both the stem angle φ of the first stem and the startAngle
0
+
120+180
+ +
+ startAngle
r1 Length
+ r2
+
+ +
Fig. 7. The stem angle, startAngle, and the distance between the centers of regular loops
188
Wootaek Kim et al.
of the loop are zeros. The stem angles φ of other stems are measured with respect to the positive y-axis, and are calculated using equation 8. φ = Dloop + π + δ
(8)
where Dloop is the startAngle of the upper regular loop. To make the stem point to the upper regular loop, π is added in computing the stem angle φ. The StartAngle of the lower regular loop is equal to the stem angle of the current stem. The distance Drl between the centers of regular loops is computed using equation 9. Drl = rc + (nub − 1) × hs + ru
(9)
rc : radius of the current regular loop ru : radius of the upper regular loop nub : number of bases of the upper stem hs : distance between adjacent base pairs of a stem 3.3
Pseudoknot Loops
In order to handle a pseudoknot loop (PKLoop) in a similar way to a regular loop, we need several variables, such as the angle between adjacent bases, the radius of a PKLoop, and the angle between stems of a PKLoop. An inscribed polygon of a PKLoop consists of several isosceles triangles (see Figure 8). Therefore, the following relation holds: xi xi ⇒ ai = arcsin( ) r r If there are total n isosceles triangles inside a PKLoop, then sin ai =
f=
n
arcsin(
i=1
xi )−π =0 r
(10)
(11)
xi r ai
Fig. 8. An inscribed polygon of a PKLoop consists of several isosceles triangles
Visualization of RNA Pseudoknot Structures
189
WS
r WS
2A
A
B WS
alpha
LP
Fig. 9. Components of a PKLoop. Ws : width of a stem, which equals the distance between adjacent bases of a loop, Lp : diagonal length of a pseudoknot, r: radius of a PKLoop, α: angle between the diagonal direction and the stem direction of a pseudoknot
Function f is monotonically decreasing after a certain point, and Newton’s method can be used to determine r that satisfies f = 0. However Newton’s method requires differentiation of function f , which takes a significant amount of computation as shown in equation 12. n
df = dr i=1
−x i r2 1 −
x2i r2
=
n i=1
r
−xi r2 − x2i
(12)
Instead of using Newton’s method, we determine the value of r by incrementing r by a small step. Once we determine the radius r of a PKLoop, several angles associated with a PKLoop can be computed using equations 13-16 (see Figure 9). Let Lp be the diagonal length of the pseudoknot region, xe and ye be the coordinates of the ending base of a pseudoknot, and xs and ys be the coordinates of the starting base of a pseudoknot. Then, Lp = (xe − xs )2 + (ye − ys )2 (13)
190
Wootaek Kim et al.
sin A =
Ws /2 r
⇒
A = arcsin(
Ws ) 2r
(14)
Lp /2 Lp ⇒ B = 2 · arcsin( ) (15) r 2r A pseudoknot in a PKLoop can be positioned in a similar way to a stem. To do this, we first orient a pseudoknot in the positive y direction by rotating it through angle α counterclockwise about its starting base. The rotation angle α is computed by equation 16. sin(B/2) =
sin α =
4
2Ws Lp
⇒
α = arcsin(
2Ws ) Lp
(16)
Experimental Results
PseudoViewer was implemented in Microsoft C and tested on several RNA structures with pseudoknots. Figures 10 and 11 show the structures of tobacco mosaic virus (TMV) RNA [10] and Cyanophora paradoxa cyanelle tmRNA [11], respectively. Bases are numbered in the frequency of 10 in principle. If a base numbers falls on a loop, it is shown in the drawing; otherwise, it is not shown to avoid overlaps. In addition to this, the starting base of each pseudoknot is shown in green background color with its base number. Pseudoknots are shown in yellow background color, and thus are easily distinguished from other structural elements. Filled circles represent the canonical base pairs (A-U or G-C), and open circles represent the G-U wobble pairs. There is no overlapping of structural elements, and the PK loop with several pseudoknots takes on a circle shape similar to a regular loop. When drawing a pseudoknot by first generating a secondary structure with an ordinary secondary drawing program and then modifying the secondary structure, pseudoknots as well as other structural elements are often distorted, and thus the readability of the drawing is much reduced. In addition to the standard drawings, as shown in Figures 10 and 11, PseudoViewer produces outline drawings as well. The outline drawings display the structure in the form of a backbone in which loops are replaced by circles and helices by line segments. PseudoViewer also provides an interactive editing facility for manually fixing drawings when overlaps occur. In the editing mode, the user can drag a regular loop to any position.
5
Conclusion and Future Work
Generating a clear representation of RNA pseudoknot structures is computationally harder than RNA secondary structures since a drawing of pseudoknots is a graph (and possibly nonplanar) rather than a tree and contains more overlapping of structural elements. We have developed a new algorithm for visualizing H-type RNA pseudoknots as a planar graph and implemented the algorithm in a
Visualization of RNA Pseudoknot Structures
191
Fig. 10. Structure of tobacco mosaic virus, visualized by PseudoViewer working program called PseudoViewer. We believe PseudoViewer is the first program for automatically drawing RNA secondary structures containing H-type pseudoknots. We adopted two basic criteria when designing PseudoViewer: (1) overlapping of structural elements should be minimized to increase the readability of the drawing, and (2) the whole RNA structure containing pseudoknots as well as the pseudoknots themselves should be recognized quickly and clearly. Experimental results showed that PseudoViewer is capable of generating a clear and aesthetically pleasing drawing of all H-type RNA pseudoknots.
192
Wootaek Kim et al.
Fig. 11. Structure of Cyanophora paradoxa cyanelle tmRNA
Visualization of RNA Pseudoknot Structures
(A)
(B)
3'
5'
(C)
3'
(D)
3'
193
(E)
3'
5'
5'
5'
5'
3'
Fig. 12. Pseudoknots of other types. (A) LL type. (B) HL OUT type. (C) HL IN type. (D) HH type. (E) HHH type. H: hairpin loop, L: bulge loop, internal loop, or multiple loop Table 1. Classification of 236 pseudoknots in PseudoBase Pseudoknot type H LL HL OUT HL IN HH HHH Unclassified total
Number of occurrences 180 12 24 11 1 6 2 236
Ratio of occurrences 76.3% 5.1% 10.2% 4.7% 0.4% 2.5% 0.8% 100.0%
The development of PseudoViewer is not complete yet, and is being extended in several directions. First, it can currently visualize typical H-type pseudoknots only. Although the most commonly occurring pseudoknots are of the Htype, PseudoViewer is being extended to handle other types of pseudoknots as well. Basepairing of a hairpin loop with a single stranded part outside the hairpin loop forms the H-type pseudoknot, which has been discussed in this paper. Basepairing of a hairpin loop with another hairpin loop forms a HH type pseudoknot, while basepairing of a hairpin loop with a single stranded part of a bulge, internal, or multiple loop forms a HL type (Figure 12). We have analyzed all the pseudoknots of PseudoBase [9] for the purpose of drawing them in combination of basic pseudoknot types and concluded that there are 6 basic pseudoknot types in total – H-type and 5 other types displayed in Figure 12 (see also Table 1). Second, PseudoViewer will be made available as a web-based application program so that it can be executed anywhere using a general web browser. Finally, PseudoViewer currently represents RNA pseudoknots as a twodimensional drawing. We would like to display RNA pseudoknots as a threedimensional drawing. Perhaps this would be the ultimate goal of our visualization work of RNA pseudoknots.
194
Wootaek Kim et al.
Acknowledgments This work has been supported by the Korea Science and Engineering Foundation (KOSEF) under grant R05-2001-000-01037-0.
References [1] Deiman, B. A. L. M., Pleij, C. W. A.: Pseudoknots: A Vital Feature in Viral RNA. Seminars in Virology 8 (1997) 166–175 181 [2] Du, Z., Hoffman, D. W.: An NMR and mutational study of the pseudoknot within the gene 32 mRNA of bacteriophage T2: insights into a family of structurally related RNA pseudoknots. Nucleic Acids Research 25 (1997) 1130–1135 181 [3] Hilbers, C. W., Michiels, P. J. A., Heus, H. A.: New Developments in Structure Determination of Pseudoknots. Biopolymers 48 (1998) 137–153 [4] Han, K., Kim, D., Kim, H.-J.: A vector-based method for drawing RNA secondary structure. Bioinformatics 15 (1999) 286–197 182 [5] Chetouani, F., Monestie, P., Thebault, P., Gaspin, C., Michot, B.: ESSA: an integrated and interactive computer tool for analyzing RNA secondary structure. Nucleic Acids Research 25 (1997) 3514–3522 182 [6] Shapiro, B. A., Maizel, J., Lipkin, L. E., Currey, K., Whitney, C.: Generating nonoverlapping displays of nucleic acid secondary structure. Nucleic Acids Research 12 (1984) 75–88 182 [7] Winnepenninckx, B., Van de Peer, Y., Backeljau, T., De Wachter, R.: CARD: A Drawing Tool for RNA Secondary Structure Models. BioTechniques 16 (1995) 1060–1063 182 [8] De Rijk, P., De Wachter, R.: RnaVIz, a program for the visualization of RNA secondary structure. Nucleic Acids Research 25 (1997) 4679–4684 [9] van Batenburg, F. H. D., Gultyaev, A. P., Pleij, C. W. A., Ng, J., Olihoek, J.: PseudoBase: a database with RNA pseudoknots. Nucleic Acids Research 28 (2000) 201–204 185, 193 [10] Gultyaev, A. P., van Batenburg, E. Pleij, C. W. A.: Similarities between the secondary structure of satellite tobacco mosaic virus and tobamovirus RNAs. Journal of General Virology 75 (1994) 2851–2856 190 [11] Knudsen, B., Wower, J., Zwieb, C., Gorodkin, J.: tmRDB (tmRNA database). Nucleic Acids Research 29 (2001) 171–172 190
The Cactus Framework and Toolkit: Design and Applications Invited Talk Tom Goodale1 , Gabrielle Allen1 , Gerd Lanfermann1 , Joan Mass´ o2 , 1 1 3 Thomas Radke , Edward Seidel , and John Shalf 1
Max-Planck-Institut f¨ ur Gravitationsphysik, Albert-Einstein-Institut Am M¨ uhlenberg 1, 14476 Golm, Germany {goodale,allen,lanfer,tradke,eseidel}@aei.mpg.de 2 Departament de F´ısica, Universitat de les Illes Balears E-07071 Palma de Mallorca, Spain [email protected] 3 Lawrence Berkeley National Laboratory Berkeley, CA [email protected]
Abstract. We describe Cactus, a framework for building a variety of computing applications in science and engineering, including astrophysics, relativity and chemical engineering. We first motivate by example the need for such frameworks to support multi-platform, high performance applications across diverse communities. We then describe the design of the latest release of Cactus (Version 4.0) a complete rewrite of earlier versions, which enables highly modular, multi-language, parallel applications to be developed by single researchers and large collaborations alike. Making extensive use of abstractions, we detail how we are able to provide the latest advances in computational science, such as interchangeable parallel data distribution and high performance IO layers, while hiding most details of the underlying computational libraries from the application developer. We survey how Cactus 4.0 is being used by various application communities, and describe how it will also enable these applications to run on the computational Grids of the near future.
1
Application Frameworks in Scientific Computing
Virtually all areas of science and engineering, as well as an increasing number of other fields, are turning to computational science to provide crucial tools to further their disciplines. The increasing power of computers offers unprecedented ability to solve complex equations, simulate natural and man-made complex processes, and visualise data, as well as providing novel possibilities such as new forms of art and entertainment. As computational power advances rapidly, computational tools, libraries, and computing paradigms themselves also advance. In such an environment, even experienced computational scientists and engineers can easily find themselves falling behind the pace of change, while they J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 197–227, 2003. c Springer-Verlag Berlin Heidelberg 2003
198
Tom Goodale et al.
redesign and rework their codes to support the next computer architecture. This rapid pace of change makes the introduction of computational tools even more difficult in non-traditional areas, e.g., social arts and sciences, where they may potentially have the most dramatic impact. On top of this rapidly changing background of computation, research and engineering communities find themselves more and more dependent on each other as they struggle to pull together expertise from diverse disciplines to solve problems of increasing complexity, e.g., simulating a supernova or the Earth’s climate. This creates a newfound need for collaborative computational science, where different groups and communities are able to co-develop code and tools, share or interchange modules and expertise, with the confidence that everything will work. As these multidisciplinary communities develop ever more complex applications, with potentially hundreds or thousands of modules and parameters, the potential for introducing errors, or for incorrectly using modules or setting parameters, also increases. Hence a software architecture that supports a wide range of automated consistency checking and multi-architecture verification becomes essential infrastructure to support these efforts. Application developers and users also need to exploit the latest computational technologies without being exposed to the raw details of their implementation. This is not only because of the time needed to learn new technologies, or because their primary interest is in what the application itself can do, and not how it does it, but also because these technologies change rapidly; e.g. time invested in learning one technology for message passing is largely wasted when a developer has to learn the next. What the developer really needs is an abstracted view of the operations needed for data distribution, message passing, parallel IO, scheduling, or operations on the Grid [1] (see section 16.5), and not the particular implementation, e.g., a particular flavour of MPI, PVM, or some future replacement. The abstract concept of passing data between processors does not change over time, even though a specific library to do it will. A properly designed application framework with suitable abstractions could allow new or alternate technologies, providing similar operations, to be swapped in under the application itself. There is a great need for developing future-proof applications, that will run transparently on today’s laptops as well as tomorrow’s Grid. At the same time, many application developers, while recognising the need for such frameworks, are reluctant to use “black box”, sealed packages over which they have little control, and into which they cannot peer. Not only do some application developers actually want to see (perhaps some) details of different layers of the framework, in some cases they would like to be able to extend them to add functionality needed for their particular application. Such transparency is nearly impossible to provide through closed or proprietary frameworks without resorting to excessively complex plug-in SDKs. For this reason, freely available, open source tools are preferred by many communities (see, for example, Linux!). Source code is available for all to see, to improve, and to extend; these improvements propagate throughout the communities that share open source tools.
The Cactus Framework and Toolkit: Design and Applications
199
The idea of open source tools, once accepted by one segment of a community, tends to be contagious. Seeing the benefits, application developers making use of such open tools are often inclined to make their own modules freely available as well, not only in the computational science areas, but also for modules specific to research disciplines. In the computational science disciplines, modular components that carry out specific tasks are increasingly able to inter-operate. Developing application components for, or in, one framework makes it easier to use them with another framework. This is a particular goal of the “Common Component Architecture” [2], for example. As another example, we describe below application modules developed for numerical relativity and astrophysics; a growing number of scientists in this domain make use of the Cactus framework [3], and many of them make their application specific modules available to their community. Not only does this encourage others to share their codes, it raises the overall quality; knowing modules will become public increases the robustness of the code, the accompanying documentation, and the modularity. For all of these reasons, and more that we touch on below, open, modular, application frameworks that can both hide computational details, and enable communities to work together to develop portable applications needed to solve the complex problems in their disciplines, are becoming more and more important in modern computational science. This will become particularly true as the Grid, described below, becomes a reality [1]. “Grid-enabling frameworks” will be crucial if application scientists and engineers are to make use of new computational paradigms promised by the Grid [4].
2
Cactus as an Application Framework
We have motivated various general reasons why application frameworks are needed, and in the rest of this paper we describe a specific example. The programming framework Cactus [5, 6], developed and used over a number of years, was designed and written specifically to enable scientists and engineers to perform the large scale simulations needed for their science. From the outset, Cactus has followed two fundamental tenets: respecting user needs and embracing new technologies. The framework and its associated modules must be driven from the beginning by user requirements. This has been achieved by developing, supporting, and listening to a large user base. Among these needs are ease of use, portability, the abilities to support large and geographically diverse collaborations, and to handle enormous computing resources, visualisation, file IO, and data management. It must also support the inclusion of legacy code, as well as a range of programming languages. It is essential that any living framework be able to incorporate new and developing, cutting edge computation technologies and infrastructure, with minimal or no disruption to its user base. Cactus is now associated with many computational science research projects, particularly in visualisation, data management, and the emerging field of Grid computing [7].
200
Tom Goodale et al.
These varied needs for frameworks of some kind have long been recognised, and about a decade ago the US NSF created a funding program for “Grand Challenges” in computational science. The idea was to bring computer and computational scientists together with scientists in specific disciplines to create the software needed to exploit large scale computing facilities to solve complex problems. This turned out to be a rather difficult task; in order to appreciate the evolution of attempts to tackle this problem, and the maturity of current solutions, it is instructive to review a bit of history before plunging directly into the present design of Cactus 4.0. Like many of these NSF Grand Challenge projects, a particular project called the “Black Hole Grand Challenge Alliance”, aimed to develop a “Supertoolkit” for the numerical relativity community to solve Einstein’s equations for the study collisions of black holes. The thought was that, as these equations are so complex, with equations of many types (e.g., hyperbolic, elliptic, and even parabolic), a properly designed framework developed to solve them would also be a powerful tool in many fields of science and engineering. (This has turned out to be true!) A centrepiece of this toolkit was, in it simplest description, a parallelisation software known as DAGH [8] (which has now evolved into a package called GrACE [9]), that would allow the physicist to write serial Fortran code that could be easily parallelised by using the DAGH library. At the same time, a series of independent parallel codes were developed to solve Einstein’s equations, with the expectation that they would later make use of DAGH for parallelism. However, some of these codes, notably the “G-code” [10], took on a life of their own, and became mini-frameworks (in Fortran!) themselves, evolving into workhorses for different collaborations within the project. As the codes and collaborations grew, it became clear that the original designs of these codes needed major revision. A prototype framework, naturally called “The Framework” [11], was designed and developed by Paul Walker at the Max Planck Institute for Gravitational Physics (AEI), with the idea that different physics codes could be plugged in, like the G-code and others, and that it would be an interface to DAGH to provide parallelism. Learning from this experiment, Paul Walker and Joan Masso began to develop Cactus in January 1997. Cactus 1.0 was a re-working of the Framework for uni-grid applications using a simple parallel uni-grid driver as a temporary replacement for DAGH. In Cactus there were two types of object, the “flesh”, a core component providing parameter parsing, parallelisation, and IO, and “thorns” which are optional modules compiled in. In October 1997 Cactus 2.0 was released, providing for the first time the ability to dynamically pick which routines were run at run-time, and some ability to order them, this ability removed the necessity of modifying the flesh to support new thorns. In these early versions of Cactus all thorns received all variables defined for a simulation; Cactus 3.0 removed this necessity, further enhancing the modularity. Cactus became the major workhorse for a collaboration of numerical relativists spanning several large projects in Europe and the US. This community requires very large scale, parallel computational power, and makes use of com-
The Cactus Framework and Toolkit: Design and Applications
201
putational resources of all types, from laptops for development to heavy-duty production simulations on virtually every supercomputing architecture, including Linux clusters, “conventional” supercomputers (e.g., SGI Origins or IBM SP systems), and Japanese vector parallel machines. Hence complete portability and very efficient parallel computation and IO are crucial for its users. Lessons learnt through its heavy use in this community led to the development of a still more modular and refined version of Cactus, Cactus 4.0, now the flesh contains almost no functionality; anything can be implemented as a thorn, not just physics routines, but all computational science layers. These computational science thorns implement a certain function, say message passing, hydrodynamics evolution, or parallel IO, and are interchangeable; any thorn that implementing a given function can be interchanged with any other one. For example, different “driver layer” thorns (see section 10) that provide message passing, can be implemented and interchanged, and can be written using any message passing library; properly written applications that make use of Cactus can use any of them without modification. Entire software packages, such as GrACE for parallelism, PETSc [12] for linear solvers, and others, can be made available for Cactus users through thorns that are designed to connect them. Hence, Cactus should be thought of as a Framework for conveniently connecting modules or packages with different functionality, that may have been developed completely outside of Cactus. In principle, Cactus can accommodate virtually any packages developed for many purposes, and it is very powerful. Moreover, while earlier versions of Cactus were limited in the number of architectures they supported (T3E, IRIX, OSF, Linux, NT), the community’s need for portability lead to a complete redesign of the make system to enhance portability and reduce compilation times, and now Cactus compiles on practically any architecture used for scientific computing, and the make system, which is described in section 6 is designed to make porting to new architectures as easy as possible. These points and others will be developed in depth in the sections below. Cactus 4.0 has been developed as a very general programming framework, written to support virtually any discipline. It has been developed through a long history of increasingly modular and powerful frameworks, always aimed at supporting existing communities. Hence it is by now a very mature, powerful, and most importantly, widely used framework; its design has been completely derived from the needs of its users, through both careful planning and much trial, error, and redesign. The first official release of the Cactus Computational Toolkit [13], which builds on the Cactus flesh to provide standard computational infrastructure is released along with Cactus 4.0, as well as application specific toolkits. The success of the Cactus approach is borne out by the wide use of Cactus not only by the numerical relativity community in five continents, but also its emerging adoption in other fields such as climate modelling, chemical engineering, fusion modelling, and astrophysics. In the next sections, we detail the design and capabilities of Cactus 4.0. Note that Cactus is not an application in itself, but a development environment in which an application can be developed and run. A frequent mis-
202
Tom Goodale et al.
interpretation is that one “runs Cactus”, this is broadly equivalent to stating that one “runs perl”; it is more correct to say that an application is run within the Cactus framework.
3
Design Criteria for Cactus 4.0
In reviewing previous versions of Cactus and other codes while designing the new Cactus framework, it became clear that the new framework had to meet some wide ranging design criteria. It must be able to run on a wide range of different machines, so requires a flexible and modular make system. Moreover people must be able to use the same source tree to build executables for different architectures or with different options on the same architecture without having to make copies of the source tree. The make system must be able to detect the specific features of the machine and compilers in use, but must also allow people to add configuration information that cannot otherwise be auto-detected in a transparent and concise way – without having to learn new programming languages. Similarly it should be easy to add new modules, and as much functionality as possible should be delegated to modules rather than the core. Data associated with individual modules should exist in separate name spaces to allow independently developed modules to co-exist without conflict. Functionally equivalent modules should be inter-changeable without other modules which use or depend on them noticing the exchange; many such modules should be able to be compiled into the executable at once and the desired one picked at program startup. A core requirement is to provide transparent support for parallelism. The framework should provide abstractions for distributed arrays, data-parallelism, domain decomposition and synchronisation primitives. Furthermore, the primitives used to manage the parallelism must be minimal and posed at a high enough level of abstraction so as not to be tedious to use or to expose system architecture or implementation dependencies. The framework must be able to trivially build applications that work in uniprocessor, SMP, or MPP environments without code modification or placement of explicit conditional statements in their code to account for each of these execution environments. If the programmer wishes to build a serial implementation on a multiprocessor platform for testing purposes, the framework must support that build-time choice. Another important class of modules is those responsible for input and output. There are many different file formats in use, and the framework itself should not dictate which is used. Nor should other modules be unnecessarily tied to any particular IO module – as new IO modules are developed people should be able to access their functionality without modification to existing code. In past versions of Cactus it was possible to set input parameters to values which were meaningless or, in the case of keyword parameters, to values not recognised by any module. This is obviously undesirable and should be picked up when the parameter file is read.
The Cactus Framework and Toolkit: Design and Applications
203
There are many frameworks (for a review, see [14]), however most of them require a new code to be written to fit within the framework, and often restrict the languages which can be used. One design criteria was to be able to support legacy codes, and, in particular, to support the large number of codes and programmers who use FORTRAN 77. Another criterion was to be able to allow applications using the framework to utilise features such as parallelism, adaptive mesh refinement (AMR) and out of core computation without having to modify their code, as long as their code is structured appropriately. All the above requirements can be summed up in saying that Cactus must be portable and modular, should abstract the interfaces to lower-level infrastructure as much as possible, but be easy to use and capable of supporting legacy codes. By themselves these requirements would lead to something with cleaner interfaces than older versions of Cactus, however we also chose to design the framework to be as flexible and future proof as possible. The framework should allow the modules to provide access to new technologies, such as those to provide collaborative working, or facilities emerging from the Grid community, while at the same time the framework should not come to depend upon such technologies. This list is by no means a complete list of requirements, however these are the more important ones coming out of our experiences with earlier versions of cactus and our desire for the framework to be as future-proof as possible.
4
Structure
The code is structured as a core – the ‘flesh’ – and modules which are referred to as ‘thorns’. The flesh is independent of all thorns and provides the main program, which parses the parameters and activates the appropriate thorns, passing control to thorns as required. It contains utility routines which may be used by thorns to determine information about variables, which thorns are compiled in or active, or perform non-thorn-specific tasks. By itself the flesh does very little apart from move memory around, to do any computational task the user must compile in thorns and activate them at run-time. Thorns are organised into logical units referred to as ‘arrangements’. Once the code is built these have no further meaning – they are used to group thorns into collections on disk according to function or developer or source. A thorn is the basic working module within Cactus. All user-supplied code goes into thorns, which are, by and large, independent of each other. Thorns communicate with each other via calls to the flesh API, plus, more rarely, custom APIs of other thorns. The connection from a thorn to the flesh or to other thorns is specified in configuration files which are parsed at compile time and used to generate glue code which encapsulates the external appearance of a thorn. When the code is built a separate build tree, referred to as a ‘configuration’ is created for each distinct combination of architecture and configuration options,
204
Tom Goodale et al.
Flesh
Driver
Read Parameter File Activate Thorns Initialise Scheduler
[Recovering From Checkpoint] [Not Recovering From Checkpoint] CCTK_RecoverParameters
CCTK_Startup
Initialise Simulation Evolve Simulation
Tidy Up
Terminate Simulation
Fig. 1. The main flow of control in the Cactus Framework. The flesh initialises the code, then hands control to the driver thorn (see section 10). The actions in the driver swimlane are detailed in figures 2, 3, and 4.
e.g. optimisation or debugging flags, compilation with MPI and selection of MPI implementation, etc. Associated with each configuration is a list of thorns which are actually to be compiled into the resulting executable, this is referred to as a ‘thornlist’. At run time the executable reads a parameter file which details which thorns are to be active, rather than merely compiled in, and specifies values for the control parameters for these thorns. Non-active thorns have no effect on the code execution. The main program flow is shown in figure 1.
5
Code Management and Distribution
Since the code is developed and maintained by a geographically dispersed team of people, a well-defined code management system is necessary. The Cactus development follows clearly defined coding guidelines and procedures which are set out in the Cactus Maintainers’ Guide [15].
The Cactus Framework and Toolkit: Design and Applications
CCTK_ParamCheck
205
Terminate Simulation
[Invalid Parameter Combinations] [Parameter Combinations Valid] CCTK_BaseGrid [Not Recovering From Checkpoint]
[Recovering From Checkpoint]
CCTK_Initial CCTK_PostInitial
CCTK_RecoverVariables CCTK_PostRecoverVariables
CCTK_PostStep
CCTK_CPInitial CCTK_Analysis
Output Data
Continue Simulation
Fig. 2. Initialisation action diagram (corresponding to the initialise simulation action in figure 1). All activities prefixed with “CCTK ” are schedule bins (see section 9) Check Termination [Simulation Over] Condition [Continue Simulation] Update Time and Rotate Timelevels CCTK_PreStep CCTK_Evol CCTK_PostStep CCTK_Checkpoint CCTK_Analysis Output Data
Fig. 3. Evolution action diagram (corresponding to the evolve simulation action in figure 1). All activities prefixed with “CCTK ” are schedule bins (see section 9) CCTK_Terminate
CCTK_Shutdown
Fig. 4. Termination action diagram (corresponding to the terminate simulation action in figure 1). All activities prefixed with “CCTK ” are schedule bins (see section 9)
206
Tom Goodale et al.
A central component of these procedures is the use of CVS [16] to manage revisions in the flesh and the thorns. The tree-like file layout structure is purposely designed to enable each thorn or arrangement to belong to entirely independent CVS repositories with their own access control, branching policies. The Cactus flesh provides simplified interfaces to checked-out and assembled together these thorns into a working application. The primary code distribution method is via CVS, for which the several wrapper scripts are provided to automate downloading the flesh and user-defined lists of thorns. Thorns are not only maintained and distributed at the central Cactus repository, but by different authors who want to make their thorns available to others through their own CVS repositories. The Cactus download scripts can take a thornlist with some embedded comments and retrieve thorns from all the appropriate CVS repositories. Additionally compressed tar files of the flesh and thorns can be downloaded from the Cactus web-site. Documentation, including an FAQ, Users and Maintainers Guide, as well as Thorn Guides for different collections of thorns, are distributed with the Cactus flesh and thorns, as well as from the Cactus Home Page [6, 15, 13].
6
A Portable and Modular Make System
The Cactus make system is a two stage process. First of all a “configuration” is created, encapsulating machine-specific information and also user defined choices; the user may define many configurations, each for specific purposes, e.g. for use on different architectures, to use particular parallel libraries, to include debugging information, to be compiled with specific compilers, etc. Once a configuration has been created, a thornlist is created for the configuration, and the flesh and thorns for this configuration are built. If the thornlist or source files for a thorn are modified, the appropriate files for this configuration and its executable may be rebuilt. GNU Make is used to avoid incompatibilities in the make program across architectures. This is freely available on a large number of platforms. The Cactus make system has shown itself to work on a large number of Unix platforms, including SGI IRIX, IBM AIX (e.g. SP systems), Linux (IA32, IA64, Alpha, PowerPC, IPAQ, Playstation), Apple OSX, Compaq OSF, Cray Unicos (T3E, SV1, etc), Sun Solaris, Hitachi SR800, Fujitsu VPP, OpenBSD, and NEC SX-5; in principle Cactus should compile on any UNIX-like platform with at most minimal effort. To date the only non-Unix platform used has been Windows, where the Cygwin [17] environment is used to provide a Unix-like environment; the resulting code is, however, independent of Cygwin if non-Cygwin compilers are used. Note that unlike a library or system utility, Cactus is not “installed”. Currently each user has a copy of the sources and generates their own configurations based upon their individual needs and preferences.
The Cactus Framework and Toolkit: Design and Applications
6.1
207
Creating a Configuration
In order to support as wide a range of computers as possible, GNU Autoconf [18] is used to detect specific features of compilers and libraries. However not all such things can be easily detected, and it was deemed important that in those cases a user compiling on a previously unsupported architecture should not need to learn detailed shell programming and M4 to extend the detection system. To this end, architecture specific information, such as flags necessary for full optimisation, or debugging, or warnings, or which libraries need to be linked against for the Fortran compiler, is stored in ‘known-architecture’ files, which are simple shell scripts which are sourced by the master configuration script once it has detected the operating system. These files are simple in format, and new architectures can be supported merely by placing a new file in the directory holding these files, with a name corresponding to the operating system name returned by the ‘config.sub’ script invoked by Autoconf. A similar system is used to configure for optional packages such as MPI, pthreads or HDF5. To add a new optional package with its own configuration options, a user may just create a directory in a specific place, and add a script file in this place which may add extra definitions to the master include file or add extra information to the make system. This is heavily used to support the various different flavours of available MPI – native MPI on SGI, Cray and IBM AIX machines, or MPICH, LAM, HPVM, MPIPro on these or other systems. When creating a configuration the user may pass instructions to the configuration scripts to override detection or defaults. For example the user may choose to use a specific C or Fortran compiler, or choose to configure the code for parallel operation by configuring with MPI, or specific the user may want this configuration have different debugging or optimisation flags. The user may also pass a specific architecture name to the configuration system in order to cross-compile for a different architecture; in this case the known-architecture file for the target platform must contain certain details which the configure script would otherwise detect. 6.2
Building a Configuration
Once a configuration has been created the user must pick a set of thorns to build. If the user doesn’t provide one, the make system will search for thorns and generate a thornlist and give the user the option to edit it. The thornlist is a plain text file and may be edited with any text editor. Once there is a thornlist associated with a configuration, the CST, which is described in detail in section 8.2, is run to validate the thorns and provide bindings between the flesh and the thorns, and then the code is built based upon the flesh source and the sources of all thorns in the thornlist. The flesh and each thorn are compiled into libraries which are linked to form the final executable. In individual thorns the thorn writer has the option of using their own Makefile or of using a simplified system whereby they list the files and subdirectories which should be compiled, and a standard make file is used for that thorn. The
208
Tom Goodale et al.
vast majority of thorn writers use the latter approach, which enables them to develop with minimal knowledge of make file syntax or operation. More advanced users may also similarly write make rules specifying dependencies between source files in their thorn. Since the object files for each configuration are kept in different directories from the original source files, and a user may be building several configurations simultaneously from one source tree, care has to be taken with the working directory of the compilation, especially when Fortran 90 modules or C++ templates are used. This is currently solved by setting the working directory of any compilation as a specific scratch directory in the configuration-specific build area. Currently each thorn’s object files are packaged into libraries which are then linked together to form the final executable. In future versions of Cactus it is planned that these libraries may be dynamically loadable, on platforms which allow this, thus allowing new thorns providing new capabilities, in particular data analysis, to be loaded at run-time in long running simulations. At build time the user may pass options to the make system to echo all commands to the screen or to switch on compiler warnings. However no option which would change the final executable is allowed at this stage, on the principle that the files stored with the configuration should completely define the way the executable is produced. 6.3
Managing Configurations
The configuration is the basic unit with which an end-user operates. The Cactus framework provides many options to manage a configuration, some examples are: building an executable (this option checks the thornlist and dependencies of the executable and its sources, rebuilds any object files which are out of date, and re-links the final executable if necessary as described above); running a test suite (see section 13); removing object files or stored dependency information to force a clean rebuild or save disk space; or producing a document describing all the thorns in the configuration’s thornlist.
7
Code Portability
The previous section has described how the make system is able to configure and make the code on all platforms. However this would be of no use if the code itself were not portable. In order to ensure code portability, the flesh is written in ANSI C. Thorns, however, may be written in C, C++, FORTRAN 77 or Fortran 90. In order to ensure that thorns are platform independent, the configuration script determines various machine-specific details, such as the presence or absence of certain non-ANSI C functions (thorns are encouraged to use equivalent functions in the Cactus utility libraries (see section 12) which are guaranteed to be there), or the sizes of variables, which vary from platform to platform – in order to avoid problems with this, e.g. in cross-language calls or parallel jobs distributed across heterogeneous platforms, Cactus defines a set of types, such as CCTK REAL,
The Cactus Framework and Toolkit: Design and Applications
209
CCTK INT, which are guaranteed to have the same size on all platforms and in all languages. This also means that runs can produce exactly the same results on different machines, and checkpoint files created on one machine can be restarted on any other.
8
Modularity
This section describes how thorns identify themselves to the flesh and the make system, and the way in which functionality is encapsulated and name spaces are realised. Two important pieces of terminology must be introduced before preceding. Grid variables are variables which can be passed between thorns or routines belonging to the same thorn through the defined flesh interface; this implies it is related to the computational grid rather than being an internal variable of the thorn or one of its routines. An implementation is an abstract name for the functionality offered by a thorn; all thorns providing the same implementation expose the same interfaces to the outside world, and only one such thorn may be active at a time. 8.1
CCL
Each thorn provides three specification files written in the Cactus Configuration Language (CCL). These files designate which implementation the thorn provides and the variables that the thorn expects to be available, describe the parameters it uses, and specify routines from the thorn which must be called. Variable scope may be either private, i.e. visible only to routines from this thorn; protected, i.e. visible to all thorns in the group of friends of this thorn’s implementation; or public, i.e. visible to all thorns which inherit from this thorn’s implementation. Grid Variables fall into three categories: Grid Scalars (GSs), which are single numbers (per processor); Grid Functions (GFs), which are distributed arrays with a size fixed to the problem size (all GFs have the same size); and Grid Arrays (GAs) which are similar to GFs but may have any size. Grid Variables must be one of the defined CCTK types (e.g. CCTK REAL, CCTK INT, etc), whose size is determined when a configuration is created (see section 6.1). Grid Functions are by default vertex-centred, however they may be staggered along any axis. Parameter scope may similarly be private, or may be declared to be ‘restricted’, which means that other thorns may choose to share these parameters. Parameters may be numeric, i.e. integer or floating point, boolean, keywords, i.e. the value is a distinct token from a defined list, or arbitrary strings. The specification of a parameter includes a specification of the allowed values, i.e. a range of numbers for numeric, a list of keywords for keyword parameters, or a set of regular expressions for string parameters. Parameters may also be declared to be steerable. This means that they are not fixed for the duration of the simulation, but may have their value changed. By default parameters are non-steerable, however a thorn author can declare
210
Tom Goodale et al.
that a parameter may be steered at all times, or when the code is recovering from a checkpoint. This is an important distinction as, while parameters are provided as local variables to routines for speed of access, a lot of parameters are used to setup data structures or to derive other values, which would thus be unaffected if the parameter value was subsequently changed; by declaring a parameter “steerable” the thorn author is certifying that changing the value of the parameter will make a difference. (Parameters values may only be changed by calling a routine in the flesh which validates the change.) Routines from the thorn may be scheduled to run at various time bins and relative to the times when other routines from this thorn or other thorns are run. The thorn author may also schedule groups within which routines may be scheduled; these groups are then themselves scheduled at time bins or within schedule groups analogously to routines. The scheduling mechanism is described in more detail in section 9. The scheduling of routines may be made conditional on parameters having certain values by use of if statements in the CCL file. Memory for variables may be allocated throughout the course of the simulation, or allocated just during the execution of a particular scheduled routine or schedule group. Additionally thorn authors may define specific functions which they provide to other thorns or expect other thorns to provide. This provides an aliasing mechanism whereby many thorns may provide a function which may be called by another thorn with a particular name, with the choice of which one is actually called being deferred until run-time. In the absence of any active thorn providing such a function an error is returned. 8.2
CST
When the executable for any particular configuration is built, the Cactus Specification Tool (CST) is invoked to parse the thornlist and then the individual thorns’ CCL files. When the CST has parsed the CCL files it performs some consistency checks. These include: checking that all definitions of a particular implementation are consistent; checking that default values for parameters are within the allowed ranges; and checking that parameters shared from other implementations exist and have the correct types. Once the CCL files have been verified the CST uses the specifications from these files to create small C files which register information with the flesh about the thorns. This mechanism allows the flesh library to be completely independent of any thorns, while at the same time minimising the amount of machinegenerated code. Along with the registration routines the CST generates various macros which are used to provide argument lists to routines called from the scheduler, as well as wrapper routines to pass arguments to Fortran routines. The parameter specifications are turned into macros which place the parameters as local variables in any routine needing access to parameters. All the macros are generated on a thorn-by-thorn basis and contain only information relevant to that thorn. The schedule specifications are turned into registration routines which register relevant thorn routines with the scheduler.
The Cactus Framework and Toolkit: Design and Applications
9
211
The Scheduling Mechanism
Routines (or schedule groups – for scheduling purposes they are the same) scheduled from thorns may be scheduled before or after other routines from the same or other thorns, and while some condition is true. In order to keep the modularity, routines may be given an alias when they are scheduled, thus allowing all thorns providing the same implementation to schedule their own routine with a common name. Routines may also be scheduled with respect to routines which do not exist, thus allowing scheduling against routines from thorns or implementations which may not be active in all simulations. Additionally the schedule.ccl file may include ’if’ statements which only register routines with the scheduler if some condition involving parameters is true. Once all the routines have been registered with the scheduler, the before and after specifications form a directed acyclic graph, and a topological sort is carried out. Currently this is only done once, after all the thorns for this simulation have been activated and their parameters parsed. The while specification allows for a degree of dynamic control for the scheduling, based upon situations in the simulation, and allows looping. A routine may be scheduled to run while a particular integer grid scalar is non-zero. On exit from the routine this variable is checked, and if still true, the routine is run again. This is particularly useful for multi-stage time integration methods, such as the method of lines, which may schedule a schedule group in this manner. This scheduling mechanism is rule-based as opposed to script-based. There are plans to allow scripting as well; see section 17 for further discussion of this.
10
Memory Management and Parallelisation
In order to allow the use of the same application codes in uni-grid, parallel and adapted-grid modes, each routine called from the scheduler is assigned an ndimensional block of data to work on; this is the classical distributed computing paradigm for finite difference codes. In addition to the block of data and its size, information about each boundary is given, which specifies if this boundary is a boundary of the actual computational domain, or an internal grid boundary. The routine is passed one block of data for each variable which has storage assigned during this routine (see section 8.2 for argument list generation). The memory layout of arrays is that of Fortran, i.e. fastest-changing index first; in C these appear as one-dimensional arrays and macros are provided to convert an n-dimensional index into a linear offset withing the array. In order to isolate the flesh from decisions about block sizes, a particular class of thorns, which provide the DRIVER implementation, is used. Drivers are responsible for memory management for grid variables, and for all parallel operations, in response to requests from the scheduler. The driver is free to allocate memory for variables in whatever way is most appropriate, some possibilities for uni-grid are: having memory for the whole domain in one block, with this block being passed to routines; all the memory
212
Tom Goodale et al.
in one block, but passing sub-blocks to routines; splitting the domain into subdomains and processing these one by one, e.g. for out of core computation; or splitting into sub-domains and then processing sub-domains in parallel. Since the application routines just get such a block of data, data layout may change with time, and indeed will do with adaptive mesh refinement or dynamic load balancing. As the driver is responsible for memory management, it is also the place which holds the parallelisation logic. Cactus 4.0 supports three basic parallelisation operations: ghost-zone synchronisation between sub-domains; generalised reduction operators (operations which take one or more arrays of data and return one or more scalar values from data across the whole domain); and generalised interpolation operators (operations which take one or more arrays and a set of coordinates and return one or more scalars for each coordinate value). Synchronisation is called from the scheduler – when a routine or schedule group is scheduled it may list a set of grid variables which should be synchronised upon exit. Routines may also call synchronisation internally, however such routines would fail when more than one block of data is processed by a processor or thread, such as with adaptive mesh refinement or out of core computation, as the routine would not yet have been invoked on the remaining blocks so ghost-zone exchange would be impossible. Similarly the other operations – reduction and interpolation – being global in nature, should only be performed one per grid, however many sub-grids there are; routines which call these routines must be scheduled as “global”. This information is then passed to the driver which should then only invoke such routines once per processor, rather than once per sub-block per processor. The flesh guarantees the presence of these parallel operations with a function registration mechanism. A thorn may always call for synchronisation, reduction or interpolation, and these are then delegated to functions provided by the driver, or return appropriate error codes if no such function has been provided. These parallel operations are in principle all that an application programmer needs to think about as far as parallelisation is concerned. Such a programmer is insulated by the scheduler and by these functions of any specific details of the parallelisation layer; it should be immaterial at the application level whether the parallelisation is performed with MPI, PVM, CORBA, threads, SUN-RPC, or any other technology. Since all driver thorns provide the DRIVER implementation, they are interchangeable, no other thorn should be affected by exchanging one driver for another. Which driver is used is determined by which is activated at run-time. At the time of writing there are four driver thorns known to the authors: a simple example uni-grid, non-parallel one (SimpleDriver); a parallel uni-grid one (PUGH); a parallel fixed mesh refinement driver (Carpet) ; and a parallel AMR one (PAGH). PUGH is the most frequently used one, and is contained in the computational toolkit available from the Cactus web site; Carpet and PAGH have been developed independently.
The Cactus Framework and Toolkit: Design and Applications
10.1
213
PUGH
PUGH is the only driver currently distributed with the Cactus Computation Toolkit. This driver is MPI based, although it may also be compiled without MPI for single processor operation. In single-processor mode PUGH currently creates one block of data per variable and passes that through to routines; there has also been discussion of enhancing this to generate smaller sub-blocks tuned to the cache size to increase performance. In parallel mode PUGH decomposes the processors in the n-dimensions such that the smallest dimension has the fewest number of processors, which allows greater cache-efficiency when constructing messages to be passed to other processors, however this may be over-ridden by parameter setting which allow any or all of the dimensions to be set by hand. There are plans to add features to allow the processor topology to be optimised for node-based machines such as the IBM SP or Hitachi SR8000 which consist of small (typically 8 or 16 processor) nodes connected together to produce a larger machine. The load-balancing algorithm in PUGH is also customisable, and a modified version of PUGH has been used in large scale distributed computing [19]. Along with PUGH, the Computational Toolkit provides the auxiliary thorns PUGHReduce and PUGHInterp to provide reduction and interpolation when PUGH is used as a driver, and PUGHSlab, which provides the hyperslabbing capability used by the Toolkit’s IO thorns.
11
IO
Input and Output are vital to any simulation, and indeed to any framework. In keeping with the overall design of Cactus, the flesh has minimal knowledge of the IO, but provides a mechanism so that thorn authors can call for output of their variables and the appropriate IO routines will be called. All thorns providing IO routines register themselves with the flesh, saying they provide an IO method, which is a unique string identifier. Associated with an IO method are three functions: one to output all variables which need output; one to output a specific variable; and one to determine if a variable requires output. The first two of these routines then have analogues in the flesh which traverses all IO methods calling the appropriate routines, or call the routine corresponding to a specific IO method. Once per iteration the master evolution loop calls the routine to output all variables by all IO methods. Thorns providing IO methods typically have string parameters which list the variables which should be output, how frequently (i.e. how many iterations between output), and where the output should go. In order to simplify such thorns, and to provide standards for parameter names and meanings, the computational toolkit contains the IOUtil thorn. This thorn provides various default parameters such as the output directory, the number of iterations between output of variables, various down-sampling parameters and other parameters which may need to be used by thorns providing output of the same data but in different formats. It also provides common utility routines for IO.
214
Tom Goodale et al.
The computational toolkit contains thorns which output data to screen, to ASCII output files in either xgraph/ygraph [20], or GNUPlot [21] format, binary output in IEEEIO [22] or HDF5 [23] formats, or as jpegs of 2d slices of datasets. This data may also be streamed to external programs such as visualisation clients instead of or in addition to being written to disk. Streamed visualisation modules exist for Amira [24] and the widely available OpenDX [25] visualisation toolkits. Since all IO thorns must operate on m-dimensional slabs of the n-d data, which is (section 10) laid out and distributed in a driver dependent manner, there is also a defined interface provided by thorns providing the HYPERSLAB implementation. This interface is driver independent, and allows an IO thorn to get or modify a particular slab of data without being tied to a particular driver.
12
Utilities
The flesh provides sets of utility functions which thorns developers may use. These include common data storage utilities such as hash tables and binary trees; an arithmetical expression parser which may be used to perform calculations on data based upon an expression given as a string; access to regular expression functions (using the GNU regex library if the configure script does not find one); and an interface to create and manipulate string-keyed tables of data. Apart from these general utilities, the flesh also provides an infrastructure to time the execution of code blocks. This consists of a way to create and manipulate timer objects, each of which has an associated set of clocks, one for each timegathering function, such as the Unix getrusage call, the wall-clock time or the MPI Wtime function. These timers and their clocks may be switched on or off or reset as necessary. The scheduler in the flesh uses these functions to provide timing information for all scheduled functions.
13
Test Suites
An essential component of any code is a well-defined way to perform regression tests. This is especially true for a framework, where the final result of a set of operations may depend upon many independently written and maintained modules. Cactus includes an automated test-suite system which allows thorn authors to package parameter files and the results they are expected to produce with their thorns, and then allows the current code to be tested against these results. The test-suite compares numbers in the output files to a specified tolerance, which is generally set to the accuracy which can be expected given different machines’ round-off error and machine precision, but can be over-ridden by the thorn author if this measure is not appropriate.
14
Thorn Computational Infrastructure
The overriding design criterion for Cactus has been to put as little in the flesh as possible, which allows the maximum flexibility for users to develop new functionality or modify existing functionality. The corollary to this is that the flesh by
The Cactus Framework and Toolkit: Design and Applications
215
itself is not very useful. Sections 10 and 11 described two sets of infrastructure thorns, the driver and the IO thorns respectively. This section describes some of the other infrastructure which is available in the computational toolkit. 14.1
Interpolation and Reduction
While the flesh provides standard interfaces for generalised interpolation and reduction operations, and guarantees that these functions may always be called, the actual operations need to be done by the driver thorn (see section 10 , or by a closely associated thorn. A thorn providing such an operation registers a function with the flesh, with a unique name associated with the operator, such as “maximum”. A thorn which needs to interpolate data in a grid variable, or to perform a reduction on it, then calls the appropriate flesh function, with the name of the operation, and the flesh then delegates the call to the thorn providing the operation. The Computational Toolkit provides several thorns for operations such as maximum, L1 and L2 norms, parallel interpolation, etc. 14.2
Coordinates
A particular grid has a single coordinate system associated with it, out of the many possibilities, and each grid array may be associated with a different coordinate system. To facilitate the use of different coordinate systems we have a thorn providing the COORDBASE implementation, which defines an interface to register and retrieve coordinate information. This interface allows thorns providing coordinate systems to store information such as the coordinates of the boundaries of the computational grid in physical space, and the coordinates of the interior of the grid – i.e. that part of the grid which is not related to another by a symmetry operation or by a physical boundary condition. The thorn also associates details of how to calculate the coordinate value of a point with the coordinate system; this data may be calculated by origin and delta information, by associating a one-dimensional array with the coordinate direction to hold the coordinate data, or, for full generality, with an n-dimensional GA (or GF) where each point in this GA (or GF) holds the coordinate value for this coordinate for this point. Based upon this coordinate system information, IO methods may output appropriate coordinate information to their output files or streams. 14.3
Boundary Conditions
Boundary conditions are another basic feature of any simulation. They fall into two broad classes – symmetry and physical boundary conditions. Additionally boundary conditions in both these classes may be either local or global. For example periodic boundary conditions are global symmetry conditions, the Cartoon [26] boundary condition is a symmetry condition but is local, and radiative boundary conditions are physical and local. If all boundary conditions were local, there would be no problem calling them from within an application routine, however as soon as global conditions are
216
Tom Goodale et al.
used, calling from a routine suffers the same problems as any parallel operation. Additionally it is undesirable that routines should have to know anything about the symmetries of the grid – they should just know about the physical boundary conditions, and when these are applied the appropriate symmetries should be performed. In order to address these needs, we have defined a scheme whereby instead of calling the boundary conditions directly, calculation routines merely call a function to indicate that a particular variable requires a particular physical boundary condition, and then the thorn providing this routine schedules a schedule group called ApplyBCs after the calculation routine (schedule groups may be scheduled multiple times, so there is no problem with conflicts between multiple thorns scheduling this group). All symmetry boundary condition and global physical boundary condition thorns schedule a routine in the ApplyBCs group, and the Boundary thorn schedules a routine to apply local physical boundary conditions in the group as well. Thus when the ApplyBCs group is active, each routine scheduled in it examines the list of variables which have been marked as needing boundary conditions applied, and, if it is a symmetry routine applies the symmetry, or if it is a physical boundary condition and the variable requires that physical boundary condition, applies the appropriate physical boundary condition – local physical boundary conditions are registered with the Boundary thorn, which then uses the routine it has scheduled in the group to dispatch variables to the appropriate routine. This scheme allows new symmetry conditions to be added with no modification of any calculation routine, and allows these routines to work appropriately in the presence of multiple blocks of data per process or thread. 14.4
Elliptic Solvers
The computational toolkit provides a thorn Ell Base which provides an experimental interface for elliptic solvers. This thorn provides a set of registration functions to allow thorns which solve certain sets of elliptic equations to be accessed via a specified interface; e.g. a thorn wanting to solve Helmholtz’s equation makes a call to the function in Ell Base passing as one argument a well-known name of a particular solver, such as PETSc [12] , which is then invoked to perform the calculation. Thus the actual elliptic solver used may be decided at run-time by a parameter choice, and new elliptic solvers may be used with no change to the application thorn’s code. 14.5
Utility Thorns
In addition to necessities such as drivers, IO, coordinates and boundary conditions, there are various general utility thorns. For example we have a thorn which takes datasets and checks for NaNs; this can be either done periodically as set by a parameter, or called from other thorns. On finding a NaN the thorn can issue a warning, report the location of the NaN, stop the code, or any combination of these.
The Cactus Framework and Toolkit: Design and Applications
217
Another thorn, which is under development at the moment, interacts with the Performance API (PAPI) [27] to allow profiling of a simulation. PAPI allows access to hardware performance counters in a platform-independent manner.
15
External Interaction
The most basic way to run a simulation is to start it and then examine the data written to disk during or after the run. A framework, however, is free to provide modules to allow interaction with a running simulation. This may range from just querying the status of the simulation, through visualising data remotely, to steering the control parameters of a simulation based upon user input or predictions and analysis performed by other programs. 15.1
HTTPD
The Cactus Computational Toolkit contains a thorn, HTTPD (based on an original idea and implementation by Werner Benger), which acts as a web-server. The basic thorn allows users to connect to the running simulation with a web-browser and to examine and, if authenticated, to modify, the values of parameters. An authenticated user may also choose to terminate or pause the simulation; the user may also tell the code to pause at a specific iteration or after a certain amount of simulation time. The ubiquitous nature of web-browsers makes this an almost ideal way to interact with the simulation. Additional thorns may be used to provide further capabilities to the web server. For example, application thorns can also add their own information pages and provide custom interfaces to their parameters. The toolkit contains the thorn HTTPDExtra which provides access to the IO methods within the framework by providing a view port by which users may view two-dimensional slices through their data using IOJpeg, and through which any file known by the simulation may be downloaded. This allows users to examine the data produced by a remotely running simulation on the fly by streaming data to visualisation tools such as OpenDX [25], Amira [24] or GNUPlot [21], see section 15.2 The combination of the ability to pause the simulation, to examine data remotely, and to steer simulation parameters provides a powerful method for scientists to work with their simulations. This combination could be enhanced further to provide a debugging facility. Currently HTTPD only supports un-encrypted HTTP connections, however there are plans to enhance it to use HTTP over TLS [28] [29]/SSL [30] or over GSI [31], providing capabilities for secure authentication and encrypted sessions. 15.2
Remote Visualisation and Data Analysis
As mentioned above, the Cactus Computational Toolkit provides mechanisms to query and analyse data from a running simulation. While it is true that because the simulation may be scheduled in the batch queue at unpredictable times,
218
Tom Goodale et al.
the scientist will not necessarily be on hand to connect to the simulation at the moment that it is launched, the simulations that create the most egregious difficulties with turnaround times typically run for hours or days. So while the interactive analysis may not necessarily capture the entire run, it will intercept at least part of it. Any on-line analysis of the running job before the data gets archived to tertiary storage offers an opportunity to dramatically reduce the turn-around time for the analysis of the simulation outcome. Keeping up with the data production rate of these simulations is a tall order, whether the data is streamed directly to a visualisation client or pre-digested data is sent. Earlier pre-Cactus experiments with a circa 1992 CAVE application called the Cosmic Worm demonstrated the benefits of using a parallel isosurfacer located on another host to match this throughput. However, it also pointed out the fact that it doesn’t take too many processors limits primarily from the I/O rate are reached. It does no good to perform the isosurface any faster if its performance is dominated by the rate at which data can be delivered to it. For the first remote visualisation capability in Cactus, it was decided to implement a parallel isosurfacer in-situ with the simulation so that it would use the same domain decomposition and number of processors as the simulation itself. The isosurfacer would only send polygons to a very simple remote visualisation client that runs locally on the user’s desktop; offering sometimes orders of magnitude in data reduction with throughput that exactly matched the production rate of the simulation code. This paradigm was further enhanced to include geodesics and other geometric visualisation techniques. Other ways to reduce the throughput to data clients is to send only sub-sets of the data. The Cactus IO methods, in particular the HDF5 method, allow the data to be down-sampled before being written to disk or streamed to a visualisation client. Another way to stream less data is to pick a data-set of a lower dimension from the full data-set, and we have IO methods which will produce 1 or 2dimensional subsets of three-dimensional data-sets. These and other remote technologies are being actively developed through a number of projects for example [32, 33, 34].
16 16.1
Applications Numerical Methods
Although Cactus was designed to support, or to be extensible to support, different numerical methods, most of the existing infrastructure has been developed around regular structured meshes with a single physical domain and coordinate system, and more specifically for finite difference methods with spatial dimensions of three or less. This is well-suited to many calculations, particularly in the problem domains for which Cactus was first developed. However there are many other problem domains for which these restrictions pose problems. Fundamental support for other numerical methods typically involves the development of a standard driver and associated thorns (with possible extensions
The Cactus Framework and Toolkit: Design and Applications
219
of CCL features). It is hoped that many of the existing infrastructure thorns, such as those for IO and coordinates, can be developed to support additional methods. For example, the addition of a hyperslabbing thorn for a particular driver should allow for IO thorns to provide for different underlying methods. With the addition of appropriate drivers (see section 17), and associated driver infrastructure thorns, the Cactus framework can be used with structured, unstructured, regular and irregular meshes, and can implement finite differencing, finite volumes or finite elements, particle and spectral methods, as well as ray tracing, Monte Carlo etc. 16.2
Scalar Wave Equation
The solution of the scalar wave equation using 3D Cartesian coordinates provides a prototypical example of how a wide class of initial value problems can be implemented with finite differencing methods in Cactus. The CactusWave arrangement contains thorns which implement this solution in each of the currently supported programming languages, along with additional thorns providing initial data and source terms [13]. 16.3
General Relativity
The introduction describes the historical relationship between Cactus and numerical relativity. The requirements of this diverse community for collaboratively developing, running, analysing and visualising large-scale simulations continue to drive the development of Cactus, motivating computational science research into advanced visualisation and parallel IO as well as Grid computing. The Cactus framework and Computational Toolkit is used as the base environment for numerical relativity codes and projects in nearly two dozen distinct groups worldwide, including the 10-institution EU Network collaboration and others in Europe, as well as groups in the US, Mexico, South Africa, and Japan. In addition, the Cactus Einstein Toolkit provides an infrastructure and common tools for numerical relativity, and groups which choose to follow the same conventions can easily inter-operate and collaborate, sharing any of their own thorns, and testing and verifying thorns from other groups (and many do). The numerical relativity community’s use of Cactus provides an example of how a computational framework can become a focal point for a field. With many different relativity arrangements now freely available, both large groups and individuals can work in this field, concentrating on their physics rather than on computational science. In this environment, smaller groups are more easily able to become involved, as they can build on top of open and existing work of other groups, while concentrating on their particular expertise. For example, an individual of group with expertise in mathematical analysis of gravitational waves can easily implement thorns to do this, using other groups’ initial data and evolution thorns. Such an approach is increasingly becoming a point of entry into this field for groups around the world.
220
Tom Goodale et al.
In addition to the leverage and community building aspect of an open framework like Cactus, the abstractions it provides make it possible for a new technology, such as fixed or adaptive mesh refinement, improved IO, new data types, etc, to be added and made available to an entire community with minimal changes to their thorns. Not only does this have obvious benefits to the user communities, it has the added effect of attracting computational science groups to work with Cactus in developing their technologies! They find not only a rich application oriented environment that helps guide development of their tools, but they also find satisfaction in knowing their tools find actual use in other communities. 16.4
Other Fields
Although Cactus began as a project to support the relativity community it is an open framework for many applications in different disciplines. Here we simply list a few of a growing number of active projects of which we are aware at present. As part of an NSF project in the US, the Zeus suite of codes for Newtonian magnetohydrodynamics has been developed into a set of thorns (in the Zeus arrangement, which use AMR, and is publicly available. In another discipline, Cactus has been used as a framework to write parallel chemical reactor flow applications, where it was found to be an effective tool for speeding up simulations across multiple processors [35]. Climate modellers at NASA and in the Netherlands have also taken interest in Cactus, and are actively developing thorns for both shallow water models and a coupling of two ocean models [36]. Other application communities prototyping applications in Cactus, or investigating its use as a framework for their applications, include the fusion simulation community, avalanche simulators, and geophysics groups, to name a few. 16.5
Grid Computing
Last but not least, we turn to Cactus as a framework, not only for traditional applications, but also for developing applications that may run in a Grid environment. The Grid is an exciting new development in computing, promising to seamlessly connect computational resources across an organisation or across the world, and to enable new kinds of computational processes of unprecedented scales. As network speeds increase, the effective “distance” between computing devices decreases. By harnessing PCs, compute servers, file servers, handhelds, etc, distributed across many locations, but connected by ever better networks, dynamically configurable virtual computers will be created for many uses [1]. However, even if the networks, resources, and infrastructure are all functioning perfectly, the Grid presents both new challenges and new possibilities for applications. Although the applications of today may (or may well not) actually run on a Grid, in order to run efficiently, or more importantly, to take advantage of new classes of computational processes that will be possible in a Grid world,
The Cactus Framework and Toolkit: Design and Applications
221
applications must be retooled, or built anew. And yet, as we stressed in the introduction, most application developers and users want to focus on their scientific or engineering discipline, and not on the underlying computational technology. For the same reasons that Cactus has been an effective tool for building applications that take advantage of advanced computational science capabilities on many different computer architectures, while hiding many details from the user, it has also been one of the leading sources of early Grid applications. Its flexible and portable architecture make it a good match for the varying needs of Grid applications. In particular, it has traditionally been used for many remote and distributed computing demonstrations [37, 38, 39, 19]. Because of the abstractions available in Cactus, it has been possible to modify the various data distribution, message passing, and IO layers so they function efficiently in a distributed environment, without modifying the applications themselves [19]. For example, it has been possible to take very complex, production code for solving Einstein’s equations which is coupled to GR hydrodynamics to simulate colliding neutron stars, and distributed across multiple supercomputers on different continents, while it is remote visualised and controlled from yet another location. This is just the beginning of what can be done on future Grids. We imagine a world where applications are able autonomously to seek out resources, and respond to both their own changing needs and the changing characteristics of the Grid itself, as networks and computational resources change with time. Applications will be able not only to parallelise across different machines on the Grid, but will also be able spawn tasks, migrate from site to site to find more appropriate resources for their current task, acquire additional resources, move files, notify users of new events, etc [4]. Early prototypes of this kind of capability have already been developed in Cactus; the “Cactus Worm” demonstration of Supercomputing 2000 was a Cactus application (any application inserted in the Cactus framework would do) that was able to: run on any given site, make use of the existing Grid infrastructure to move itself to a new site on a Grid, register itself with a central tracker so its users could monitor or control it, and then move again if desired [40]. This demonstration of new types of computation in a Grid environment foreshadows a much more complex world of the future. However, the underlying Grid technology is quite complex, and varies from site to site. Even if an application programmer learns all the details of one Grid technology, and implement specific code to take advantage of it, the application would likely lose some degree of portability, crucial in the Grid world. Not only does one wish to have applications that run on any major computing site, with different versions of Grid infrastructure, but also one wants them to run on local laptops or mobile devices, which may have no Grid infrastructure at all! Learning from the Cactus Framework, one solution is to develop a Grid Application Toolkit (GAT), that abstracts various Grid services for the application developer, while inter-operating with any particular implementation. The GridLab project [41], is currently underway to do just this. The goal is to develop a toolkit for all applications, whether they are in the Cactus framework or not. The GAT will provide an abstracted view of Grid services of various kinds to the
222
Tom Goodale et al.
user/developer, and will dynamically discover which services are actually available that provide the desired functionality (e.g., “find resource”, “move file”, “spawn job”, etc). It is far beyond the scope of this article to go into further detail, but we refer the reader [4, 42, 43] for more detail.
17
Future Plans
Cactus 4.0 is a great step from previous versions of Cactus, providing much more modularity and flexibility. However there are numerous shortcomings, and many things are planned for future versions. Support for unstructured meshes is intended, which opens up the use of Cactus for finite volume and finite element calculations. We have had many discussions with communities, e.g. aerospace and earthquake simulation, which use these methods, and have well-developed plans on how to support such meshes. Another plan we have is to enable more complicated scenarios, consider the following: (i) Many CFD calculations also use multi-block grids – grids which are themselves made up of many sub-grids, some being structured and some unstructured, e.g. simulation of flow past an aircraft wing may use an overall structured mesh, with a small unstructured grid block around the wing and its stores; (ii) Climate modelling simulations have many distinct physical domains, such as sea, icebergs, land and air, each of which has its own physical model, and then interaction on the boundaries; (iii) Some astrophysical simulations require several coordinate patches to cover a surface or volume. These are all sub-classes of what we refer to as “multi-model”. While all can be done currently within Cactus using Grid Arrays, this is not the most natural way to provide this functionality, and requires more communication and coordination between individual module writers than is desirable. We have defined a specification to enable any combination of these types of scenarios and intend to enable this functionality in a future release. Currently all Grid Functions or Grid Arrays are distributed according to their dimension across all processors. This is the most common need, however there are many situations where one would like to define a variable which only lives on the boundary of a grid. This can be faked at the moment by defining a GA with one lower dimension than GFs and ignoring it on internal processors, however even then the distribution of this GA may not correspond to the distribution of a GF on that face. To solve this problem we intend to introduce a new class of grid objects, sub-GFs, which will have the appropriate distribution. Another class of methods not currently supported are particle methods, such as Particle-In-Cell (PIC) and Smoothed-Particle-Hydrodynamics (SPH). Both these methods require a small amount of additional communication infrastructure to enable exchange of particle information between different processors. Currently thorns can only be activated when the parameter file is read, and not during the course of the simulation. An obvious enhancement is to enable thorn activation, and even de-activation, at any point in the program execution.
The Cactus Framework and Toolkit: Design and Applications
223
This would allow the code to act as a compute server, receiving requests for simulations from external sources, and, if thorns were dynamically loadable libraries as opposed to statically linked, would allow new functionality to be incorporated at will into long-running simulations. Another enhancement would be to allow scripting as an alternative to the current scheduling mechanism. The current mechanism allows thorns to interoperate and for simulations to be performed with the logic of when things happen encapsulated in the schedule CCL file; other frameworks do the same thing by providing a scripting interface, which gives more complete control of the flow of execution, at the expense of the user needing to know more of the internals. Both schemes have advantages and disadvantages. In the future we would like to allow users to script operations using Perl, Python, or other scripting languages. Currently Cactus only allows thorns in C, C++, and Fortran. Addition of other languages is fairly straightforward as long as there exists a method to map between the language’s data and C data. We plan to add support for thorns written in Perl, Python and Java in the future; this will be done by defining an interface which would allow a thorn to inform the make system and the CST about the mappings and how to produce object files from a language’s source files, thus allowing support for any language to be added by writing an appropriate thorn. In section 6 it was stated that Cactus is not installed. In the future we would like to support the installation of the Cactus flesh and the Computational Toolkit arrangements in a central location, with other arrangements installed elsewhere, for example in users’ home directories, thus providing a consistent checked out version of Cactus across all users on a machine. In principle it would also be possible for configurations to be centralised to some extent – the object files and libraries associated with the Computational Toolkit for a particular configuration could be provided by the system administrator, and only the object files and libraries of user-local arrangements would then need to be compiled by the user. However, given the dynamic nature of code development it is not clear that such centralised configurations would be practicable. All the above require modifications to the flesh. However the primary method to add functionality is by adding new thorns. One basic feature intended for the future is to develop a specification for a set of parallel operation, the Cactus Communication Infrastructure (CCI) which will abstract the most commonly used parallel operations, and provide a set of thorns which implement this infrastructure using common communication libraries such as MPI, PVM, CORBA, SUN-RPC, etc. This would then allow driver thorns themselves to be built independently of the underlying parallel infrastructure, and greatly increase the number of parallel operations available to other thorns. Another set of useful functionality would be to develop remote debuggers and profilers based upon the current HTTPD thorn and the expression parser. Indeed HTTP is far from being the only possible way to interact with a remote code, and thorns could be written to present allow interaction with a simulation via
224
Tom Goodale et al.
a SQL or LDAP interface, or as a web-service; there is already work on steering directly from visualisation clients using the streamed HDF5 interface [44]. Multi-grid methods are commonly used to solve elliptic equations, however all current methods of putting elliptic solvers into Cactus require knowledge of the driver or at least of the parallelisation technology. It should be possible to develop a set of thorns which use the scheduler in a driver and parallel-layer independent manner to solve elliptic problems. The current set of thorns allows IO in many formats, however it is straightforward to add other formats, and there is at least one group with plans to write a NetCDF IO thorn. Additionally the infrastructure for IO makes it plausible to write IO methods that do transformations on the data first, e.g. spectral decomposition, before writing them out to disk or a stream; such methods would need to call routines from lower-level IO methods, and we have designed an interface which presents lower-level IO methods as stream-equivalents, thus insulating the higher-level IO methods from knowledge of specific lower-level IO methods.
Acknowledgements Cactus has evolved and been developed over many years, with contributions from a great many people. Originally an outgrowth of work in the NCSA numerical relativity group in the early 1990’s, the first framework with the name Cactus was developed at the Max Planck Institute for Gravitational Physics by Paul Walker and Joan Mass´ o. Its creation and further development was influenced greatly, and directly contributed to, by many friends, especially Werner Benger, Bernd Br¨ ugmann, Thomas Dramlitsch, Mark Miller, Manish Parashar, David Rideout, Matei Ripeanu, Erik Schnetter, Jonathan Thornburg, Malcolm Tobias and the numerical relativity groups at the Albert Einstein Institute and Washington University. It has also been generously supported by the MPG, NCSA, Intel, SGI, Sun, and Compaq, and by grants from the DFN-Verein (TiKSL, GriKSL), Microsoft, NASA, and the US NSF grant PHY-9979985 (ASC).
References [1] Foster, I.: (2002). The Grid: A New Infrastructure for 21st Century Science. Physics Today, Febrary. 198, 199, 220 [2] Component Component Architecture (CCA) Home Page http://www.cca-forum.org. 199 [3] Cactus Numerical Relativity Community http://www.cactuscode.org/Community/Relativity.html. 199 [4] Allen, G., Seidel, E., and Shalf, J.: (2002). Scientific Computing on the Grid. Byte, Spring. 199, 221, 222 [5] Cactus: http://www.cactuscode.org. 199 [6] Cactus Users Guide http://www.cactuscode.org/Guides/Stable/UsersGuide/UsersGuideStable.pdf. 199, 206
The Cactus Framework and Toolkit: Design and Applications
225
[7] Allen, G., Benger, W., Dramlitsch, T., Goodale, T., Hege, H., Lanfermann, G., Merzky, A., Radke, T., and Seidel, E.: (2001). Cactus Grid Computing: Review of Current Development. In R. Sakellariou, J. Keane, J. Gurd, and L. Freeman, editors, Europar 2001: Parallel Processing, Proceedings of 7th International Conference Manchester, UK August 28-31, 2001 . Springer. 199 [8] Parashar, M. and Browne, J. C.: (2000). IMA Volume on Structured Adaptive Mesh Refinement (SAMR) Grid Methods, chapter System Engineering for High Performance Computing Software: The HDDA/DAGH Infrastructure for Implementation of Parallel Structured Adaptive Mesh Refinement, pages 1–18. Springer-Verlag. 200 [9] Grid Adaptive Computational Engine (GrACE) http://www.caip.rutgers.edu/~parashar/TASSL/Projects/GrACE/. 200 [10] Anninos, P., Camarda, K., Mass´ o, J., Seidel, E., Suen, W.-M., and Towns, J.: (1995). Three-dimensional numerical relativity: The evolution of black holes. Phys. Rev. D, 52(4):2059–2082. 200 [11] Walker, P.: (1998). Horizons, Hyperbolic Systems, and Inner Boundary Conditions in Numerical Relativity. Ph.D. thesis, University of Illinois at Urbana-Champaign, Urbana, Illinois. 200 [12] Balay, S., Gropp, W., McInnes, L. C., and Smith, B.: (1998). PETSc - The Portable, Extensible Toolkit for Scientific Compuation. http://www.mcs.anl.gov/petsc/. 201, 216 [13] Cactus Computational Toolkit Thorn Guide http://www.cactuscode.org/Guides/Stable/ ThornGuide/ThornGuideStable.pdf. 201, 206, 219 [14] TASC: Software Engineering Support of the Third Round of Scientific Grand Challenge Investigations: Task 4, Review of Current Frameworks http://sdcd.gsfc.nasa.gov/ESS/esmf tasc/t400.html. 203 [15] Cactus Maintainers Guide http://www.cactuscode.org/Guides/Stable/ MaintGuide/MaintGuideStable.pdf. 204, 206 [16] Concurrent Versions System (CVS) Home Page http://www.cvshome.org/. 206 [17] Cygnus Solutions (Cygwin) Home Page: http://www.cygwin.com. 206 [18] The Autoconf Home Page http://www.gnu.org/software/ autoconf/autoconf.html. 207 [19] Allen, G., Dramlitsch, T., Foster, I., Karonis, N., Ripeanu, M., Seidel, E., and Toonen, B.: (2001). Supporting Efficicient Execution in Heterogeneous Distributed Computing Environments with Cactus and Globus. In Proceedings of Supercomputing 2001, Denver . http://www.cactuscode.org/Papers/GordonBell 2001.ps.gz. 213, 221 [20] The xgraph and ygraph Home Pages http://jean-luc.aei-potsdam.mpg.de/Codes/xgraph, http://www.aei.mpg.de/~pollney/ygraph. 214 [21] GNUPlot Home Page http://www.gnuplot.org. 214, 217 [22] IEEEIO Home Page http://zeus.ncsa.uiuc.edu/~jshalf/FlexIO/. 214 [23] Hierarchical Data Format Version 5 (HDF5) Home Page http://hdf.ncsa.uiuc.edu/HDF5. 214 [24] Amira - Users Guide and Reference Manual and AmiraDev - Programmers Guide, Konrad-Zuse-Zentrum f¨ ur Informationstechnik Berlin (ZIB) and Indeed - Visual Concepts, Berlin, http://amira.zib.de. 214, 217 [25] Open DX Home Page http://www.opendx.org. 214, 217
226
Tom Goodale et al.
[26] Alcubierre, M., Brandt, S., Br¨ ugmann, B., Holz, D., Seidel, E., Takahashi, R., and Thornburg, J.: (2001). Symmetry without Symmetry: Numerical Simulation of Axisymmetric Systems using Cartesian Grids. Int. J. Mod. Phys. D, 10:273–289. Gr-qc/9908012. 215 [27] The Performance API (PAPI) Home Page http://icl.cs.utk.edu/projects/papi/. 217 [28] Dierks, T. and Allen, C.: The TLS Protocol Version 1.0: RFC 2246 January 1999. 217 [29] Rescorla, E.: HTTP Over TLS: RFC 2818 May 2000. 217 [30] Frier, A., Karlton, P., and Kocher, P.: The SSL 3.0 Protocol. Netscape Communications Corp. November 18, 1996. 217 [31] Foster, I., Kesselman, C., Tsudik, G., and Tuecke, S.: (1998). A Security Architecture for Computational Grids. In Proceedings of 5th ACM Conference on Computer and Communications Security Conference, pages 83–92. 217 [32] DFN Gigabit Project “Tele-Immersion: Collision of Black Holes” (TIKSL) Home Page http://www.zib.de/Visual/projects/TIKSL. 218 [33] DFN-Verein Project “Development of Grid Based Simulation and Visualization Techniques” (GRIKSL) Home Page http://www.griksl.org. 218 [34] Shalf, J. and Bethel, E. W.: Cactus and Visapult: An Ultra-High Performance Grid-Distributed Visualization Architecture Using Connectionless Protocols. Submitted to IEEE Computer Graphics and Animation. Submitted. 218 [35] Camarda, K., He, Y., and Bishop, K. A.: (2001). A Parallel Chemical Reactor Simulation Using Cactus. In Proceedings of Linux Clusters: The HPC Revolution, NCSA 2001 . 220 [36] Dijkstra, H. A., Oksuzoglu, H., Wubs, F. W., and Botta, F. F.: (2001). A Fully Implicit Model of the three-dimensional thermohaline ocean circulation. Journal of Computational Physics, 173:685715. 220 [37] Allen, G., Goodale, T., Mass´ o, J., and Seidel, E.: (1999). The Cactus Computational Toolkit and Using Distributed Computing to Collide Neutron Stars. In Proceedings of Eighth IEEE International Symposium on High Performance Distributed Computing, HPDC-8, Redondo Beach, 1999 . IEEE Computer Society. 221 [38] Allen, G., Benger, W., Goodale, T., Hege, H., Lanfermann, G., Merzky, A., Radke, T., and Seidel, E.: (2000). The Cactus Code: A Problem Solving Environment for the Grid. In Proceedings of Ninth IEEE International Symposium on High Performance Distributed Computing, HPDC-9, Pittsburgh. 221 [39] Benger, W., Foster, I., Novotny, J., Seidel, E., Shalf, J., Smith, W., and Walker, P.: (March 1999). Numerical Relativity in a Distributed Environment. In Proceedings of the Ninth SIAM Conference on Parallel Processing for Scientific Computing. 221 [40] Allen, G., Dramlitsch, T., Goodale, T., Lanfermann, G., Radke, T., Seidel, E., Kielmann, T., Verstoep, K., Balaton, Z., Kacsuk, P., Szalai, F., Gehring, J., Keller, A., Streit, A., Matyska, L., Ruda, M., Krenek, A., Frese, H., Knipp, H., Merzky, A., Reinefeld, A., Schintke, F., Ludwiczak, B., Nabrzyski, J., Pukacki, J., Kersken, H.-P., and Russell, M.: (2001). Early Experiences with the EGrid Testbed. In IEEE International Symposium on Cluster Computing and the Grid. Available at http://www.cactuscode.org/Papers/CCGrid 2001.pdf.gz. 221 [41] GridLab: A Grid Application Toolkit and Testbed Project Home Page http://www.gridlab.org. 221
The Cactus Framework and Toolkit: Design and Applications
227
[42] Allen, G., Goodale, T., Russell, M., Seidel, E., and Shalf, J.: (2003). Grid Computing: Making the Global Infrastructure a Reality, chapter Classifying and Enabling Grid Applications. Wiley. 222 [43] Seidel, E., Allen, G., Merzky, A., and Nabrzyski, J.: (2002). GridLab — A Grid Application Toolkit and Testbed. Future Generation Computer Systems, 18:1143– 1153. 222 [44] Allen, G., Benger, W., Goodale, T., Hege, H., Lanfermann, G., Merzky, A., Radke, T., Seidel, E., and Shalf, J.: (2001). Cactus Tools for Grid Applications. Cluster Computing, 4:179–188. http://www.cactuscode.org/Papers/CactusTools.ps.gz. 224
Performance of Message-Passing MATLAB Toolboxes Javier Fern´ andez , Antonio Ca˜ nas , Antonio F. D´ıaz , Jes´ us Gonz´alez, Julio Ortega , and Alberto Prieto Dept. of Computer Technology and Architecture ETSII, University of Granada, 18071-Granada, Spain {javier,acanas,afdiaz,jesus,julio,aprieto}@atc.ugr.es Tele: +34 958 248994 Fax: +34 958 248993
Abstract. In this work we compare some of the freely available parallel Toolboxes for Matlab, which differ in purpose and implementation details: while DP-Toolbox and MultiMatlab offer a higher-level parallel environment, the goals of PVMTB and MPITB, developed by us [7], are to closely adhere to the PVM system and MPI standard, respectively. DP-Toolbox is also based on PVM, and MultiMatlab on MPI. These Toolboxes allow the user to build a parallel application under the rapid-prototyping Matlab environment. The differences between them are illustrated by means of a performance test and a simple case study frequently found in the literature. Thus, depending on the preferred message-passing software and the performance requirements of the application, the user can either choose a higher-level Toolbox and benefit from easier coding, or directly interface the message-passing routines and benefit from greater control and performance. Topics: Problem Solving Environments, Parallel and Distributed Computing, Cluster and Grid Computing.
1
Introduction
Matlab has established itself as a de-facto standard for engineering and scientific computing. The underlying high-quality state-of-the-art numerical routines, the interactive environment, matrix-oriented language and graphical and visualization tools are some of the most frequently cited reasons for its widespread use [18]. Over 400 Matlab-based books covering virtually any topic [16] and a user community of more than 500.000 [17] account for two more reasons attracting new users to this scientific computing environment. In recent years a number of parallel Toolboxes for Matlab have been made available by researchers and institutions (see [7, Table 1], and [3] for a survey). The timeliness of such Toolboxes is evident, taking into account the rapid popularization of clusters of computers as a valid and cost-effective Parallel Computing platform, and the popularity of Matlab as a software prototyping tool.
Supported in part by the Spanish MCYT TIC2000-1348 project.
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 228–242, 2003. c Springer-Verlag Berlin Heidelberg 2003
Performance of Message-Passing MATLAB Toolboxes
229
Following the cluster computing trend, many standard Linux distributions (RedHat, Suse) now include the PVM [11] and LAM/MPI [15, 14] systems as regular packages, making it yet easier for cluster administrators to provide messagepassing environments to their users. Together with some of the Toolboxes considered here, the Matlab environment could then be used as a fast learning tool for the corresponding message-passing library, due to its interactive nature. In this sense, these Toolboxes make it easier for the extended Matlab user base to turn to Parallel Computing. On the other hand, previous knowledge of the message-passing library is of course reusable. Even the highest-level Toolbox commands can be conceptually related to the underlying library calls. For instance, the Shift command in MultiMatlab can be related to the MPI_Cart_Shift() and MPI_Sendrecv() routines. An experienced C/MPI developer might consider prototyping the desired application under Matlab, taking advantage of its sophisticated numeric and graphic routines, and of its interactive nature as an additional debugging facility, in order to translate (perhaps compile) the final debugged prototype back into C language. Finally, these Toolboxes can be used merely to reduce the execution time of sequential code in the interpreted Matlab environment by explicit (messagepassing) parallelization. The medium-coarse granularity of many Matlab prototypes makes them suitable for the so-called outer loop [19] parallelization, in agreement with the manufacturer’s opinion on the possible uses of parallelism under Matlab. Computer and interconnect performances, as well as per-node memory size, have all increased since [19] was written, but it remains true that SMP parallelism is better managed by the OS (and possibly a machine-tuned BLAS, see [20]) than by Matlab, and that the (explicit) parallelization effort should address the coarsest granularity available in the prototype, which is usually well suited to the message-passing paradigm in clusters of computers. In Section 2 the four Toolboxes considered here are briefly described. Section 3 shows the results of a ping-pong test programmed and run under each Toolbox, and Section 4 evaluates their performance on a case study. Conclusions are summarized in Section 5.
2
Message-Passing Matlab Toolboxes
Table 1 lists the Toolboxes studied in this work. All of them use the Matlab ability of calling externally compiled C or FORTRAN programs (MEX-files) in order to gain access to the PVM or MPI library routines, offer a (growing with date) number of commands and interfaced library calls, and provide transparent support for most Matlab data-types. The user is expected to issue a spawn command (Start in MultiMatlab) to start remote Matlab processes which cooperate in a parallel application by explicit message-passing. Additional information on these Toolboxes is related below: – The Distributed Parallel Toolbox [21] is a multi-layered project whose lowlevel layer (dplow, 56 commands) interfaces around 47 PVM routines. It
230
Javier Fern´ andez et al.
Table 1. Message-Passing Matlab Toolboxes Toolbox library cmds date URL DP-TB PVM 56 Mar–99 http://www-at.e-technik.uni-rostock.de/dp MultiMatlab MPI 27 Nov–97 http://www.cs.cornell.edu/Info/People/lnt/ /multimatlab.html PVMTB PVM 93 Jun–99 http://atc.ugr.es/javier-bin/pvmtb eng MPITB MPI 153 Feb–00 http://atc.ugr.es/javier-bin/mpitb eng
was developed at Rostock University. It also includes an intermediate layer (dphigh) and a high level environment (DP-MM), consisting of 15 commands, which means the user does not need to learn PVM. The group library has not been interfaced, so collective functionality (including pvm_barrier and pvm_reduce) is missing, but could be implemented in terms of the basic DP-TB routines if required. Coverage of the PVM library is greater than 50%. DP-TB works under both Windows and Linux. – MultiMatlab [23, 2, 18] is a high level Toolbox which frees the user from learning MPICH. Around 20 of its 27 commands correspond to MPI bindings. It was developed at Cornell University, with cooperation from The MathWorks. CTC is now working on CMTM [4], which runs under Windows and MPI/Pro, thus covering MPI-based Matlab demand on this platform. – PVMTB [7, 9] is a complete interface to PVM, including library calls such as pvm_spawn and pvm_tickle. It also interfaces the TEV_MASK macros and allows Matlab programmed hooks to be registered as hoster, tasker, resource manager or receive functions. – MPITB [7, 8] is an almost complete interface to the MPI 1.2 standard, excluding the MPI_Type_*, MPI_Op_* and MPI_Pcontrol bindings. It also includes several MPI 2.0 bindings required to spawn instances and communicate with them —MPI_Info_*, MPI_Comm_spawn_*, MPI_Comm_get_parent, MPI_Comm_accept, MPI_Comm_[dis]connect, MPI_*_port and MPI_*_name. Figure 1 shows the role of the MEX interface in these Toolboxes. Both DPTB and MultiMatlab utilize the directory technique (called indirection table in [18]), in which a centralized MEX-file is linked to the message-passing library and provides entry points (table offsets) which are either library call wrappers (e.g. pvm_config, Nproc) or more highly elaborated routines (pvme_upkarray and Recv do support Matlab data-types, for instance). Strictly speaking, only the directory MEX-file in DP-TB and MultiMatlab sits on top of the messagepassing libraries in Fig. 1. Data is comfortably received in the left-hand side of the Matlab call, as in DP-TB’s [data,info]=pvm_upkdouble(nitem,stride) or MultiMatlab’s [data,src,tag]=Recv(rank), allowing for compact Matlab expressions such as Sum=Sum+Recv(i).
Performance of Message-Passing MATLAB Toolboxes
231
Matlab Application PVM Application
DP-MM, dphigh
MPI Application
dplow, PVMTB PVM
MultiMatlab, MPITB Matlab
MPI
Operating System Network
Fig. 1. Overview diagram of Toolbox role. The Toolbox calls PVM/MPI to provide message-passing functionality, like any other PVM/MPI application. It also calls the Matlab API to pass arguments from/to the Matlab process. Several Matlab processes run on distinct cluster computers, each one being an instance of this diagram
Clearly, commands such as MultiMatlab’s Eval (or DP-TB’s dpmmeval) imply coordination with slave processes, which must in some way be enforced at slave process startup. DP-TB uses a MEX starter code (dpmatlab.mex) for this purpose, equivalent to the MultiMatlab’s “separate top-level MEX routine” [18]. In PVMTB and MPITB designs [7] each message-passing routine has its corresponding MEX wrapper. The libraries have been dynamically linked to avoid disk waste. Source code for the MEX wrappers is extensively reutilized. Commitment to the original routine parameter list means that PVMTB has commands such as info=pvm_upkdouble(var,nitem,stride), and MPITB has [info,stat]=MPI_Recv(var,src,tag,comm). This requires a more elaborate user code, since var is not returned on the left-hand side. Compact Matlab expressions such as the accumulation shown above must be broken down into two steps (receive-add). However, using this syntax no new storage is allocated for var, avoiding additional memory operations in applications where the same storage can be reutilized many times. With our Toolboxes, Matlab users can rely on the provided startup Mfile for slave Matlab processes, or design their own M-file starter codes in the interpreted Matlab language (not MEX-files), suited to their own applications. PVM hoster, tasker, resource manager, message handlers and receive functions, as well as MPI error handlers and attribute copy/delete functions, can also be programmed in Matlab language and registered using the corresponding PVMTB or MPITB commands.
232
2.1
Javier Fern´ andez et al.
Spawning Matlab Instances with PVMTB and MPITB
Each PVM-spawned child process has a PVMEPID environment variable (explained in man pvm_reg_tasker(3PVM)) whose existence can be checked in the Matlab’s startup.m file: addpath ( <path to PVMTB> ); if~isempty(getenv(’PVMEPID’)), startup_Slave; else, disp(’Help on PVM: help pvm’) end
The same role is played by the LAMPARENT environment variable in MPITB. Spawned Matlab instances (Fig. 2(c)) can thus be discriminated from the parent instance launched from the user shell (Fig. 2(a)), and forced to perform an additional startup script —startup_Slave in the example above. This script can implement any desired protocol, and is written in Matlab language. Using the PVMTB commands pvm_start_pvmd (Fig. 2(b)) and pvm_spawn (Fig. 2(c)), the Matlab user can start the Parallel Virtual Machine and children Matlab processes from within Matlab. MPITB users should invoke lamboot from the shell prior to issuing MPI_Init and MPI_Comm_spawn from within Matlab—since the LAM daemons are not part of the MPI standard, there are no MPI commands to start them. Most applications only need to specify one or more commands to be executed in the slave instance (Fig. 2(d)). The commands would be sent as strings that could be run on slave computers by using the Matlab eval built-in. Thus, startup_Slave could be written as: global TAG DEFAULT RAW PLACE, parent=pvm_parent; hostname pvm_recv(parent,TAG); pvm_unpack; for PVMLoopIdx=1:NUMCMDS pvm_recv(parent,TAG); pvm_unpack; eval(cmd); end if QUIT, quit, end
TAG=7; DEFAULT=0; RAW=1; PLACE=2; % show hostname if "xterm -e matlab" % parent should send NUMCMDS & QUIT % and NUMCMDS strings called "cmd" % containing desired MATLAB commands
A message tag number is reserved for PVMTB use. The parent Matlab should send two variables to each spawned Matlab instance: NUMCMDS, indicating the desired number of commands to execute, and QUIT, to finish Matlab children after the commands are executed. This flag can be cleared while developing or debugging and set when the prototype is finished. It is also customary to spawn xterm -e matlab during the debugging phase, in order to use Matlab’s built-in debugger on children instances. The pvm_unpack Toolbox command, when called without arguments, will unpack all variables packed in the current receive buffer using their original
Performance of Message-Passing MATLAB Toolboxes host A startup.m PVMTB Matlab OS NET
host B
OS NET
(a) User starts Matlab in host A. The startup.m code adds PVMTB to Matlab path. PVMEPID is absent.
host A start pvmd(B) PVMTB PVM Matlab OS NET
233
host B pvmd PVM OS NET
(b) User starts PVM from within Matlab, adding B to the Virtual Machine. The PVM daemon in B is started via an rsh connection.
host A spawn(matlab) PVMTB PVM Matlab OS NET
host B startup Slave.m PVMTB Matlab PVM OS NET
host A host B init/pack/send NCMD recv/upk/eval QUIT PVMTB PVMTB ✲ PVM Matlab Matlab PVM cmd ... OS cmd OS NET NET
(c) The spawn the pvmd in B which in turn The PVMEPID is detected and sourced.
command causes to run Matlab, sources startup.m. environment var. startup Slave.m is
(d) Matlab at B receives NUMCMDS and QUIT, waits for cmd strings, evaluates them as they are received and finally quits if instructed to do so by the QUIT flag.
Fig. 2. Sample control protocol for spawned Matlab instances names. So the master is expected to send NUMCMDS times a string called cmd which will be evaluated at the child instance. The default startup_Slave provided with PVMTB is more elaborate, using a 5s-timed pvm_trecv to avoid blocking interactively spawned children (useful for the tutorial), and a conditional branch for NUMCMDS==0, so that the loop only ends when cmd==’break’|’quit’|’exit’ (useful for applications where the total number of commands is not known beforehand). The startup_Slave for MPITB is even more elaborate, building a new intercommunicator which includes the parent and spawned Matlab processes. The 5s-timed receive is implemented using MPI_Wtime and MPI_Iprobe. PVMTB or MPITB users might implement any other conceivable protocol in their startup files, since any desired PVM or MPI call has been interfaced and is available. DP-TB users can reproduce the protocols described here, since all the required PVM calls are interfaced in dplow. A different protocol is used in the following sections, since the child(ren) will run just one (and the same) command. The startup code can be simplified, exporting the command string from the parent as an environment variable under PVMTB/DP-TB, or broadcasting it under MPITB. MultiMatlab’s startup
234
Javier Fern´ andez et al.
protocol (included in the top-level C-MEX routine) cannot be avoided or modified using Toolbox commands.
3
Performance Test
The Toolboxes were tested in a cluster of 8 Pentium II 333MHz PCs interconnected with a 100Mbps Fast Ethernet switch. A ping-pong test was programmed under each Toolbox, consisting of two M-files: a master Ping.m to be run in the source computer, and the slave Pong.m script for the echo Matlab process. In DP-TB and PVMTB, the Ping.m script starts the PVM daemon using pvm[e]_start_pvmd, the child Matlab process is run using pvm_spawn, and the command (Pong) is passed in an environment variable with pvm_export(’CMD’). This variable is detected and evaluated by the child Matlab in its startup.m script, i.e., the DP-MM starter is not used. Neither are the PVMTB pvm_psend and pvm_precv commands used; the objective is to produce almost identical code for both Toolboxes. The direct TCP route is prescribed using pvm_setopt. The pvm_barrier command in the PVMTB Ping script fragment shown below has been replaced by pvm_recv/send in DP-TB. for SZ=0:ND-1, array=zeros(SZ,1); % for each size allocate SZx1 vector pvm_barrier(’ppong’,2); T=clock; % synch before chrono starts for i=1:NTIMES % msg-pass loop pvm_initsend(PLACE); pvm_pkdouble(array); pvm_send(tid,TAG); pvm_recv(tid,TAG); pvm_upkdouble(array); end T=etime(clock,T); Data(SZ+1).time=T/2/NTIMES; S=whos(’array’); Data(SZ+1).size=S.bytes; clear array end % for SZ
The pvm_initsend command specifies in-place data encoding. The Pong script swaps the order of the two innermost lines (send/recv). The for loop is repeated NTIMES=10 times to reduce loop initialization and clock resolution influence on the results. The round-trip time T/NTIMES is halved to compute the one-way (transmit, ping) time. According to [13] (and common practice), the test should have been repeated a number of times in order to obtain reproducible figures (minimum of averages). However, the peaks shown in Fig. 3 can be easily identified and suggest the typical distribution and magnitude of the perturbations that can be expected. The MPITB code is very similar, using MPI_Spawn to start the Matlab child and MPI_Intercomm_merge to create a common intracommunicator. The command is explicitly passed as a string using MPI_Bcast in the new communicator. The child’s startup.m script cooperates in the collective merge and broadcast operations, and then the command string (Pong) is evaluated. The Ping loop shown below is a direct translation of the former PVM fragment to MPI:
Performance of Message-Passing MATLAB Toolboxes
235
MPI_Barrier(NEWORLD); T=clock; % synch before chrono starts for i=1:NTIMES % msg-pass loop MPI_Send(array,1,TAG,NEWORLD); MPI_Recv(array,1,TAG,NEWORLD); end
LAM client-to-client mode and homogeneous operation is prescribed adjusting the LAM TROLLIUSRTF environment variable before MPI_Init invocation.
PingPong Test
−3
x 10
DPTB MultiMATLAB PVMTB MPITB PVM MPI
1.2
Transmit time (s)
1
0.8
0.6
0.4
0.2
0
0
500
1000
1500
Message size (bytes)
Fig. 3. Results of the ping-pong test Table 2. Toolbox overhead in the 0–1500 bytes range, segregated by library Toolbox/library DP-TB MultiMatlab PVMTB MPITB PVM MPI
0B overhead 1500B overhead 676 µs 3.350 1047 µs 1.837 369 µs 3.037 691 µs 1.384 369 µs 1.827 692 µs 1.214 227 µs 1.871 588 µs 1.179 202 µs 1.000 570 µs 1.000 121 µs 1.000 499 µs 1.000
236
Javier Fern´ andez et al.
The MultiMatlab code is more compact, requiring only a Start command to launch the echo Matlab process, which becomes the target for the Eval command without any additional startup.m programming. Eval(1,[’Pong(’ int2str(ND) ’,’ int2str(NTIMES) ’)’]); for SZ=0:ND-1, array=zeros(SZ,1); % for each size allocate SZx1 vector Barrier; T=clock; % synch before chrono starts for i=1:NTIMES % msg-pass loop Send(1,array); array=Recv(1); end T=etime(clock,T); Data(SZ+1).time=T/2/NTIMES; S=whos(’array’); Data(SZ+1).size=S.bytes; clear array end % for SZ
The tests were run up to 1500 bytes (ND=188), and the results are shown in Fig. 3, which also includes C language tests for both PVM and LAM/MPI. In our setup, LAM has the shortest transmit time, followed by PVM and MPITB, by PVMTB and MultiMatlab, and by DP-TB. PVMTB and MPITB overheads are minimum in the sense that they only add a MEX-file call and input/output argument processing to the respective library call. Segregating by library, MultiMatlab doubles the MPITB overhead on MPI (18%–38%), and DP-TB overhead is 3–4 times that of PVMTB (21%–84%), due to the additional directory look-up and memory operations required to return the received data on the left-hand side. Table 2 summarizes these conclusions.
4
Application Example
Simple tests like the one above can be misleading. Gropp and Lusk explain in [13] how, as communication software becomes more sophisticated, simple tests become less indicative of the performance of a full application. Some perils of performance measurement are described in a “list of mistakes often made”. In this section we compute π by numeric integration. This example, taken from [22] and cited in [5], is proposed as an exercise in many textbooks. Even this simple, embarrassingly parallel application brings into discussion some of the common pitfalls described in [13], such as: – – – –
Ignore overlap of computation and communication. Confuse total bandwidth with point-to-point bandwidth. Ignore cache effects. Use a communication pattern different from the application, measure with just two processors, or measure with a single communication pattern.
Another advantage of this example is that we can show all the relevant Matlab code without getting involved in complex code explanations. Interested readers can find more complex applications in [6] (Control Problems), [7] (Wavelet
Performance of Message-Passing MATLAB Toolboxes
237
Analysis), [10] (Chemical Manufacturing) and [12] (NanoElectronics). In particular, the latter paper includes a detailed scattering model with efficiencies ranging from 98% (2 CPUs, 1.96x speedup) to 84% (120 CPUs, 101x speedup). The Parallel NANOMOS simulator used to implement this model was built on top of PVMTB and is available from the CELAB CVS repository [12]. Returning to the π example, the C sequential code proposed in [5] applies the mid-point integration rule to arctan in [0, 1] (there are faster, more accurate methods for approximating π [1]): width = 1.0 / N; sum = 0; % subdivision width for (i=0; i
In this example we chose N=408800 subdivisions of the [0, 1] interval, accounting for 13 decimal digits of precision and around 0.5s computation time under Matlab. The loop can be trivially staggered among C computers, making each of them compute a subset for (i=inum-1; i
DP-TB allows a more compact Psum=Psum+pvm_upkdouble(1,1) expression. As in the ping-pong test, group functionality (pvm_joingroup / pvm_reduce in this case) was not used under PVMTB to avoid an “apples to oranges” comparison with DP-TB, resulting in very similar code for both Toolboxes. In this way, not only were the tests run under the same OS and the same library (in each case, the same PVM or LAM/MPI version) compiled with the
238
Javier Fern´ andez et al.
same compiler and optimization switches, but also the test codes were kept as similar as possible to each other, avoiding the use of library functionality not available under all Toolboxes. More complex applications could conceivably benefit from more sophisticated library calls, such as PVM message handlers or mailboxes, elaborate collective MPI operations — or even an MPI_Sendrecv_replace call, as in [7], functionality that is not present in all the Toolboxes here considered. For this simple example we can restrict ourselves to the common functionality provided by all Toolboxes without being concerned about hypothetical performance improvements due to the use of different library calls. The same criterion was followed with MPITB, giving the sentence: for numt=1:C MPI_Recv(Sum,MPI_ANY_SOURCE,TAG,NEWORLD); Psum=Psum+Sum; end
The most compact master code is MultiMatlab Mast.m: tic Psum=0; Eval([1:C],[’Work(’ int2str(C) ’,’ int2str(N) ’)’]) for i=1:C , Psum=Psum+Recv; end Psum=Psum/N; T=toc;
The slave computation loop can be vectorized to avoid its interpretation cost under Matlab. Again, the most compact version is MultiMatlab Work.m: rank=ID; width=1/N; lsum=0; i=rank-1:C:N-1; x=(i+0.5)*width; lsum=sum(4./(1+x.^2)); Send(0,lsum)
% index subset for this rank % vector of abscissa mid-points % vectorized - no for loops
MPITB uses [info rank]=MPI_Comm_rank(NEWORLD) instead of ID, and the more verbose MPI_Send(lsum,0,TAG,NEWORLD), and requires the already mentioned startup.m support for merging the communicators, while PVMTB and DP-TB Work.m scripts explicitly receive the inum index: parent=pvm_parent; ... % Direct TCP route pvm_recv(parent,TAG); pvm_upkdouble(inum); width=1/N; lsum=0; ... pvm_initsend(PLACE); pvm_pkdouble(lsum); pvm_send(parent,TAG);
Figure 4 shows the measured time, accounting for slave computation, argument transmit and result collect times. Absolute times are almost indistinguishable, since computation time is in the order of 500ms (expected 62.5ms for C==8)
Performance of Message-Passing MATLAB Toolboxes Computation Time
Scalability
0.6
8 DPTB MultiMATLAB PVMTB MPITB Sequential
0.5
7 6
Speedup
Time (s)
0.4 0.3 0.2
5 4 3
0.1 0
239
2
2
4 6 # Computers
8
1
2
4 6 # Computers
8
Fig. 4. Results of the scalability test Table 3. Algorithm Time and Speedup using C=2, 4, 8 computers Toolbox DP-TB MultiMatlab PVMTB MPITB
C=2 255.807 259.408 254.102 257.290
Time (ms) C=4 C=8 133.928 69.440 136.134 69.264 129.118 63.730 131.676 64.770
C=2 1.984 1.957 1.997 1.973
Speedup C=4 C=8 3.790 7.309 3.728 7.328 3.931 7.964 3.854 7.836
while expected transmit time ranges from 0.227ms (MPITB) to 0.676ms (DPTB, see Table 2), giving computation-to-communication ratios in the order of 101 –103 , depending on the number of computers C and the Toolbox used. The differences can be highlighted by moving these absolute times to the denominator, as in the associated speedup graph in Fig. 4. In this application, up to C==8 messages come back to the master Matlab process, although not in perfect synchronization, due to the initial sequential data distribution. The performance advantage of MPITB over PVMTB in Table 2 is lost in the final accumulation for loop in the master process. The greater advantage of MultiMatlab over DP-TB is also lost. In PVMTB and MPITB, the receive buffer Sum is reutilized, while temporal storage is allocated and discarded for each iteration in DP-TB and MultiMatlab code. Table 3 quantifies this small performance loss.
240
5
Javier Fern´ andez et al.
Conclusions
Message-passing is the natural programming paradigm for clusters of computers, and several free Toolboxes are available for Matlab users willing to turn to Parallel Computing in this kind of platform. This is in agreement with the manufacturer’s point of view about the possible uses of parallelism under Matlab: SMP parallelism is better managed by the OS (and possibly a machine-tuned BLAS) than by Matlab. Also, the parallelization effort should address the coarsest granularity available in the prototype, which is usually well suited to the message-passing paradigm in clusters of computers. The user can choose the message-passing software preferred, either PVM or an MPI implementation. The small performance differences between the two systems shown in the ping-pong test in Section 3 disappear as soon as Matlab computation takes place between successive messages, as in the receive-add loop studied in Section 4. Depending on the granularity of the application, the user can either choose a higher-level Toolbox and benefit from easier, more compact coding, or directly interface the message-passing routines and benefit from greater control and performance. For coarse-grained, embarrassingly parallel applications with high computation-to-communication ratios, the performance loss is very small. Other parallel applications with smaller granularity, complex communication patterns or repeated synchronization or rendez-vous steps, might require the higher performance attainable at the cost of more verbose programming.
References [1] J. Arndt, C. Haenel: “π – Unleashed”, Springer-Verlag, 2001, ISBN: 3540665722, http://www.jjj.de/pibook/pibook.html 237 [2] C. Chang, G. Czajkowski, X. Liu, V. Menon, C. Myers, A. E. Trefethen, L. N. Trefethen: “The Cornell MultiMatlab Project”, Proceedings of the POOMA’96 Conference, Sanfa Fe, New-Mexico, http://www.acl.lanl.gov/Pooma96/abstracts/anne/POOMA96.html 230 [3] R. Choi: Parallel Matlab survey, http://supertech.lcs.mit.edu/∼cly/survey.html 228 [4] Cornell Theory Center: “New Parallel Programming Tools for Matlab”, http://www.tc.cornell.edu/news/releases/2000/cmtm.asp 230 [5] H. Dietz: “Parallel Processing HOWTO”, January 1998, Linux Documentation Project, http://aggregate.org/PPLINUX/ 236, 237 [6] S. Dormido-Canto, A. P. Madrid, S. Dormido: “Programming on Clusters for Solving Control Problems”, Proceedings of the 4th Asian Control Conference ASCC2002, Suntec Singapore, Singapore, September 2002, Session WA9-11 PaperID 1343 in 236 http://www.ece.nus.edu.sg/ascc2002/advprog/Prog Summary.pdf http://atc.ugr.es/∼javier/investigacion/papers/Seb-ASCC.pdf
[7] J. Fern´ andez-Baldomero: “Message Passing under Matlab”, Proceedings of the HPC 2001 (Adrian Tentner, Ed.), pp.73–82. ASTC 2001, Seattle, Washington, 228, 230, 231, 236, 238 http://www.scs.org/confernc/astc/astc01/prelim-program/html/hpc-pp.html http://atc.ugr.es/∼javier/investigacion/papers/H010FernandezBaldomero002.pdf
Performance of Message-Passing MATLAB Toolboxes
241
[8] J. Fern´ andez-Baldomero: MPITB home page, http://atc.ugr.es/javier-bin/mpitb eng 230 [9] J. Fern´ andez-Baldomero: PVMTB home page, http://atc.ugr.es/javier-bin/pvmtb eng 230 [10] V. Garc´ıa-Osorio, B. E. Ydstie: “Parallel, Distributed Modeling of Simulation of Chemical Systems”, American Institute of Chemical Engineers, Annual Meeting, Reno NV, November 2001. http://www.aiche.org/conferences/techprogram/ paperdetail.asp?PaperID=2009&DSN=annual01 237 [11] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Mancheck, V. Sunderam: “PVM: Parallel Virtual Machine. A Users’ Guide and Tutorial for Networked Parallel Computing”, The MIT Press, Cambridge, Massachusetts, 1994, http://www.netlib.org/pvm3/book/pvm-book.html 229 [12] S. Goasguen, A. Butt, K. D. Colby, M. S. Lundstrom: “Parallelization of the Nanoscale Device Simulator nanoMOS 2.0 using a 100 Nodes Linux Cluster”, Proceedings of the 2nd IEEE Conference on Nanotechnology IEEE-NANO’02, Arlington VA, USA, August 2002, Session WA7#0840 in http://www.ewh.ieee.org/tc/nanotech/nano2002/techprog.html http://atc.ugr.es/∼javier/investigacion/papers/SebNano02.pdf http://atc.ugr.es/∼javier/investigacion/papers/IEEE-nano.pdf http://falcon.ecn.purdue.edu:8080/cgi-bin/cvsweb.cgi/ 237 [13] W. Gropp, E. Lusk: “Reproducible Measurements of MPI Performance Characteristics”, Proceedings of the EuroPVM/MPI’99 Conference, Barcelona, Spain, September 1999, http://www-unix.mcs.anl.gov/∼gropp/bib/papers/ 234, 236 [14] W. Gropp, E. Lusk, A. Skjelum: “Using MPI: Portable Parallel Programming with the Message-Passing Interface”, The MIT Press, 1994, http://www.mcs.anl.gov/mpi/usingmpi/index.html 229 [15] LAM Home Page, http://www.lam-mpi.org 229 [16] The MathWorks, Inc.: “Matlab based books” web page, http://www.mathworks.com/support/books 228 [17] The MathWorks, Inc.: “User stories” web page, http://www.mathworks.com/products/user story http://www.mathworks.com/products/matlab/description/overview.shtml 228 [18] V. S. Menon, A. E. Trefethen: “MultiMATLAB: Integrating MATLAB with High-Performance Parallel Computing”, Proceedings of Supercomputing’97, ACM SIG ARCH and IEEE Computer Society, 1997, http://www.supercomp.org/sc97/proceedings/TECH/MENON/INDEX.HTM
228, 230, 231 [19] C. Moler: “Why there isn’t a parallel Matlab”, Matlab news & notes, Cleve’s Corner Spring 1995, http://www.mathworks.com/company/newsletter/pdf/spr95cleve.pdf 229 [20] C. Moler: “Matlab incorporates LAPACK”, Matlab news & notes, Cleve’s Corner Winter 2000, http://www.mathworks.com/company/newsletter/clevescorner/ winter2000.cleve.shtml 229 [21] S. Pawletta, W. Drewelow, P. Duenow, T. Pawletta, M. Suesse: “A Matlab Toolbox for Distributed and Parallel Processing”, Proceedings of the Matlab’95 Conference (C. Moler, S. Little, Eds.), MathWorks Inc., Cambridge, MA, October 1995, http://citeseer.nj.nec.com/pawletta95matlab.html 229 [22] M. J. Quinn: “Parallel Computing Theory and Practice, 2nd Edition”, McGraw Hill, New York, 1994. 236
242
Javier Fern´ andez et al.
[23] A. E. Trefethen, V. S. Menon, C. C. Chang, G. J. Czajkowski, C. Myers, L. N. Trefethen: “MultiMatlab: Matlab on Multiple Processors”, Tech. Report 96-239, Cornell Theory Center, 1996, ftp://ftp.tc.cornell.edu/pub/tech.reports/tr239.ps 230
Evaluating the Performance of Space Plasma Simulations Using FPGA’s Ben Popoola and Paul Gough Space Science Center, University of Sussex Falmer, Brighton, BN1 9QT, U.K {o.m.popoola,m.p.gough}@sussex.ac.uk
Abstract. This paper analyses the performance of a custom compute machine, that performs electrostatic plasma simulations, using Field Programmable Gate Array’s (FPGAs). Although FPGA’s run at slower clock speeds than their off-the-shelf counterparts, the processing power lost in the reduced number of clock cycles per second is quickly recovered in the high degree of spatial parallelism that is achievable within the devices. We describe the development of the architecture of the machine and its support for the C-programming language via the use of a cross-compiler. Results are presented and a discussion is given on the constraints of FPGAs in particular and the hardware design process in general.
1
Introduction
Scientific computing is characterised by developing a mathematical model to investigate some physical phenomenon of interest, verifying that the mathematical model developed can produce reasonable results within specified input constraints and discretising the mathematical model developed for use on digital computers. The discretized model which usually consists of a large number of differential equations, to which a numerical solution is sought, forms the basis of a computer simulation model. When expressed as a sequence of computer instructions, the physical phenomenon under interest is investigated using a computer simulation program. Termed computer experiments [15], simulation programs consist of following the temporal evolution of a system, from time t = 0 in discrete time steps of dt within a computional box that represents some finite region of space in the problem domain. One of the challenges in scientific computing is in achieving technically accurate and efficient simulation models, given finite and constrained computing resources. The large amount of computing power required to solve the simulation models, had led to the proposal and/or development of varying general and special purpose computer architectures since the introduction of modern digital computing. This has been fueled further by the advent of Very Large Scale Integrated (VLSI) circuit technology that has allowed the computer architect to utilise millions of transistors on a single silicon chip. J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 242–254, 2003. c Springer-Verlag Berlin Heidelberg 2003
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
243
One of the more successful architectures built from these proposals in the 1970’s, was the CRAY-1 [22] supercomputer. It was one of only a handful of computers available at the time, that offered vector processing techniques as a means of rapidly computing replicated structures of data commonly found in simulation programs. One of the main problems posed by supercomputers, apart from their comparatively high cost, is in the slow turn-around time in the implementation of new architectures that utilise modern integrated circuit technology techniques. This arises from the long design times and the complicated nature of their custom architectures. A more recent achievement in obtaining high performance in scientific computing, is in the use of parallel computing architectures over a cluster of workstations. The earliest approach used a Network of Workstations (NOW’s) but a more modern method is in the use of off-the-shelf computing components to build Beowulf clusters [5]. Using these systems requires users to write algorthims using some sort of portable communication library [3], one of which is the Message Passing Interface(MPI) [26, 13]. Computing clusters have brought their own challenges to the scientific computing arena as users try to ensure that all processors in a computing cluster perform an equal proportion of work, a technique known as load balancing. A equally challenging problem in using computing clusters is in the granularity of the algorthims used, which is highly dependent on the communication bandwidth between participating processors. All the techniques mention so far, use Application Specific Integrates Circuits (ASICs) devices, whose functionality cannot be altered once it has been fabricated on silicon. Recent advances in transistor and packaging technology has seen, over the last twenty years, the emergence of the Field Programmable Gate Arrays (FPGAs) [8, 1]. An FPGA is a reprogrammable chip that provides system designers with the opportunity of implementing their own custom architectures on silicon, via software, without incurring the high costs associated with fabricating ASICs. Although functions implemented in FPGAs run at slower clock speeds than the same function implemented in an ASIC, they provide the user the opportunity of implementing application specific code speed-ups in hardware, through the use of high degrees of spatial parallelism. Algorithm improvements can also be re-implemented as FPGAs are reprogrammable. As most ASICs are fabricated as general purpose devices, clock cycles that are lost in “raw speed” in FPGA implementations of specific functions, can be easily recovered and surpassed by the use of well structured spatially and temporally parallel algorithms. This paper describes our approach in using FPGAs in the solution of space plasma simulations and is organized as follows. In section two, we introduce the mathematical model encountered in electrostatic plasma calculations and describe the principle behind FPGAs and their use in developing custom parallel architectures. Section three describes the implementation of our custom Coprocessing Units(CUs), used to accelerate the solution of the simulation model. The
244
Ben Popoola and Paul Gough
Initiialise the system
Assign charge to the mesh
Solve for the potential
time step
Accelerate the particles and move the particles
Calculate the grid defined force field
Fig. 1. The design flow of a typical plasma simulation program
next section describes our results and is followed by a discussion of the challenges faced in using FPGAs in scientific computing.
2
Background Work
Rather than develop a mathematical model from scratch for space plasma simulations, with all the technically challenging problems of energy conversvation, convergence errors and floating point rounding errors, a decision was made to use a plasma simulation code that was already established, available and had obtainable source code. A 1D simulation code that the Space Group of the University of Sussex is familiar with and has been used to investigate various phenomenon in space plasmas is the 1D electrostatic code(ES1)[4]. It can typically be used to evolve in time, hundreds of thousands of particles with position and velocity, where each iterative time step involves the following cycle of events: – The assignment of charge onto a computational grid from the particles’ positions x and velocities v. – Integration of the field equations on the grid to obtain the electric E and magnetic B fields. – Calculating, from the electric and magnetic fields, the grid-defined force F field. – An interpolation of the force field to obtain the new particles positions and velocities - particle movement. This procedure is repeated in a continuous loop for the lifetime of the simulation with the user being offered the opportunity of viewing the results either
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
245
immediately, intermittently, or after a simulation has been completed by saving the results to a disk file for post-processing. An important calculation that lies at the heart of the ES1 code, is in obtaining the solution of the field equations for a bounded plasma by the use of a rapid elliptic solver [25] based on the Fast Fourier Transform (FFT) [6, 21]. The FFT is the method used in the ES1 code as a Poisson solver to solve the 1D Poisson partial-differential equation −∆u = f (x). Although FPGAs have been in use since the 1980’s, only recently are they becoming available with a large amount of logic elements (LE) and internal embedded RAM. It is not uncommon today to find devices in the Altera APEX 20KE logic family which have upto 51,840 LE (APEX1500KE) which roughly translates into about 1.5 million usable gates. To put this number into per perspective, the integer unit of our MIPS I compatible processor utilises about 3 per cent of the above mentioned device (APEX1500KE). A method used to design custom architectures for FPGAs is with traditional schematic entry software tools used for digital logic design. The software generally consists of a variety of low-level digital logic components which can be wired together to form more complicated circuits. The user also has the opportunity of obtaining higher-level functions, such as transcendential algorithms, as Intellectual Property (IP) cores from third party sources. However the method used here is to a provide a description of the functionality of the various algorithms at the Register Transfer Level (RTL) using a variant Hardware Description Language (HDL), VHDL. An emerging standard of capturing custom architecures, which is not explored here, is in the use of the C-like hardware programming languages HandleC and System-C. Which ever method is used, a full Electronic Design Automation (EDA) tools suite is required that allows a capturing of the design, simulation, synthesis, back simulation and the final step of place and routing. The place and routing tools’ output is a disk file in the form of a serial object file. This describes the inner working of the FPGA, how each logic elements will function, how all the logic elements are interconnected and how the I/O pins are mapped to external devices. Once this file is downloaded onto an FPGA a working device with user specified functionality is obtained. With the availability of FPGAs that are capable of potentially containing the simulation model, the choice of how to implement the design in custom hardware must be made. The method used here has been influenced by previous studies that have shown that integrating a processor with hardware function units eliminates the communication bandwidth between an off-board processor and reconfigurable logic [12, 27, 11]. Here we describe the design of the MIPS R3000 compatible processor with an accompliment of Coprocessing Units (CU’s) to accelerate a modified version of the ES1 code. Each CU must conform to our coprocessor interface specification and meet the specific requirements given below. A CU can be as small as a unit that accelerates a specific portions of a code, to special purpose units from LU decomposition matrix solvers to custom visualization units that provide standard computer graphics output. The constraints
246
Ben Popoola and Paul Gough
that we apply to any coprocessing unit currently is that if they use floating point arithmetic they should comply with the IEEE-754 Standard [16, 9] on floating point arithmetic, must be able to reside on the same FPGA as the MIPS 1 compatible core and should not increase the maximum CPU pipeline-stage latency. Apart from these restrictions each CU can exhibit its’ own degrees of internal spatial and temporal parallelism. The next section of this paper describes the overall architecture of the custom computing machine and the implementation of three IEEE-754 compliant coprocessors a floating point, fast fourier transform and particle mover units.
3
Implementation
As the intention of this project is to physically implement a prototype system, rather than produce a VHDL simulation model, a Reduced Instruction Set Computer (RISC) [20, 18, 23], with a Harvard bus architecture that is easy to implement has been chosen as our general purpose CPU. This led to the choice of a design that is compatible with the MIPS R3000 [17, 14] processor, mainly due to its widely available documentation. Our implementation, as in the original, is a single issue CPU with temporal parallelism obtained through the use of five pipelined stages. Our current version supports a 32-bit instruction set and is characterised by a load-store architecture. A decision not to implement a superscalar multi-issue processor was made, in anticipation that the CU’s will be the “work horses” of the system rather than the CPU. Our implementation also, does not implement Level 1 (L1) instruction and data caches on chip. Instead we use ZBT SRAM [19] as off-chip Level 2 (L2) cache with access times of about 5ns. This has the disadvantage of not only limiting our system to a maximum clock frequency of 200Mhz, but also limits the number of particles available in a simulation, in our prototype model, due to the relatively high cost per bit of SRAM compared to SDRAM. It does offer however, the prototype system, the advantage of sustained memory throughput unlike SDRAM that requires a complicated cache design and non-sustainable throughputs, hence delays, due to cache misses. An alternative approach could have been to use the internal RAM available on FPGA’s as L1 cache, however as the current trend in FPGAs, is to provide an amount of internal RAM that is proportional to the LE count this would have left our design FPGA type dependent. To date we currently have 3 working CUs for plasma simulation acceleration and a visualization unit that provides output to a standard VGA monitor ( To be described in a forth-coming paper). These units and the overall architecture are shown in figure 2. Although all units can run in parallel, each CU has a special control function that allows it to take complete control of the data buses. This prevents memory conflicts with hardware units trying to access memory simultaneously. This is especially important due to the implementation of left and right data caches.
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
247
To PC Communications Interface
SOFTWARE Visualization Software
To VDU
Graphics Visualization Unit
Plasma Simulation Software
MIPS I Compatible Processor
Mathematics & Physics Algorithms HARDWARE
Storage
Fig. 2. The overall framework of an FPGA based simulation environment. A FPGA is used to implement a variety of mathematical functions used in plasma simulations. Plasma simulation codes are written in the C programming language and interface directly to the underlying hardware through an Application Programmers Interface (API)
The left data cache can be considered to be the standard data cache found in Harvard implemented architectures. The right cache is used as a scratch area for implementing algorithms that require writing-back to the same area of memory, like the FFT. Our implementation allows a sustainable throughput of pipelined functions by reading data from memory side A, performing a function, and saving the result to side B, whilst simultaneously reading into the function new data values from side A. This allows an efficient pipelined implementation of algorithms without delays due to memory reads and writes. An even better approach would have been to use commodity dual-port RAM (DPRAM), however the very high cost per-bit of this type of RAM, excluded its use in our prototype model. To support the monitoring, programming and data collection of the machine we connect it to our local area network using the ethernet protocol. This could have been implemented as a CU, but was considered too expensive in terms of the number of Logic Elements it would have required. Instead we interface the system to a cheaply available commodity network IC that has ethernet support. We could use this interface in the future to support simulations over the World Wide Web. Each processing unit transparently adds functionality to the CPU’s instruction set. Algorithms that should be implemented in hardware are chosen by profiling the ES1 code and calculating the relative portion of time spent. This is done for simulations containing a different number of particles and a varying
248
Ben Popoola and Paul Gough
number of grid boxes. A decision is then made on whether an algorithm should be implemented in hardware or not. Based on this profiling scheme, our current design consists of 3 CU’s specifically related to the ES1 code 1. A hardware FFT to solve Poissons’ equation. 2. A particle mover that calculates particle positions and velocities. 3. An optimized single-precision floating point unit with hardware sine and cosine functions. Each CU that implements an arithmetic function achieves parallelism by implementing a linear pipeline. These are described below. 3.1
The Fast Fourier Transform
The discrete Fourier transform is Fn =
N −1
fk .W jk
W jk = e−j2πnk/N
(1)
k=0
with 0 ≤ n ≤ N − 1 An N-point FFT, N being a power of 2 requires O(N log2 N ) operations and N/2 butterfly operations are computed for the log2 stages. Spatial parallelism is achieved within the butterfly unit by calculating the real and imaginary parts of the complex variables concurrently. The limit to the temporal parallelism in our present FFT implementation is the rate at which the address of the variables are calculated. A higher level abstraction of the FFT is shown in figure 3.
Latches
fr[k]
+
fr[k]
fr[k + n2]
*
cos[p]
Real Part
+ Gr
-
fr[k + n2]
sin[p]
FFT Control
To Memory
Ram Controller
*
fi[k + n2] FFT FSM xx xxxxxx xx xx xx xx xx xx xx xx xx xx FFT xx xx Stage xx xx Calculator xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxx Butterfly Unit
Bit Reverse
fi[k]
- Gi +
fi[k]
-
fi[k + n2]
Imaginary Part
Latches
Address Generator
Ram Controller
To Memory
Fig. 3. The FFT coprocessing unit with a single butterfly unit consisting of a 15-stage arithmetic pipeline. The efficiency of the pipeline is limited by the rate at which the address of the arithmetic operators can be calculated
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
249
E(n-1)i PM Control
PM Fsm
NGP Calculator
To Memory
Ram Controller
v(n-2)
Particle Number Generator
Velocity Solver
Address Generator
Position Solver
x(n-1)
Ram Controller
v(n)
+
+
x(n+1)
Latches
To Memory
Fig. 4. The particle mover used to calculate the velocity and position from the electric. This particular CU users the Nearest Grid Point schema consists of an arithmetic pipeline which allows a velocity calculation to be performed in 5-stages with another 5 required for the particle position calculation 3.2
The Particle Mover
From the force Fi on each particle i, obtained from the electric field, E (in the electrostatic case) using the Lorentz force equation, we calculate the new velocity vi and position xi . Of the many methods used to calculate the force within a grid our current model uses the simplest, the Nearest Grid Point (NGP) method. In the nearest grid point schema, the new particle velocity and position are calculated in dimensionless coordinates based on the equations below. vin = vin−2 + Ejn−1
(2)
xn+1 = xn−1 + vn i i
(3)
where Ej is electric field associated with bin j. The throughput to these equations is continuous with a difference of 5 clock cycles between each veloctiy and position calculation. This is shown in figure 4. 3.3
The C-Language Support
Software support is provided by the use of the LCC cross-compiler [10, 7]. This is run as a host on a freeBSD UNIX workstation with a backend implementation of our own processor as the target. Although GCC [24] and its associated binutils are generally regarded as a better optimizing compiler, the LCC cross compiler
250
Ben Popoola and Paul Gough
Fig. 5. The figures above shows the instruction set profile when using the CCode only to implement a FFT to solve the Poisson equation using a various number of bins. The figure on the left shows the result from 4 to 64 bins. The one on the right is a magnified version for 64 bins is smaller and has, from our point of view, more accessible documentation. Also most of our optimizing is performed by the hardware coprocessing units. As our CPU design has a MIPS compatible base, we used the MIPS R3000 machine description (md) file as the starting point for creating our own backend to the cross-compiler. The output produced by LCC is assembly code, so a custom assembler has been written to convert the output to machine code. If we had used GCC we could have used the GCC cross-assember, GAS, but this was not looked into due to the relative ease of writing an assembler. Using GCC would also have provided automatic support for the C++ programming language and the advantages of object-oriented programming. The CUs are integrated into the C programming language structure as a series of user callable functions that are handcoded, in assembly language, into a library. This library is linked into the main body of code by the assembler.
4
Results and Discussion
Although for our small sample size, of FFT and particle points, a considerable performance gain has been achieved (see figures 7 and 8) a considerable hinderance to the software model can be attributed to the compiler itself. A 15 percent drop in the software performance - ’nops’ with respect to the FFT - arises from the compiler’s inability to utilise the branch delay slot. A further 10 percent reduction is due to data hazards. (see figure 5). A proportion of the add unsigned instructions - ’addu’ - can also be attributed to data hazards. However bearing these in mind, a substantial increase
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
251
Fig. 6. This shows (on the left) the result of using C-Code only to implement the nearest grid point algorithim for a varying number of particles. The figure on the right is the enlarged case of 128 particles. Data hazards instructions and nops, due to control hazards, can be seen to significantly increase the software implementation time
in performance is still gained in the hardware implementation when compared to that in software only, on the same system. This can also be seen in the case of the particle mover (shown in figure 8). These results have been generated with the system running at 40Mhz. A limiting factor of the hardware units is our restriction that they should all run at the same speed of 40 Mhz, which is the current maximum speed of our MIPS R3000 compatible core with the integrated units, although individually the units run faster. Whether each CU should be allowed to run at its full speed and an asynchronous communication protocol be set up between the units has not yet been investigated. Neither has the idea of implementing a plasma simulation code completely in hardware. However both of these methods would make the test and verification of the system more complicated.
5
Conclusion and Future Work
We have shown above that by implementing certain mathematical functions directly in hardware we can reduce the calculation time of an electrostatic plasma simulation. The performance of the system can be improved by writing a custom compiler rather than using a cross-compiler to produce more efficient machine code. Using a software/hardware approach still has many benefits - the network connectivity is implemented by using a software tcp library - and will be used in our future work.
252
Ben Popoola and Paul Gough
Fig. 7. The figure on the left shows the results of implementing the FFT purely in C-code (top blue curve) relative to the hardware implementation (lower red curve) for a varying number of bins. The corresponding percentage performance increases of the hardware version to that of the software is shown plotted on the left
Fig. 8. A comparism of the C-code implementation (upper blue curve) to that of the hardware CU (lower red curve) is shown in the figure on the left. The result of the performance increases of the hardware CU relative to the software implementation is given on the right
Evaluating the Performance of Space Plasma Simulations Using FPGA’s
253
The next stage of this project is to increase the overall speed of the system by implementing our hardware models in structural, rather than synthesizable behavioural VHDL. This should increase the system performance from around 40 Mhz to between 70 and 100 Mhz. We are also in the process of developing a set of programs to test the sysyem against desktop computers. Our experience gained so far has also encouraged us to move away from customising the one dimensional ES1 code, to directly implementing a two dimensional electrmagnetic plasma simulation code, with a coprocessor implemented for each stage of the design flow. The rapidly changing FPGA market has seen vendors implementing devices that not only contain different types of embedded hardware processors, but also advanced macro units - the new Altera Stratix devices [2] embed a number of dsp units containing hardwired adders and multipliers. How we will incorporate these into our work is still uncertain, but their use should also increase the overall performance of the system. There are advantages of implementing a soft processor in hardware as the coprocessing units can communicate synchronously. A hardwire processor core embedded within a FPGA forces asynchronus communcation between the core and custom logic through the vendor defined bus standard. Also, a varying number of softcore processors with varying databus widths can be implemented on a single FPGA to suit the application at hand, in contrast to the fixed number of an embedded hardwired processor. With the prediction that in the future designing for FPGAs will be moved away from the hardware designers domain to that of the software programmer, we will see an increase in projects that use FPGAs in scientific compututing.
Acknowledgments This research is funded by the Particle Physics Research Council (PPARC), U.K. The authors would like to thank Alex Benyon, Field Applications Engineer of Altera Europe, for his helpful advice and assistance on using Alteras family of FPGAs. Thanks goes also to Andy Buckley, Eduardo Bezerra and Marianne Pouchet, of the Space Science Group, University of Sussex, for their useful comments and suggestions.
References [1] Altera Corporation, San Jose, CA. The APEX 20K Programmable Logic Device Family Data Book, 2001. 243 [2] Altera Corporation, San Jose, CA. The Stratix Programmable Logic Device Family Data Book, 2002. 253 [3] J Bala, V Bruck. A portable and tunable collective communication library for scalable parallel computers. Technical report, IBM T. J Watson Research Center, 1993. 243 [4] A.B Birdsall, C.K Langdon. Plasma Physics via Computer Simulation. Institute of Physics Publishing Ltd, 1991. 244
254
Ben Popoola and Paul Gough
[5] R Buyya, editor. High Performance Cluster Computing: Architectures and Systems. Prentice Hall, 1999. 243 [6] J W Cooley, J W Tukey. An algorithm for the machine calculation of complex fourier series. Math. Comp. 19., pages 297–301, 1965. 245 [7] D R Fraser, C W Hanson. A code generation interface for ansi c. Software-Practice and Experience, 21(9):963–988, Sept 1991. 249 [8] R Freeman. User programmable gate arrays. IEEE Spectrum, December 1988. 243 [9] D Goldberg. What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1):5–48, 1991. 246 [10] C W Hanson, D R Fraser. A Retargetable C Compiler: Design and Implementation. Addison-Wesley Publishing Co, 1995. 249 [11] W T Hauck, S Fry. The chimaera reconfigurable functional unit. IEEE Symposium on Field-Programmable Custom Computing Machines, 1997. 245 [12] J Hauser, J Wawrzynek. Garp: A mips processor with a reconfigurable coprocessor. In Proceedings of the IEEE Symposium on Field Programable Custom Computing Machines, FCCM’97, April 1997. 245 [13] D Hempel, R Walker. The emergence of the mpi message passing standard for parallel computing. Computer Standards and Interfaces, 7:51–62, 1999. 243 [14] D Hennessy, J Patterson. Computer Organisation and Design - The Hardware Software Interface. Morgan Kaufman, 1998. 246 [15] J W Hockney, R W Eastwood. Computer Simulations using Particles. Institute of Physics Publishing Ltd, 1999. 242 [16] IEEE. ANSI/IEEE Standard 754 - 1985, Standard for Binary Floating Point Arithmetic. Institute for Electrical and Electronic Engineerings. 246 [17] G Kane. MIPS RISC Architecture. Prentice Hall, 1988. 246 [18] R E Kessler. The alpha 21264 microprocessor. IEEE Micro, March/April 1999. 246 [19] Micron Technology Inc. Designing with ZBT SRAM, 2002. Micron technology app. notes. 246 [20] D Patterson. Reduced instruction set computers. Comm. ACM, 28(1):8–21, Jan 1985. 246 [21] M C Pease. An adaptation of the fast fourier transform for parallel processing. Journal of the ACM, 15:252–264, 1968. 245 [22] R Russell. The cray-1 computer system. Communications of the ACM, 21(1), 1978. 243 [23] P S Song, M Denman, and J Chang. The powerpc 604 risc microprocessor. IEEE Micro, October 1994. 246 [24] R M Stallman. Using and Porting GNU CC - Cross Compiler. Free Software Foundation Inc, 1998. 249 [25] C Temperton. Direct methods for the solution of the discrete poisson equation: Some comparissons. Journal of Computational Physics, 31:1–20, 1979. 245 [26] J Walker, D Dongarra. Mpi: A standard message passing interface. Supercomputer, 12(1):56–68, January 1996. 243 [27] P Wittig, R Chow. Onechip: An fpga processor with reconfigurable logic. IEEE Symposium on FPGA’s for Custom Computing Machine, pages 126–135, 1996. 245
Remote Parallel Model Reduction of Linear Time-Invariant Systems Made Easy Peter Benner1 , Rafael Mayo2 , Enrique S. Quintana-Ort´ı2, and Gregorio Quintana-Ort´ı2 1
2
Institut f¨ ur Mathematik, Technische Universit¨ at Berlin D-10623 Berlin, Germany [email protected] Tel.: +30-314-28035, Fax: +30-314-79706 Depto. de Ingenier´ıa y Ciencia de Computadores, Universidad Jaume I 12071–Castell´ on, Spain {mayo,quintana,gquintan}@icc.uji.es Tel.: +34-964-728257, Fax: +34-964-728486
Abstract. This paper describes a library for model reduction of largescale, dense linear time-invariant systems on parallel distributed-memory computers. Our library is enhanced with a mail service which serves as a demonstrator of the capabilities of the library. Remote requests submitted via e-mail are executed on a cluster composed of Intel Pentium-II nodes connected via a Myrinet switch. Experimental results show the numerical and parallel performances of our model reduction routines. Keywords: Model reduction, absolute error methods, matrix sign function, mail service.
1
Introduction
Model reduction is of fundamental importance in many modeling and control applications involving linear time-invariant (LTI) systems. In state-space form, such systems are described by the following models: Continuous LTI system: x(t) ˙ = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t),
t > 0, t ≥ 0.
x(0) = x0 ,
(1)
Discrete LTI system: xk+1 = Axk + Buk , yk = Cxk + Duk ,
x0 = x0 , k = 0, 1, 2, . . . .
(2)
In both models, A ∈ IRn×n is the state matrix, B ∈ IRn×m , C ∈ IRp×n , D ∈ IRp×m , n is said to be the order of the system, and x0 ∈ IRn is the initial
Supported by the Fundaci´ o Caixa-Castell´ o/Bancaixa PI-1B2001-14 and CICYT TIC2002-04400-C03-01.
J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 255–268, 2003. c Springer-Verlag Berlin Heidelberg 2003
256
Peter Benner et al.
state of the system. The associated transfer function matrix (TFM) is G(s) = C(sI − A)−1 B + D. Here, we assume that the state matrix A is stable. In the continuous-time case, this implies that the spectrum of A is contained in the open left half plane while in the discrete-time case, the spectrum of A must lie inside the unit circle. This condition ensures that the system is stable, but we do not assume minimality of the system. In the model reduction problem, we are interested in finding a reduced-order LTI system, Continuous LTI system: ˆx(t) + B ˆu x ˆ˙ (t) = Aˆ ˆ(t), ˆ u(t), yˆ(t) = Cˆ xˆ(t) + Dˆ
t>0 t ≥ 0.
x ˆ(0) = x ˆ0 ,
(3)
Discrete LTI system: ˆxk + B ˆ uˆk , x ˆk+1 = Aˆ ˆ uk , yˆk = Cˆ xˆk + Dˆ
x ˆ0 = x ˆ0 , k = 0, 1, 2, . . . ,
(4)
ˆ+D ˆ which ˆ ˆ ˆ −1 B of order r, r n, and associated TFM G(s) = C(sI − A) approximates G(s). There is no general technique for model reduction that can be considered as optimal in an overall sense since the reliability, performance, and adequacy of the reduced system strongly depends on the system characteristics. The model reduction methods we study here rely on truncated state-space transformations T ∈ IRn×n and Y −1 = [Tr , Lr ], with Tl ∈ IRr×n where, given Y = TlT , LTl and Tr ∈ IRn×r , Aˆ = Tl ATr ,
ˆ = Tl B, B
Cˆ = CTr ,
ˆ = D. and D
(5)
Model reduction methods based on truncated state-space transformations usually differ in the measure they attempt to minimize. Balanced truncation (BT) methods [16, 20, 22, 24], singular perturbation approximation (SPA) methods [15], and Hankel-norm approximation (HNA) methods [12] all belong to the ˆ ∞. family of absolute error methods, which try to minimize ∆a ∞ = G − G Here, G∞ denotes the L∞ - or H∞ -norm of a stable, rational matrix function defined as (6) G∞ = ess sup σmax (G(ω)), ω∈IR √ where := −1 and σmax (M ) is the largest singular value of the matrix M . The family of relative error model reduction methods attempt to minimize, ˆ ∞ , defined implicitly by G − G ˆ = ∆G. ˆ on the other hand, the relative error ∆ Among these, the balanced stochastic truncation (BST) methods [9, 13, 25] are specially interesting. Currently, BST methods are not included in the model reduction library and the associated mail service described here. Therefore, we do not include BST methods in our presentation. Model reduction of large-scale systems arises, among others, in control of large flexible mechanical structures or large power systems as well as in circuit
Remote Parallel Model Reduction
257
simulation and VLSI design; see, e.g., [7, 8, 11, 17]. LTI systems with state-space dimension n of order 102 to 104 are common in these applications. All absolute error model reduction methods for LTI systems with dense state matrices have a computational cost of O(n3 ) floating-point operations (flops). Large-scale applications thus clearly benefit from using parallel computing techniques to obtain the reduced system. The usual approach a few decades ago was to solve these problems on very expensive parallel supercomputers, but the situation is changing in the last years. Clusters constructed from commodity systems (personal computers and local and system area switches) and “open” software have started to be widely used for parallel scientific computing. The enormous improvements in the performance of the commodity hardware has provided an affordable tool for dealing with large-scale scientific problems. However, many end-users (control engineers, VLSI circuit designers, etc.) are not familiar with the basics of parallel algorithms and/or parallel architectures. Thus, if a parallel library of control algorithms is to be useful to a broader spectrum of the scientific community, it should be part of our work to provide, along with the library, the necessary tools to facilitate its use. The tool we propose to use, e-mail, has been employed from the first days of the Internet not only for personal communication but also for submission of jobs to remote systems, access to databases, document retrieval, etc. In this work we explore the use of e-mail to provide a parallel model reduction service to the scientific community. The advantage of providing this service via e-mail is that the end-user remains completely isolated from the complexities of the software and the hardware system, and the maintenance of both. A secondary goal of this service is to serve as a demonstrator of the capabilities of the parallel library, which can be then downloaded and installed on any (parallel) architecture with MPI and ScaLAPACK. Our library provides parallel routines for model reduction using the three mentioned absolute error methods. These approaches are briefly reviewed in Section 2. The contents and structure of the library are described in Section 3. The library is completed with a mail service, presented in Section 4, which allows remote access to our parallel cluster and enables the user to apply any of the model reduction algorithms on a specific system. Finally, the numerical and parallel performances of the routines in the parallel library is reported in Section 5, and some concluding remarks are given in Section 6.
2
Model Reduction via Absolute Error Methods
In this section we briefly review the methods for absolute error model reduction. Serial implementations of these algorithms can be found in the Subroutine Library in Control Theory – SLICOT1 . Note that we use implementations that are different from the serial versions in SLICOT but result in the same reducedorder model. Our implementation is particularly efficient on parallel computing platforms. 1
Available from http://www.win.tue.nl/niconet/NIC2/slicot.html.
258
Peter Benner et al.
All these methods are strongly related to the controllability Gramian Wc and the observability Gramian Wo of the LTI system. In the continuous-time case, the Gramians are given by the solutions of two coupled Lyapunov equations A Wc + Wc AT + BB T = 0, AT Wo + Wo A + C T C = 0,
(7)
while, in the discrete-time case, the Gramians are the solutions of two coupled Stein equations A Wc AT − Wc + BB T = 0, (8) AT Wo A − Wo + C T C = 0. As A is assumed to be stable, the Gramians Wc and Wo are positive semidefinite, and therefore there exist factorizations Wc = S T S and Wo = RT R. Matrices S and R are called the Cholesky factors of the Gramians. The Lyapunov equations are solved in our model reduction algorithms using the Newton iteration for the matrix sign function introduced by Roberts [19]. The Stein equations are solved using the Smith iteration proposed in [21]. Both iterations are specially adapted taking into account that, for large-scale nonminimal systems, the Cholesky factors are often of low (numerical) rank. By not allowing these factors to become rank-deficient, a large amount of workspace and computational cost can be saved. For a detailed description of this technique, see, e.g., [1]. Consider now the singular value decomposition (SVD) T Σ1 0 V1 T , (9) SR = [U1 U2 ] 0 Σ2 V2T where the matrices are partitioned at a given dimension r such that Σ1 = diag (σ1 , . . . , σr ), Σ2 = diag (σr+1 , . . . , σn ), σj ≥ 0 for all j, and σr > σr+1 . Here, σ1 , . . . , σn are known as the Hankel singular values of the system. If σr > σr+1 = 0, then r is the McMillan degree of the system. That is, r is the state-space dimension of a minimal realization of the system. The so-called square-root (SR) BT algorithms determine the reduced-order model in (5) using the projection matrices −1/2
T l = Σ1
−1/2
V1T R and Tr = S T U1 Σ1
.
(10)
In case Σ1 > 0 and Σ2 = 0, the reduced-order model computed by using Tl , Tr from (10) in (5) is a minimal realization of the TFM G(s). The balancing-free square-root (BFSR) BT algorithms often provide more accurate reduced-order models in the presence of rounding errors [24]. These algorithms share the first two stages (solving the coupled equations and computing the SVD of SRT ) with the SR methods, but differ in the procedure to obtain Tl and Tr . Specifically, the following two QR factorizations are computed, ¯ ˆ R R T T , R V1 = [Q1 Q2 ] S U1 = [P1 P2 ] , (11) 0 0
Remote Parallel Model Reduction
259
ˆ R ¯ ∈ IRr×r are upper where P1 , Q1 ∈ IRn×r have orthonormal columns, and R, triangular. The reduced system is then given by the projection matrices Tl = (QT1 P1 )−1 QT1 ,
Tr = P1 ,
(12)
and (5). SR SPA or BFSR SPA reduced-order models can be obtained by first using the SR BT or BFSR BT algorithms, respectively, to obtain a minimal realization ˜ B, ˜ C, ˜ D) ˜ be a realization of this minimal of the original system. Let the tuple (A, system and partition A11 A12 ˜ = B1 , C˜ = [ C1 C2 ], A˜ = , B A21 A22 B2 according to the desired size r of the reduced-order model, i.e., A11 ∈ IRr×r , B1 ∈ IRr×m , C1 ∈ IRp×r , etc. Then the SPA reduced-order model is obtained by applying the following formulae Aˆ ˆ B Cˆ ˆ D
:= A11 − A12 (γI − A22 )−1 A21 , := B1 − A12 (γI − A22 )−1 B2 , := C1 − C2 (γI − A22 )−1 A21 , ˜ − C2 (γI − A22 )−1 B2 , := D
(13)
where γ = 0 for continuous LTI systems and γ = 1 for discrete LTI systems [15, 23, 24]. BT and SPA model reduction methods aim at minimizing the H∞ -norm ˆ However, they usually do not succeed in finding norm of the error system G − G. an optimal approximation. Using the Hankel norm of a stable rational transfer function, defined by (14) GH = σ1 (G), where σ1 (G) is the maximum Hankel singular value of G, it is possible to compute ˆ H for a given order r of the reducedan approximation minimizing G − G ˆ is quite involved. Due to order system [12]. The derivation of a realization of G space limitations, we refer the reader to [12, 27] and only describe the essential computational tools required in an implementation of the HNA method. In the first step, a balanced minimal realization of G is computed. This can be done using the SR version of the BT method described above. Then a transfer function ˜+D ˜ is computed as follows: first, the order r of the ˜ ˜ G(s) = C(sI − A˜ )−1 B reduced-order model is chosen such that the Hankel singular values of G satisfy σ1 ≥ σ2 ≥ . . . σr > σr+1 = . . . = σr+k > σr+k+1 ≥ . . . ≥ σn ,
k ≥ 1.
Then, by applying appropriate permutations, the balanced transformation of G is re-ordered such that the Gramians become ˇ Σ . σr+1 Ik
260
Peter Benner et al.
ˇ B, ˇ C, ˇ D ˇ is partitioned according The resulting balanced realization given by A, to the partitioning of the Gramians, i.e., A11 A12 ˇ = B1 , Cˇ = [ C1 C2 ], Aˇ = , B A21 A22 B2 where A11 ∈ IRn−k×n−k , B1 ∈ IRn−k×m , C1 ∈ IRp×n−k , etc. Then the following ˜ formulae define a realization of G: 2 ˇ + σr+1 C1T U B1T , ˇ 11 Σ A˜ = Γ −1 (σr+1 AT11 + ΣA ˜ = Γ −1 (ΣB ˇ 1 − σr+1 C1T U ), B ˇ − σr+1 U B T , C˜ = C1 Σ 1 ˜ = D + σr+1 U. D
Here, U := (C2T )† B2 , where M † denotes the pseudoinverse of M , and Γ := ˇ 2 − σ 2 In−k . This step only requires matrix scaling, matrix multiplication, Σ r+1 ˜ and a QR decomposition. Now we can compute an additive decomposition of G ˜ ˆ ˆ ˆ such that G(s) = G(s) + F (s) where G is stable and F is anti-stable. Then G is an optimal r-th order Hankel norm approximation of G. We have implemented ˜ via a block diagonalization of A, ˜ where we first the additive decomposition of G ˜ compute a block Schur form using the sign function of A and then annihilate the off-diagonal block by solving a Sylvester equation using again a sign functionbased iterative solution procedure. For further details of the implementation of the HNA method and computational aspects see [3].
3
Parallelization, Structure and Contents of the Library
The absolute error methods basically perform matrix operations such as the solution of linear systems and linear matrix equations, and the computation of matrix products and matrix decompositions (LU, QR, SVD, etc.). All these operations are basic matrix algebra kernels parallelized in ScaLAPACK and PBLAS. Thus, our parallel model reduction library plicmr2 heavily relies on the use of the available parallel infrastructure in ScaLAPACK, the serial computational libraries LAPACK and BLAS, and the communication library BLACS (see Figure 1). Details of the contents and parallelization aspects of plicmr are given, e.g., in [1, 2]. The functionality and naming convention of the parallel routines closely follow analogous routines from SLICOT. The contents of plicmr are structured in four main groups: – Model reduction computational routines. • pab09ax: BT method. • pab09bx: SPA method. • pab09cx: HNA method. 2
http://spine.act.uji.es/~plicmr
Remote Parallel Model Reduction
261
PLICMR
ScaLAPACK
PBLAS Global Local
LAPACK BLACS
BLAS MPI, PVM,...
Fig. 1. plicmr and the underlying libraries – Matrix functions. • pmb05rd: Computation of the sign function of a matrix. – Linear matrix equation solvers. • psb03odc: Solution of coupled Lyapunov equations (7) using the Newton iteration for the matrix sign function. • psb03odd: Solution of coupled Stein equations (8) using the Smith iteration. • psb04md: Solution of a Sylvester equation using the Newton iteration for the matrix sign function. – Linear algebra. • pmb03td: Computation of the SVD of a product of two rectangular matrices. • pmb03ox: Estimation of the numerical rank of a triangular matrix. A standardized version of the library is integrated into the subroutine library PSLICOT [5], with parallel implementations of a subset of SLICOT. It can be downloaded from ftp://ftp.esat.kuleuven.ac.be/pub/WGS/SLICOT. The version maintained at http://spine.act.uji.es/~plicmr might at some stages contain more recent updates than the version integrated into PSLICOT. The library can be installed on any parallel architecture where the above-mentioned computational and communication libraries are available. The efficiency of the parallel routines will depend on the performance of the underlying libraries for matrix computation (BLAS) and communication (usually, MPI or PVM).
262
Peter Benner et al.
Mail Agent Service Request
Service Response
Mail Agent/PBS
PBS queue system
Interface
CLUSTER
Fig. 2. Components of the mail service system
4
Mail Service
In this section we describe the access procedure and the structure of the service for parallel model reduction via e-mail. This service is provided on a parallel cluster3 , composed of 32 Intel Pentium-II personal computers, running a Linux operating system, and connected with a 1 Gbps Myrinet network. The user can submit a job using any standard application for sending an e-mail message to [email protected]. A service request specifies the parameters of the job (like the model reduction algorithm to employ, the dimensions of the system, the desired order for the reduced system, etc.) in the body of the message. Validation of access is performed via a login and password included in the e-mail. Different compression tools are supported to reduce the size of the files specifying the system matrices, provided as attachments. The results of the execution are returned to the user via e-mail, with the matrices of the reduced system as attached (compressed) files. The service and the specific formats of the e-mail body and attachments are further described in http://spine.act.uji.es/~plicmr. The model reduction mail server is structured as a Mail Agent (MA) and a set of scripts which define the interface between this agent and PBS (portable batch system), a batch queuing and workload management system (see Figure 2). The MA is the operating system mail daemon that processes e-mails. On reception of an e-mail addressed to [email protected], the MA creates a process, plicmr-ma, which then employs the preprocessing scripts in the MA/PBS Interface to extract the contents of the e-mail, save the system matrices in the local file system, generate the submission scripts for PBS, and submit 3
The cluster is owned by the Parallel Scientific Research Group, at the University Jaume I; http://spine.act.uji.es/psc.html.
Remote Parallel Model Reduction
263
1. Receive e−mail
MA
2. Create process
7. Return response
PLICMR−MA
3. Extract contents Save matrices Generate subm. scripts
6. Activates postprocessing scripts Return result Interface Scripts
PBS 4. Submit job
5. Launch job
Fig. 3. Actors and actions involved in processing a service request for parallel model reduction the job. PBS is in charge of launching the execution of the job in the cluster and return the result to the plicmr-ma using the postprocessing (epilogue) scripts in the MA/PBS Interface. The plicmr-ma then generates an appropriate e-mail with the matrices of the reduced-order system and uses the MA to send it back to the user. Figure 3 illustrates the overall process, beginning with the reception of an e-mail with a service request from a remote user, and ending when an e-mail, containing the results of the execution of the request, is returned to the user.
5
Experimental Results
All the experiments presented in this section were performed on Intel Pentium-II processors using ieee double-precision floating-point arithmetic (ε ≈ 2.2204 × 10−16 ). The parallel algorithms were evaluated on a parallel distributed cluster of 32 nodes. Each node consists of an Intel Pentium-II processor at 300 MHz, and 128 MBytes of RAM. We employ a BLAS library, specially tuned for the Pentium-II processor as part of the ATLAS and ITXGEMM projects [26, 14], that achieves around 180 Mflops (millions of flops per second) for the matrix product (routine DGEMM). The nodes are connected via a Myrinet crossbar network; the communication library BLACS is based on an implementation of the communication library MPI specially developed and tuned for this network. The performance of the interconnection network was measured by a simple loopback message transfer resulting in a latency of 33 µsec and a bandwidth of 200 Mbit/sec. We made use of the LAPACK, PBLAS, and ScaLAPACK libraries wherever possible.
264
Peter Benner et al.
We only report results using the BT BFSR algorithms for a continuous LTI system. Similar results were obtained for the SPA and HNA algorithms and for discrete LTI systems. 5.1
Numerical Performance
In this subsection we compare the reliability and adequacy of the reduced systems computed using the SLICOT routines with those of the PLiCMR routines. For this purpose, we evaluate the response of the reduced systems, |G( ω)|, at frequencies ω, by means of two large-scale examples from the collection in [6]. The first example models the atmosphere in the midlatitude of the Pacific. The area is discretized using an horizontal division of 1000 × 1000 km2 and a vertical division 10 km long. The time is discretized in periods of 9 hours. The model is composed of 598 states, a single input, and a single output. The second example is full-order model (FOM) of a large dynamical system, and has six badly damped poles, i.e., three pairs of complex conjugate poles with large imaginary part which are responsible for the three significant peeks in the frequency response. Except for these peeks, the dynamics of this system are smooth as all the remaining poles are real and well separated from the imaginary axis. There is no difficulty to be expected in reducing the order of this system significantly. Figure 4 reports the frequency responses of the original system and the reduced models of orders r=9 and 10 for the atmosphere and the FOM examples, respectively. In both cases, no noticiable difference is encountered between the behaviour of the reduced systems computed by means of the SLICOT (ab09ad) and the PLiCMR (pab09ax) routines. As a matter of fact, in both examples the frequency response of the reduced systems perfectly match that of the original system for all frequencies.
4
BT−BFSR
BT−BFSR
3
10
10
original ab09ad pab09ax
original ab09ad pab09ax 2
10
3
10
1
|G(j ω)|
|G(j ω)|
10
0
10 2
10
−1
10
1
10
−2
0
10
frequency (rad/sec.)
1
10
10
1
10
2
10
3
10
frequency (rad/sec.)
4
10
5
10
Fig. 4. Frequency response of the atmosphere (left) and the FOM (right) examples
Remote Parallel Model Reduction
5.2
265
Parallel Performance
We employ two different large-scale examples to evaluate the parallel performance of the routines in PLiCMR. The first example comes from a finite element discretization of a steel cooling process described by a boundary control problem for a linearized 2-dimensional heat equation; see [18] and references therein. The system has 6 inputs and outputs, and n = 821 or 3113 states, depending on the meshsize. As there is no significant gap in the Hankel singular values of the system, in this experiment we computed a reduced system of fixed order r = 40. The system with n = 821 was reduced using the SLICOT routines in about 4 minutes. The parallel routines computed the reduced system in half of this time, using 4 processors. However, the system with n = 3113 could not be solved using the serial routines as the system matrices were too large to be stored in a single node. Our parallel routines provided the solution in around 15 minutes (using 16 processors). The second example is a random continuous LTI system constructed as follows. First, we generate a random positive semidefinite diagonal Gramian Wc = diag (Σq1 , Σq2 , 0q3 , 0q4 ), where Σq1 ∈ IRq1 ×q1 contains the desired Hankel singular values for the system and Σq2 ∈ IRq2 ×q2 . Then, we construct a random positive semidefinite diagonal Gramian Wo = diag (Σq1 , 0q2 , Σq3 , 0q4 ), with Σq3 ∈ IRq3 ×q3 . Next, we set A to a random stable diagonal matrix and compute F = −(AWc + Wc AT ) and G = −(AT Wo + Wo A). Thus, F = diag (f1 , f2 , . . . , fq1 +q2 , 0q3 +q4 ) , G = diag (g1 , g2 , . . . , gq1 , 0, . . . , 0, gq1 +q2 +1 , . . . , gq1 +q2 +q3 , 0q4 ) . A matrix B ∈ IRn×(q1 +q2 ) such that F = BB T is then obtained as B = diag f1 , f2 , . . . , fq1 +q2 . The procedure for obtaining C is analogous. The LTI system is finally transformed into A := U T AU , B := U T B, and C := CU using a random orthogonal transformation U ∈ IRn×n . The system thus defined has a minimal realization of order r = q1 . The Cholesky factors satisfy rank (S) = q1 + q2 and rank (R) = q1 + q3 . We first evaluate the execution time of the parallel model reduction algorithms. In the example, we set n = 1000, m = p = 100, and q1 = q2 = q3 = 50. As there is no noticeable gap in the Hankel singular value distribution of the system, we obtain a reduced-order model of order r = 40. The left-hand plot in Figure 5 reports the execution time of the BFSR BT serial routine for model reduction in SLICOT and the corresponding parallel algorithm as the number of nodes, np , is increased. The results show a considerable acceleration achieved by the parallel algorithm (with even super speed-ups). This is partially due to the efficiency of the Lyapunov solvers used in our algorithms which compute factors of the solutions in compact (full-rank) form instead of
266
Peter Benner et al. Execution time of BFSR BT model reduction algorithms
600
Scalability of parallel model reduction algorithms
ab09ad pab09ax
500
60
Mflop ratio per node
Execution time (sec.)
pab09ax pab09bx pab09cx
70
400
300
200
50 40 30 20
100
0 0
10
2
4
6
8
Number of nodes
10
12
14
0
4
9
16
Number of nodes
25
Fig. 5. Performance of the parallel model reduction algorithms square matrices, thus requiring less computations. Comparison of the results on 2 and 4 nodes roughly shows the efficiency of the parallel algorithm. The execution time is reduced by a factor of almost 2 (the number of resources, that is nodes, is doubled). Using a larger number of nodes does not achieve a significant √ reduction of the execution time due to the small ratio n/ np . We next evaluate the scalability of our parallel algorithms. As the memory of the system does not allow to test the serial algorithms on larger problems, in the √ √ experiment we fix the problem size per node using n/ np = 800, m/ np = 400, √ √ p/ np = 400, and q/ np = 200, with q1 = q2 = q3 = q. In the right-hand plot in Figure 5 we report the Mflop ratio per node for the parallel model reduction algorithms. This is often referred as the scaled speed-up. The figure shows a high scalability of all three algorithms, as there is only a minor decrease in the Mflop ratio per node as the number of nodes is increased up to 25 (a problem of size n = 4000). The scalability confirms that a larger problem can be solved by increasing proportionally the number of nodes employed. A more detailed analysis of the numerical accuracy and parallelism of the kernels in the parallel library for model reduction can be found in [4].
6
Concluding Remarks
In this paper we have described a parallel library for model reduction of largescale, dense LTI systems using absolute error methods. Our experiments report a notable reduction in the execution time when the parallel algorithms are used. An important part of this reduction is due to the use of compact factors of the Gramian matrices. Future extensions of the library will include parallel routines for relative error model reduction. A further extension will provide a service which allows an interaction between remote users and the parallel system through a web environment.
Remote Parallel Model Reduction
267
References [1] P. Benner, E. S. Quintana-Ort´ı, and G. Quintana-Ort´ı. Balanced truncation model reduction of large-scale dense systems on parallel computers. Mathematical and Computer Modeling of Dynamical Systems, 6(4):383–405, 2000. 258, 260 [2] P. Benner, E. S. Quintana-Ort´ı, and G. Quintana-Ort´ı. PSLICOT routines for model reduction of stable large-scale systems. In Proc. 3rd NICONET Workshop on Numerical Software in Control Engineering, Louvain-la-Neuve, Belgium, January 19, 2001, pages 39–44. 2001. 260 [3] P. Benner, E. S. Quintana-Ort´ı, and G. Quintana-Ort´ı. Computing optimal Hankel norm approximations of large-scale systems. In preparation. 260 [4] P. Benner, E. S. Quintana-Ort´ı, and G. Quintana-Ort´ı. Experimental evaluation of the parallel model reduction routines in PSLICOT. SLICOT Working Note 2002–7, http://www.win.tue.nl/niconet/ 266 [5] I. Blanquer, D. Guerrero, V. Hern´ andez, E. S. Quintana-Ort´ı, and P. Ru´ız. Parallel-SLICOT implementation and documentation standards. SLICOT Working Note 1998–1, http://www.win.tue.nl/niconet/, September 1998. 261 [6] Y. Chahlaoui and P. Van Dooren. A collection of benchmark examples for model reduction of linear time invariant dynamical systems. SLICOT Working Note 2002–2, February 2002. Available from http://www.win.tue.nl/niconet/NIC2/reports.html. 264 [7] J. Cheng, G. Ianculescu, C. S. Kenney, A. J. Laub, and P. M. Papadopoulos. Control-structure interaction for space station solar dynamic power module. IEEE Control Systems, pages 4–13, 1992. 257 [8] P. Y. Chu, B. Wie, B. Gretz, and C. Plescia. Approach to large space structure control system design using traditional tools. AIAA J. Guidance, Control, and Dynamics, 13:874–880, 1990. 257 [9] U. B. Desai and D. Pal. A transformation approach to stochastic model reduction. IEEE Trans. Automat. Control, AC–29:1097–1100, 1984. 256 [10] E. Elmroth, P. Johansson, B. K˚ agstr¨ om, and D. Kressner. A Web computing environment for the SLICOT library. SLICOT Working Note 2001–2, http://www.win.tue.nl/niconet/, June 2001. [11] L. Fortuna, G. Nummari, and A. Gallo. Model Order Reduction Techniques with Applications in Electrical Engineering. Springer-Verlag, 1992. 257 [12] K. Glover. All optimal Hankel-norm approximations of linear multivariable systems and their L∞ norms. Internat. J. Control, 39:1115–1193, 1984. 256, 259 [13] M. Green. Balanced stochastic realization. Linear Algebra Appl., 98:211–247, 1988. 256 [14] J. Gunnels, G. Henry, and R. A. van de Geijn. A family of high-performance matrix multiplication algorithms. In V. N. Alexander, J. Dongarra, B. A. Julianno, R. S. Renner, and C. J. Kenneth Tan, editors, Computational Science - ICCS 2001, Part I, Lecture Notes in Computer Science 2073, pages 51–60, 2001. 263 [15] Y. Liu and B. D. O. Anderson. Controller reduction via stable factorization and balancing. Internat. J. Control, 44:507–531, 1986. 256, 259 [16] B. C. Moore. Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE Trans. Automat. Control, AC-26:17– 32, 1981. 256 [17] C. R. Paul. Analysis of Multiconductor Transmission Lines. Wiley–Interscience, Singapur, 1994. 257
268
Peter Benner et al.
[18] T. Penzl. Algorithms for model reduction of large dynamical systems. Technical Report SFB393/99-40, Sonderforschungsbereich 393 Numerische Simulation auf massiv parallelen Rechnern, TU Chemnitz, Germany, 1999. 265 [19] J. D. Roberts. Linear model reduction and solution of the algebraic Riccati equation by use of the sign function. Internat. J. Control, 32:677–687, 1980. (Reprint of Technical Report No. TR-13, CUED/B-Control, Cambridge University, Engineering Department, 1971). 258 [20] M. G. Safonov and R. Y. Chiang. A Schur method for balanced-truncation model reduction. IEEE Trans. Automat. Control, AC–34:729–733, 1989. 256 [21] R. A. Smith. Matrix equation XA + BX = C. SIAM J. Appl. Math., 16(1):198– 201, 1968. 258 [22] M. S. Tombs and I. Postlethwaite. Truncated balanced realization of a stable nonminimal state-space system. Internat. J. Control, 46(4):1319–1330, 1987. 256 [23] A. Varga. Balancing-free square-root algorithm for computing singular perturbation approximations. In Proc. of the 30th IEEE Conf. Dec. Control, pages 1062–1065, 1991. 259 [24] A. Varga. Efficient minimal realization procedure based on balancing. In Prepr. of the IMACS Symp. on Modeling and Control of Technological Systems, volume 2, pages 42–47, 1991. 256, 258, 259 [25] A. Varga and K. H. Fasol. A new square–root balancing–free stochastic truncation model reduction algorithm. In Prepr. 12th IFAC World Congress, volume 7, pages 153–156, Sydney, Australia, 1993. 256 [26] R. C. Whaley and A. Petitet and J. Dongarrra. Automated empirical optimizations of software and the ATLAS project. Parallel Computing, 27(1–2):3–15, 2001. 263 [27] K. Zhou, J. C. Doyle, and K. Glover. Robust and Optimal Control. Prentice-Hall, Upper Saddle River, NJ, 1996. 259
An Approach to Teaching Computer Arithmetic Ester M. Garz´on, Inmaculada Garc´ıa, and Jos´e Jes´ us Fern´ andez Dept. Arquitectura de Computadores y Electr´ onica Universidad de Almer´ıa, 04120 Almer´ıa, Spain {ester,inma,jose}@ace.ual.es Tph/Fax: +34 950 015 486
Abstract. In this work we present an initiative to support teaching computer representation of numbers (both integer and floating point) as well as arithmetic in undergraduate courses in computer science and engineering. Our approach is based upon a set of carefully designed practical exercises which highlights the main properties and computational issues of the representation. In conjunction to the exercises, an auxiliary computer-based environment constitutes a valuable support for students to learn and understand the concepts involved. For integer representation, we have focused on the standard format, the well known 2’s complement. For floating point representation, we have made use of an intermediate format as an introduction to the IEEE 754 standard. Such an approach could be included in an introductory course related to either computer structure, discrete mathematics or numerical methods.
God made the integers, man made the rest. L. Kronecker. German mathematician.
1
Introduction
Computational science is nowadays making it possible to solve grand-challenge problems thanks to greatly improved computational techniques and powerful computers. Numerical computing is the basis underlying computational science. Virtually, any technical, medical or scientific discipline relies heavily on numerical computing. Scientists and engineers make intensive use of numerical methods and powerful computers to solve complex problems, which may range from the modeling of the microstructure of the atom to building design [1]. Numerical computing is foremost based upon floating point computation. However, computer representation of integer numbers [2] is also involved since it is always implicit and it is essential for almost any task in which the use of a computer is present. The most important computer representation for integer numbers is the well known 2’s complement, which constitutes the standard used in almost every modern computer. The main issues related to that format, which any computer scientist or engineer should be familiarized with, are the numeric ranges and how the arithmetic operations are carried out. Such issues are learned J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 269–283, 2003. c Springer-Verlag Berlin Heidelberg 2003
270
Ester M. Garz´ on et al.
in first-level courses in computer science and engineering, usually making use of pencil-and-paper exercises as support. Floating point computation [3] is undoubtedly of enormous importance in computational science. Computers have supported floating point computation since their earliest days, although using different representations developed by each computer manufacturer. However, in early 1980’s, the fruitful cooperation between academic computer scientists and the most important hardware manufacturers allowed the establishment of a binary floating point standard, commonly known as the IEEE 754 Floating Point Representation [4]. In essence, the standard aimed at (1) making floating point arithmetic as accurate as possible, within the constraints of finite precision arithmetic; (2) producing consistent and sensible outcomes in exceptional situations (e.g., overflow, underflow, infinite, ...); (3) standardizing floating point operations across computer systems; and (4) providing the programmer with control over exception handling (e.g. division by zero). Nowadays, most computers offer this standard for floating point computation. Due to the great importance of floating point computation, computer scientists and engineers should have an excellent knowledge of what a finite floating point representation and arithmetic involve and, in particular, of the ubiquitous IEEE 754 standard. First of all, from the point of view of numerical computing, computer scientists and engineers should be aware of extremely important issues such as, for instance, precision, ranges or algebraic properties related to any finite floating point representation. On the other hand, several aspects in the design of a computer system require a good knowledge on floating point. First, from the point of view of the computer architect, who has to deal with the design of instruction sets including floating point operations. Second, from the point of view of the compiler and programming language design, in the sense that the semantics of the language has to be defined precisely enough to prove statements about programs. Third, from the point of view of the operating system (as far as exception handling concerns), in the sense that trap handlers may be defined by the users/programmers to deal with the exceptions, according to the problem at hand. The Joint IEEE Computer Society and ACM Task Force on Computing Curricula actually develops curricular guidelines for undergraduate programs in different computing disciplines [5]. Machine-level representation of data has always been a core unit within the Computer Architecture and Organization area in the introductory phase of the undergraduate curriculum, and tightly related to the Programming Languages and Computational Science areas. In spite of its enormous importance, floating point representation still remains shrouded in mystery by the average computer science or engineering student, and only well understood by experts. Several initiatives have arisen in the nineties [6, 7, 8] and recently [10, 9] to make the floating point arithmetic accessible to the students. Some of them are focused directly on the IEEE floating point standard[7, 9, 10], explaining the representation itself and all the issues involved in it. Others [6, 8] make use of an intermediate floating point repre-
An Approach to Teaching Computer Arithmetic
271
sentation so as to illustrate the main concepts. The works of [6] and [10] also include exercises to clarify the concepts of precision and resolution involved in any finite floating point representation. Finally, [10] provides a set of computer programs to show clearly what the floating point computation using the IEEE standard involves. In this work, we present our own initiative to support the teaching of the computer representation of numbers and arithmetic (both integer and floating point), intended to be included in a first level course for undergraduate computer science or engineering students. Our initiative combines (1) a set of key practical exercises for both the integer and the floating point cases, (2) the use of a supporting environment consisting of a set of auxiliary computer programs, and (3) the use of intermediate floating point representations as an introduction to the IEEE standard, with the aim of facilitating the illustration of all the computational issues involved in the 2’s complement integer representation as well as in any floating point representation. Such an approach could be included in either a introductory course related to computer structure, discrete mathematics or numerical methods.
2
Teaching Computer Representation and Arithmetic of Integers
The most extended computer representation format for integer numbers is the well known 2’s complement. The range of the 2’s complement integer representation using strings of p bits is [−2p−1 , 2p−1 − 1]. This is a representation specially well suited from the point of view of the computer hardware, since it does not require additional special hardware for integer subtraction. In order to illustrate all the issues related to this representation, a complete set of carefully designed practical exercises have been designed, supported by an auxiliary computer-based environment. 2.1
Representation of Integer Numbers
The type of exercises in this category help the students to have experiences with the representation itself, the range of numbers that is covered, and different situations in which numbers do not fit ranges. In that sense, the exercises that are proposed include (1) conversions of integer numbers between decimal and 2’s complement formats using different word lengths, and vice versa; (2) calculation of the minimum number of bits that are needed to represent given numbers; (3) determination of 2’s complement representation ranges for different word lengths. As an example of practical exercise, we ask students to do decimal-to-2’s complement conversions of given numbers and then feed the results into the reverse conversion, using our support environment. As a result of such a “pipeline”, only those numbers that fit the corresponding representation ranges will result in themselves. Students are then encouraged to think about the reasons underlying
272
Ester M. Garz´ on et al.
the different results. With this type of exercises, students experience, on their own, the limits of the representation ranges and acquire skill to realize and deal with such situations in real life. 2.2
Integer Arithmetic
Exercises in this category mainly include the computation of integer arithmetic operations (addition, subtraction, multiplication and division) using different word lengths. This type of exercises and the software that has been developed illustrate the algorithms used to carry out the integer arithmetic in the hardware units. In addition, the understanding of exceptional situations derived from overflow is specially facilitated. Auxiliary programs have been included in the support environment to show in detail the procedures of the integer addition and subtraction, and, specially, the Booth algorithm for integer multiplication as well as the restoring and non-restoring algorithms for integer division. Such procedures are shown by the software according to the notation used in [2]. For the particular case of multiplication, our environment also affords the chance to show the procedure in a pencil-and-paper format. As an example of practical exercise related to integer arithmetic, we ask students to carry out different arithmetic operations with certain numbers and different word lengths, using the support environment. The numbers have been carefully chosen so that all possible situations occur, specially related to overflow. Figure 1 shows the output yielded by the support environment that describes the procedure of the integer multiplication of −5 and +7 using a word length of 4 bits through the Booth algorithm: The first column denotes the iteration of the algorithm; the second column represents the register that supposedly contains the multiplicand; the third column indicates the action to do in the corresponding step of the algorithm; finally, last column represents the doublesized register which initially contains the multiplier at its lower significant half, and into which the result of the multiplication is progressively computed and stored. The extra bit needed by the Booth algorithm is the right-most one in the last column.
3
Teaching Floating Point Representation and Arithmetic
Any floating point representation makes use of a exponential notation to represent real numbers, in which any number is decomposed into mantissa or significand and an exponent for a, normally, implicit base. For instance, a nonzero real number is represented by ±m × B E , with 1 ≤ m < B where m denotes the mantissa, E the exponent, and B the implicit base. Any computer-based representation involves a finite number of bits to represent the main fields of the floating point representation, mantissa and exponent. Consequently, rounding techniques have to be used for the real numerical value
An Approach to Teaching Computer Arithmetic
273
Multiplication (−5) × (+7)
Iter 0 1 2 3 4
multiplicand: −5 ≡ 10112 Multiplicand Action 1011 1011 1011 1011 1011 1011 1011 1011 1011
multiplier: +7 ≡ 01112 Product-Multiplier
Initial Values Prod=Prod - Multiplicand Right Shift No Operation Right Shift No Operation Right Shift Prod=Prod + Multiplicand Right Shift Final Result:
-35
0000 0101 0010 0010 0001 0001 0000 1011 1101
0111 0111 1011 1011 0101 0101 1010 1010 1101
0 0 1 1 1 1 1 1 0
==> 11011101
Fig. 1. Integer multiplication by means of the Booth algorithm. The operands −5 and +7 are multiplied using a word length of 4 bits to fit the number of bits of the representation format. Therefore, floating point computation is by nature inexact. In the particular case of the IEEE 754 standard, the single precision of the standard involves 32 bits, one bit for the sign, 23 bits are used for the mantissa and the exponent is represented by 8 bits, using an implicit binary base. Double precision in the standard makes use of 64 bits, 52 for the mantissa and 11 for the exponent. Finite floating point representation involves a considerable number of issues which any computer scientist or engineer should be aware of. The most important issues derived from the use of such a representation are: – There exists an interesting trade-off in terms of precision and range of the format. The number of bits in the exponent and the mantissa defines the range and the precision, respectively. – The gap between successive floating point numbers of the representation varies along the real numerical intervals. The gap is smaller as the magnitudes of the numbers themselves get smaller, and bigger as the numbers get bigger. – There is a relatively great gap between zero and the nearest non-zero floating point number. However, the use of denormalized numbers allows underflow to be gradual, evenly filling such a gap. Denormalized numbers are denoted by a zero exponent field and an unnormalized mantissa, assuming that the implicit bit is 0. In this way, the value of a denormalized number in the single precision IEEE standard is given by: Real value = (−1)s × (0.m) × 2−126 – The machine epsilon, mach refers to the gap between 1.0 and the smallest floating point number greater than 1.0. It provides an idea of the accuracy
274
– – – – –
Ester M. Garz´ on et al.
of the floating point operations: floating point values are accurate to within a factor of about 1 + mach [10]. Except for denormalized numbers, neighboring floating point numbers differ by one bit in the last bit of the mantissa, so mach = 2−nm . For the single precision IEEE standard, mach = 2−23 10−7 . The zero number is represented by a string of zero bits. Rounding, truncating and cancellation errors as well as error accumulation are heavily involved in the floating point arithmetic. Special quantities (Infinite, Not-a-Number) are designed to handle exceptional situations. Trap handlers for exception handling are under control of the user/programmer. Floating point representation introduces serious anomalies with respect to the conventional algebra, in the sense that some fundamental rules of arithmetic, such as the associative or distributive properties, are no longer guaranteed.
We have developed a relatively simple floating point representation format faithfully resembling all the important issues in the IEEE 754 floating point standard, with the aim of attenuating the relatively tedious aspect of teaching these kind of issues. This format is based on a user-defined word length so that the concepts related to precision and range is easily illustrated. The base format makes use of 12 bits, 1 for sign, 5 for the exponent and 6 for the mantissa. Because the anomalies arising as a result of a finite floating point representation are amplified, the use of this base format allows to easily show them, and therefore, facilitating the understanding by the student. Table 1 summarizes the most important features of this representation. This shorter version of the IEEE standard 754 has a five-fold aim: (1) an easy identification of the relationship between the real number and its floating point representation as well as the effects of the truncation/rounding in the conversion; (2) a clear illustration of the concepts related to the range/precision trade-off; (3) the understanding of the algorithms underlying the floating point arithmetic operations; (4) a clear identification of which, when and why anomalies arise in finite precision floating point computation (such anomalies include the effects of rounding errors in arithmetic operations, the non-fulfillment of the properties of the conventional algebra, and the exceptional situations); (5) the word length may be small enough to allow calculations by-hand, if convenient. In summary, the floating point format proposed here is specially well suited for illustrating all issues related to finite precision floating point representation, and in particular the IEEE standard 754. In that sense, a complete set of practical exercises have been thoroughly devised to highlight such aspects. Some of the exercises are based on the aforementioned format, but many others work on a general floating point format, by specifying different lengths for the exponent and mantissa fields. These practical exercises are accompanied by a computational environment that turns out to be an excellent auxiliary tool, from the pedagogical point of view, to illustrate such not-so-trivial issues. The practical exercises that have been devised fall into different categories:
An Approach to Teaching Computer Arithmetic
275
Table 1. The Simpler Floating Point Representation The most important features of the representation Length of the representation: 12 bits: sign: 1 bit exponent: ne =5 bits mantissa: nm =6 bits Exponent Bias: 2ne −1 − 1 = 15 = 011112 Range of the exponent: [−14, 15] Machine epsilon: mach = 2−nm = 2−6 Supported rounding modes: Truncation , Round to Nearest Denormalized numbers: Fully supported Supported exceptions: Overflow, Underflow, Invalid, Division by Zero The most important (positive) special values Special value Representation Value Largest normalized: 0 11110 111111 +1.984375 × 215 Smallest normalized: 0 00001 000000 +1.000000 × 2−14 Largest denormal.: 0 00000 111111 +0.984375 × 2−14 Smallest denormal.: 0 00000 000001 +0.015625 × 2−14 + Zero 0 00000 000000 +0.0 + Infinite: 0 11111 000000 --Not-a-Number (NaN): 0 11111 111111 ---
– The first category of exercises is intended to help students to learn the procedure to convert a real number to its floating point representation, analysing the effects of the rounding modes in terms of the relative error. In that sense, the exercises deal with the determination of the representation of different real numbers with the simpler floating point format (12 bits length), using different rounding techniques. Specially care has been taken to choose proper real values to give rise special situations in the conversions (for example, the smallest normalized floating point number, the biggest floating point number, the extreme values in the denormalized floating point range, etc). Figure 2 shows some examples of representations, as reported by the computer program in charge of the conversion. The first example was computed using truncation as the rounding mode, resulting in that the real number 1.999 is represented as 1.984375. The second, which turns out to be a denormalized number in this format, used round-to-nearest rounding mode. The third and fourth are examples of values bringing about exceptions: 0.0000001 is too small and 65500.0 is to big to be represented in the format. The output of the program shows the input real number, the representation, and the value of the floating point representation. – The following group of exercises aims at facilitating the understanding of the concepts of precision, range and the precision-range trade-off. Exercises in this group include the following aspects:
276
Ester M. Garz´ on et al.
Real Number: 1.999000 = 1.999000e+00 = 0.999500 x 2^1 Representation: 0 01111 111111 Float. P. Value: 1.984375 = 1.984375e+00 = 1.984375 x 2^0 Real Number: 0.000040 = 4.000000e-05 = 0.655360 x 2^-14 Representation: 0 00000 101010 Float. P. Value: 0.000040 = 4.005432e-05 = 0.656250 x 2^-14 !! Denormalized Number !! Real Number: 0.0000001 = 1.000000e-07 = 0.838861 x 2^-23 Representation: 0 00000 000000 Float. P. Value: 0.0000000 = 0.000000e+00 = 0.000000 x 2^0 ! Exception: Underflow ! Real Number: 65500.000000 = 6.550000e+04 = 0.999451 x 2^16 Representation: 0 11111 000000 Float. P. Value: +Infinite ! Exception: Overflow !
Fig. 2. Representation of certain values according to our floating point format, using 5 and 6 bits for exponent and mantissa, respectively • determination of the representation of different real numbers using different formats (i.e., varying bits for exponent and mantissa). • determination of the gap between floating point numbers in some intervals of different representation formats. • determination of the ranges covered for different floating point formats (including the sub-range of denormalized numbers, and that of normalized numbers). • computation of the minimum number of bits dedicated for the exponent and mantissa fields to fulfill certain given constraints, for example, to be able to distinguish between the representations of two given real numbers very close to each other. As an example, Figure 3 summarizes the process to work out the minimum number of bits (in exponent and in mantissa fields) required to distinguish the real numbers 15.9 and 15.925 and, at the same time, to represent the real number 100000. Students start working from the simpler format, using ne = 5, nm = 6, and then they progressively increase the number of bits in mantissa until 15.9 and 15.925 get distinguishable representations. In that figure, it can be seen that nm = 8 is the minimum number of bits required, which makes 15.9 and 15.925 get represented as 15.875 and 15.90625, respectively. Then, students try to get the representation of 100000 using the format just computed, obtaining an Overflow exception. They progressively increase the number of bits in the exponent field until 100000 gets representable. The final answer is that the minimum format to fulfill the requirements of the assignment is to use ne = 6, nm = 8. – The following class of exercises is related to arithmetic, and is intended to help students to learn the procedures to carry out the floating point arithmetic operations. Exercises in this group are mainly focused on the computation of floating point additions, subtractions, multiplications and divisions
An Approach to Teaching Computer Arithmetic
277
Determination of the minimum number of bits in mantissa field to distinguish 15.9 and 15.925 15.9 15.925 Floating Point Format Representation Representation ne = 5, nm = 6 0 10010 111111 0 10010 111111 ne = 5, nm = 7 0 10010 1111110 0 10010 1111110 ne = 5, nm = 8 0 10010 11111100 0 10010 11111101 ne = 5, nm = 8 15.9→15.875 15.925 →15.90625 Determination of the minimum number of bits to distinguish 15.9 and 15.925 and represent 100000 Real number: 100000 Floating Point Format Representation Value ne = 5, nm = 8 0 11111 00000000 +Infinite ne = 6, nm = 8 0 101111 10000110 99840.0
Fig. 3. Procedure to determine the minimum floating point format to fulfill the requirements that the real numbers 15.9 and 15.925 are distinguishable and that the real number 100000 is representable. Top: the minimum mantissa length to distinguish 15.9 and 15.925 is worked out to be nm = 8. Bottom: ne = 6 is the minimum exponent length required to represent 100000 using different pairs of operands, experiencing with the different rounding modes, and measuring the relative error of the results. The operands that are proposed are specially chosen so that different situations occur. Specially interesting is, for instance, the addition of pairs of operands that are enormously different in magnitude (for instance, our simpler format, using ne = 5 and nm = 6, makes 1000.0 + 2.5 equal to 1000.0). Figure 4 is intended to show the output of the program in charge of illustrating the process of floating point addition. As can be seen, the program specifies in detail all the stages in the procedure of such an arithmetic operation: (1) conversion of operands to their representation; (2) alignment of mantissas; (3) addition of mantissas; and finally (4) the result of the normalization and rounding. Such a program is extremely useful for students to discern all the issues concerning the round-off problems in arithmetic operations. This operation was performed using our simpler floating point representation (ne = 5, nm = 6). – This category deals with the algebraic anomalies that arise as a result of the finite nature of the floating point representation. It is imperative that students have insights into the machinations of floating point arithmetic on computers and, in particular, into those anomalies, to succeed in real life computation problems. So, exercises in this category include: • analysis of the cancellation error that arises in the subtraction of two operands extremely close to each other. • practical exercises to show the effect of accumulation of rounding errors. For instance, multiplication of a given floating point number by a integer constant by means of accumulated sum of the former, using different rounding modes. Students have to do a graph showing the evolution of the relative error during the accumulated sum. Figure 5 shows the result
278
Ester M. Garz´ on et al.
Floating Point Addition 1.0 + 0.999, using 2 guard bits. Format using: 5 bits/Exp. 6 bits/Mantissa; 2 Guard bits.
1.- Representation of operands. s e m s e Operand 1 -> 0 01111 000000 -> 0 01111 Operand 2 -> 0 01110 111111 -> 0 01110
.m 1.000000 1.111111
= 1.000000 = 0.999000
2.- Alignment of operands. Operand Operand
1 -> 2 ->
s e 0 01111 0 01111
.m g 1.000000 00 0.111111 10
3.- Addition of mantissas. s e .m g Operand 1 -> 0 01111 1.000000 00 Operand 2 -> 0 01111 0.111111 10 -----------------------------------Addition -> 0 01111 1.111111 10 4.- Normalization and Rounding of the result. Result
->
0 10000
1.000000
= 2.000000
Result of the Addition: 0 10000 000000 Decimal Value: 2.000000 = 2.000000e+00 = 1.000000 x 2^1
Fig. 4. Procedure and result of the floating point addition of 1.0 and 0.999 of the multiplication 1.999 × 10 as shown by the auxiliary programs. Here, the intermediate results from the accumulated sum are shown. On the left, the sums are carried out using truncation. On the right, the round-to-nearest rounding mode is used. The last line shows the final result of the multiplication. Clearly, the superiority of the roundto-nearest rounding mode is evident. These results are rapidly generated using one of the programs of our environment. Students have to analyze the results, and generate graphs of the error evolution. • practical exercises to show that some fundamental rules of the conventional algebra are not fulfilled in floating point computation: ∗ floating point addition is not associative. ∗ floating point multiplication is not associative. ∗ floating point multiplication does not necessarily distribute over addition ∗ ordering of operations is significant. ∗ the cancellation property is not always valid, i.e., there exist positive floating point numbers A, B, C such that A+B = A +C and B = C. ∗ multiplication of a floating point number by its inverse is not always equal to 1. ∗ it is almost always wrong to ask whether two floating point numbers are equal.
An Approach to Teaching Computer Arithmetic
279
Multiplication 1.999 × 10 Truncation Round to Nearest Iter Representation Value Representation Value ------------------------------------- ----------------------------1 -> 0 01111 111111 -> 1.99900 0 01111 111111 -> 1.99900 2 -> 0 10000 111111 -> 3.96875 0 10000 111111 -> 3.96875 3 -> 0 10001 011111 -> 5.93750 0 10001 011111 -> 5.93750 4 -> 0 10001 111110 -> 7.87500 0 10001 111111 -> 7.93750 5 -> 0 10010 001110 -> 9.75000 0 10010 001111 -> 9.87500 6 -> 0 10010 011101 -> 11.62500 0 10010 011111 -> 11.87500 7 -> 0 10010 101100 -> 13.50000 0 10010 101111 -> 13.87500 8 -> 0 10010 111011 -> 15.37500 0 10010 111111 -> 15.87500 9 -> 0 10011 000101 -> 17.25000 0 10011 000111 -> 17.75000 10 -> 0 10011 001100 -> 19.00000 0 10011 001111 -> 19.75000
Fig. 5. Multiplication 1.999 × 10 by means of an accumulated sum. The simpler floating point representation has been used (5/6 bits for exponent/mantissa) One of the most interesting examples of algebraic anomalies is related to the computation of Harmonic series: Hn = 1 + 1/2 + 1/3 + . . . + 1/n. We use such a series to show how the ordering of operations may be extremely significant. We propose to compute the series, first, literally as in the formula and, second, in reverse order. The results help students to learn that summing the smallest values first, progressively increasing in magnitude, yields more accurate final results. Such an ordering avoids loosing the lowest precision bits of the smallest quantities in summing with values very different in magnitude. Figure 6 shows the results generated by our software package. This exercise aims at highlighting the fact that the ordering of floating point operations may be significant. On the left, the results for an ordering according to the original formula. On the right, the results for the reverse ordering. The results include the current term that is to be added in the series, and the intermediate value of the accumulated sum. On the other hand, such a series is mathematically proven to diverge. However, in floating point representations, the series converge due to the roundoff errors. We have also designed practical exercises on that. – The last category of exercises comprises those related to exceptions. Since the IEEE standard 754 allows exception handling to be under control of the user/programmer, it is extremely important to familiarize students with the situations that produce exceptions, how to manage them, and the floating point values resulting from exceptions. The exercises in this category are mainly intended to come exceptions into manifest: • practical exercises to show that some arithmetic operations with, in principle, normal numbers may result in exceptions because the operation yields a result not representable in the format. For instance, the multiplication of extremely small floating point numbers may give rise to an
280
Ester M. Garz´ on et al.
Computation of Harmonic series
10
1 n=1 n
Index ----1 2 3 4 5 6 7 8 9 10
Number Sum 1..N --------------------------1/1 = 1.00000 1.000000 1/2 = 0.50000 1.500000 1/3 = 0.33333 1.828125 1/4 = 0.25000 2.062500 1/5 = 0.20000 2.250000 1/6 = 0.16667 2.406250 1/7 = 0.14286 2.562500 1/8 = 0.12500 2.687500 1/9 = 0.11111 2.812500 1/10 = 0.10000 2.906250
1
1 n=10 n
Number Sum N..1 --------------------------1/10 = 0.10000 0.100000 1/9 = 0.11111 0.210938 1/8 = 0.12500 0.335938 1/7 = 0.14286 0.476562 1/6 = 0.16667 0.640625 1/5 = 0.20000 0.843750 1/4 = 0.25000 1.093750 1/3 = 0.33333 1.421875 1/2 = 0.50000 1.921875 1/1 = 1.00000 2.937500
Fig. 6. Computation of the Harmonic series with N = 10 using our simpler floating point format (5 bits exp., 6 bits mantissa), and using different orderings
Underflow exception, or the addition/multiplication of two big numbers may result in an Overflow exception, etc. Two concrete examples are that 0.0009 ∗ 0.001 produces an Underflow exception and 1875∗ 35 brings about an Overflow one, if our simpler format (with ne = 5 and nm = 6) is used. • computation of A + ∞, A ∗ ∞, A/0, A/∞, etc, given a floating point number A. • computation of ∞ + 0, ∞ + ∞, ∞ − ∞, ∞ ∗ 0, ∞ ∗ (−∞), 0/0, ∞/∞, ∞/0, etc. Figure 7 shows the exception that is brought about and the corresponding floating point value that would be generated for different singular situations. Such a behavior should be exhibited by any floating point system guided by the IEEE standard. To conclude this section, we should mention that all the practical exercises that we have devised are supported by the computer-based environment that will be described in the following section. The programs in the environment are intended to provide answers and their justifications to all the questions and exercises formulated in the assignments.
4
The Support Computational Environment
We have developed a support computer-based environment for students to practice number representation and arithmetic in computers, facilitating them the understanding of the computational issues in this field. The environment consists of computer programs, written in standard C programming language, which
An Approach to Teaching Computer Arithmetic
281
internally use techniques derived from multiprecision arithmetic to compute the representations and the arithmetic operations. The floating point values returned by the programs are computed using double precision data types. We have tested the programs in different computer platforms (PC, SGI, Sun, Alpha) and operating systems (Windows, Linux, IRIX, Solaris) obtaining coherent results. The programs can be executed on console, specifying the input operands and options in command line. In the particular case of the programs related to floating point, the options include the possibility of specifying the number of bits for the exponent and mantissa fields so that students can experience with representations different from the default (ne = 5, nm = 6). The environment is multilingual in the sense that Spanish and English languages are currently supported. An option in compilation time allows to activate the language that has been chosen. We have also developed a friendly web-based front-end for the programs which is based on the client/server model. Students need not deal directly with the console and command lines but, instead, they access the environment through their web browser/client (e.g. netscape) and, transparently, execute the programs in the web server. This front-end is based on HTML (HyperText Markup Language) forms and CGI (Common Gateway Interface) technology [11]. Students use their web browser/client to (1) select the program to be executed, (2) specify the operands and options in the form, and (3) send the data to the server. CGI technology is in charge of obtaining the data sent by the web client, executing in the server the corresponding program with the proper parameters and, finally, sending the results back to the browser/client. Our environment, currently available at http://www.ace.ual.es/RAC/, is structured into two main blocks: on the one hand, all the programs concerning integer representation and arithmetic, and those concerning floating point representation and arithmetic. There is an accompanying manual containing the theoretical concepts and an exhaustive list of assignments that covers the wide spectrum of the issues in computer number representation and arithmetic. This environment proves to be, from the pedagogical point of view, an excellent auxiliary tool for students to learn about the finite length number repre-
Singular operation Exception Returned value A−∞ Overflow -Infinite ∞+∞ Overflow +Infinite ∞−∞ Invalid NaN A∗∞ Overflow ±Infinite ∞∗0 Invalid NaN ∞ ∗ (−∞) Overflow -Infinite A/0 Division-by-zero ±Infinite A/∞ Underflow ±Zero 0/0 Invalid NaN ∞/∞ Invalid NaN −∞/0 Overflow -Infinite
Fig. 7. Different situations that produce exceptions in our floating point format. The symbol ± means that the sign is dependent on the signs of the input operands
282
Ester M. Garz´ on et al.
sentation and arithmetic in computers from a hands-on, practical, perspective. Thanks to this environment and the set of carefully devised exercises, students can have insights into the not-so-trivial issues in this field.
5
Conclusions
In this article we have described an approach to teaching computer representation of numbers and arithmetic. This initiative mainly consists of a complete set of thoroughly designed practical exercises conceived to emphasize all the important issues in finite length computer arithmetic. A computational environment that turns out to be a very valuable support tool for students to practice is also provided. Our experience after several years of lecturing and teaching computer arithmetic in an introductory course on computer organization in undergraduate computer science curricula allows us to claim the success of this initiative. As far as the integer representation is concerned, we have experienced that the use of carefully designed practical exercises allows students to learn easily how the 2’s complement represents the integer numbers, and all the issues regarding range and word lengths. The algorithms and the computer hardware involved in the integer arithmetic are also facilitated. In conjunction to the practical exercises, the computer-based environment devised in this work provides students with a valuable support for their learning. From the pedagogical point of view, we have experienced that teaching floating point representation based upon a simple format which helps us illustrate all the issues in any floating point representation system is successful. We use this simple format as an introduction to the IEEE 754 floating point standard. Students feel more comfortable to deal with the IEEE standard once they are familiarized with the simple format and all the computational issues involved. The use of a simpler format allows all the singularities of floating point representation to be highlighted and amplified, and consequently, students are more impressed by those effects. In ultimate instance, this approach facilitates lecturers to achieve the goal as a academic staff of making students aware of the computational issues involved in finite length number representation and arithmetic. This approach may be valuable for introductory undergraduate courses related to computer organization, programming, discrete mathematics and numerical methods in computer science or engineerings.
Acknowledgments This work has been partially supported by the Spanish CICYT through grants TIC99-0361 and TIC2002-00228.
An Approach to Teaching Computer Arithmetic
283
References [1] C. W. Ueberhuber Numerical Computation. Methods, Software, and Analysis, Vols. 1&2. Springer-Verlag, 1997. 269 [2] D. A. Patterson and J. L. Hennessy. Computer Organization and Design. The Hardware/Software Interface Morgan Kaufmann Pub., 1998. 269, 272 [3] D. E. Knuth. The Art of Computer Programming, 3rd ed, volume 2, Seminumerical Algorithms. Addison-Wesley, 1998. 270 [4] W. Kahan. IEEE Standard 754 for Binary Floating-Point Arithmetic. WWW document, 1996. http://www.cs.berkeley.edu/ wkahan/ieee754status/ieee754.ps. 270 [5] Curriculum 2001 Joint IEEE Computer Society/ACM Task Force. “Year 2001 Model Curricula for Computing ((CC-2001),” Final report, December 15, 2001. 270 [6] T. J. Scott. “Mathematics and computer science at odds over real numbers,” ACM SIGCSE Bulletin, 23(1):130–139, 1991. 270, 271 [7] D. Goldberg. “What every computer scientist should know about floating-point arithmetic,” ACM Comp. Surveys, 23:5–48, 1991. 270 [8] C. W. Steidley. “Floating point arithmetic basic exercises in mathematical reasoning for computer science majors,” Computers in Education Journal, 2(4):1–6, 1992. 270 [9] W. Kahan. Ruminations on the design of floating-point arithmetic. WWW document, 2000. http://www.cs.nyu.edu/cs/faculty/overton/book/docs/). 270 [10] M. L. Overton. Numerical Computing and the IEEE Floating Point Standard. SIAM, 2001. 270, 271, 274 [11] S. Guelich, S. Gundavaram, G. Birznieks. CGI Programming on the World Wide Web. O’Reilly, 2000. 281
Fast Sparse Matrix-Vector Multiplication for TeraFlop/s Computers Gerhard Wellein1 , Georg Hager1 , Achim Basermann2 , and Holger Fehske3 1
2
Regionales Rechenzentrum Erlangen, D-91058 Erlangen, Germany C&C Research Laboratories, NEC Europe Ltd, D-53757 Sankt Augustin, Germany 3 Institut f¨ ur Physik, Universit¨ at Greifswald, D-17487 Greifswald, Germany Abstract. Eigenvalue problems involving very large sparse matrices are common to various fields in science. In general, the numerical core of iterative eigenvalue algorithms is a matrix-vector multiplication (MVM) involving the large sparse matrix. We present three different programming approaches for parallel MVM on present day supercomputers. In addition to a pure message-passing approach, two hybrid parallel implementations are introduced based on simultaneous use of message-passing and shared-memory programming models. For a modern SMP cluster (HITACHI SR8000) performance and scalability of the hybrid implementations are discussed and compared with the pure message-passing approach on massively-parallel systems (CRAY T3E), vector computers (NEC SX5e) and distributed shared-memory systems (SGI Origin3800).
1
Introduction
Clusters of multiprocessor shared-memory (SMP) nodes are presently the common way to exceed the TFlop/s barrier. Although these systems outperform all other architectures with respect to peak performance there is still the question how to write portable codes which run and scale efficiently. The simplest approach is to use massively-parallel (MPP) codes totally ignoring the sharedmemory within SMP nodes. A more promising approach seems to be the “hybrid programming model” which uses both message-passing model (between SMP nodes) and shared-memory model (within SMP nodes) simultaneously and maps most closely to the memory structure of the underlying target architecture. It is evident that the benefit of either programming model depends on the algorithm and the programming style used. Some studies for application programs have already been done in this context [1, 2, 3, 4, 5, 6] leading to different results. Numerical cores of many applications are iterative sparse matrix algorithms such as sparse eigenvalue solvers. In practice, such operations will have a performance bound by the memory access. Since the LINPACK benchmark [7] usually does not cover this quantity, an iterative solver benchmark (sparsebench) has been developed recently by Dongarra et al. [8]. The sparsebench benchmark solves sparse linear systems using different methods and storage schemes but is available in a non-parallel version for small problem sizes only. The aim of this work is to discuss three different parallel implementations of a sparse matrix-vector multiplication (MVM) for SMP clusters: J.M.L.M. Palma et al. (Eds.): VECPAR 2002, LNCS 2565, pp. 287–301, 2003. c Springer-Verlag Berlin Heidelberg 2003
288
Gerhard Wellein et al.
MPP-mode : Each MPI process is assigned to exactly one processor of a SMP node. This implementation makes no use of the hardware structure. Vector-mode : One SMP node is identified as a single process with high peakperformance and memory bandwidth similar to a single vector processor. Shared-memory parallelisation of long vector loops is done. This implementation might be beneficial for vector codes on SMP nodes providing fast collective thread operations. Task-mode : Using OpenMP we perform an explicit programming of each processor of a SMP node to overlap communication and computation. An asynchronous data transfer can be simulated by defining one processor to be responsible for communication. This strategy improves performance on SMP nodes without hardware support for asynchronous data transfer and for codes where a sufficient overlap of communication and computation exists. The different implementations have been applied to extreme sparse matrices (originating from large scale eigenvalue problems in theoretical physics) with dimensions ranging from 103 to 108 and have been run on a maximum of 1024 processors. For completeness a detailed performance comparison with classical supercomputers is given.
2
Implementation of Matrix-Vector Multiplication
Generally, the performance of iterative sparse eigenvalue solvers is determined by a matrix-vector multiplication step involving the large sparse matrix. There are two widely used methods to exploit the sparsity of the matrix to minimize the memory requirements in the MVM step which show different performance characteristics. The Compressed Row Storage (CRS) format [9] is generally used for RISC systems but has serious drawbacks on vector processors due to short inner loop lengths. The Jagged Diagonal Storage (JDS) scheme [9] provides long inner vector loops at the cost of minor performance drawbacks on RISC systems for matrices used in our work [10]. To achieve high performance on a wide range of supercomputer architectures, a parallel JDS scheme was implemented. 2.1
JDS Matrix-Vector Multiplication
The JDS format is a very general storage scheme for sparse matrices making no assumptions about the sparsity pattern. In a compression step, the zero elements are removed, nonzero elements are shifted to the left and the rows are m