Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
6067
Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wasniewski (Eds.)
Parallel Processing and Applied Mathematics 8th International Conference, PPAM 2009 Wroclaw, Poland, September 13-16, 2009 Revised Selected Papers, Part I
13
Volume Editors Roman Wyrzykowski Konrad Karczewski Czestochowa University of Technology Institute of Computational and Information Sciences, Poland E-mail:{roman, xeno}@icis.pcz.pl Jack Dongarra University of Tennessee, Department of Electrical Engineering and Computer Science, Knoxville, TN 37996-3450, USA E-mail:
[email protected] Jerzy Wasniewski Technical University of Denmark, Department of Informatics and Mathematical Modeling, 2800 Kongens Lyngby, Denmark E-mail:
[email protected]
Library of Congress Control Number: 2010930174 CR Subject Classification (1998): D.2, H.4, D.4, C.2.4, D.1.3, F.2 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues ISSN ISBN-10 ISBN-13
0302-9743 3-642-14389-X Springer Berlin Heidelberg New York 978-3-642-14389-2 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180
Preface
We are pleased to present the proceedings of the 8th International Conference on Parallel Processing and Applied Mathematics – PPAM 2009, which was held in Wroclaw, Poland, September 13–16, 2009. It was organized by the Department of Computer and Information Sciences of the Czestochowa University of Techno logy, with the help of the Wroclaw University of Technology, Faculty of Computer Science and Management. The main organizer was Roman Wyrzykowski. PPAM is a biennial conference. Seven previous events have been held in different places in Poland since 1994. The proceedings of the last four conferences have been published by Springer in the Lecture Notes in Computer Science series 2003, vol.3019; Pozna´ n, 2005, vol.3911; (Nalecz´ ow, 2001, vol.2328; Czestochowa, Gda´ nsk, 2007, vol. 4967). The PPAM conferences have become an international forum for exchanging ideas between researchers involved in parallel and distributed computing, including theory and applications, as well as applied and computational mathematics. The focus of PPAM 2009 was on models, algorithms, and software tools which facilitate efficient and convenient utilization of modern parallel and distributed computing architectures, as well as on large-scale applications. This meeting gathered more than 210 participants from 32 countries. A strict refereeing process resulted in the acceptance of 129 contributed presentations, while approximately 46% of the submissions were rejected. Regular tracks of the conference covered such important fields of parallel/distributed/grid computing and applied mathematics as: – – – – – –
Parallel/distributed architectures and mobile computing Numerical algorithms and parallel numerics Parallel and distributed non-numerical algorithms Tools and environments for parallel/distributed/grid computing Applications of parallel/distributed computing Applied mathematics and neural networks
Plenary and Invited Speakers The plenary and invited talks were presented by: – – – –
Srinivas Aluru from the Iowa State University (USA) Dominik Behr from AMD (USA) Ewa Deelman from the University of Southern California (USA) Jack Dongarra from the University of Tennessee and Oak Ridge National Laboratory (USA) – Iain Duff from the Rutherford Appleton Laboratory (UK) – Anne C. Elster from NTNU, Trondheim (Norway)
VI
– – – – – – – – –
Preface
Wolfgang Gentzsch from the DEISA Project Michael Gschwind from the IBM T.J. Watson Research Center (USA) Fred Gustavson from the IBM T.J. Watson Research Center (USA) Simon Holland from Intel (UK) Vladik Kreinovich from the University of Texas at El Paso (USA) Magnus Peterson from the Synective Labs (Sweden) Armin Seyfried from the Juelich Supercomputing Centre (Germany) Boleslaw Szyma´ nski from the Rensselaer Polytechnic Institute (USA) Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)
Workshops and Minisymposia Important and integral parts of the PPAM 2009 conference were the workshops: – Minisymposium on GPU Computing organized by Jos´e R. Herrero from the Universitat Politecnica de Catalunya (Spain), Enrique S. Quintana-Ort´ı from the Universitat Jaime I (Spain), and Robert Strzodka from the Max-PlanckInstitut f¨ ur Informatik (Germany) – The Second Minisymposium on Cell/B.E. Technologies organized by Roman University of Technology (Poland), and Wyrzykowski from the Czestochowa David A. Bader from the Georgia Institute of Technology (USA) – Workshop on Memory Issues on Multi- and Manycore Platforms organized by Michael Bader and Carsten Trinitis from the TU M¨ unchen (Germany) – Workshop on Novel Data Formats and Algorithms for High-Performance Computing organized by Fred Gustavson from the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Workshop on Scheduling for Parallel Computing - SPC 2009 organized by Maciej Drozdowski from the Pozna´ n University of Technology (Poland) – The Third Workshop on Language-Based Parallel Programming Models WLPP 2009 organized by Ami Marowka from the Shenkar College of Engineering and Design in Ramat-Gan (Israel) – The Second Workshop on Performance Evaluation of Parallel Applications on Large-Scale Systems organized by Jan Kwiatkowski, Dariusz Konieczny and Marcin Pawlik from the Wroclaw University of Technology (Poland) – The 4th Grid Application and Middleware Workshop - GAMW 2009 organized by Ewa Deelman from the University of Southern California (USA), and Norbert Meyer from the Pozna´ n Supercomputing and Networking Center (Poland) – The 4th Workshop on Large Scale Computations on Grids - LaSCoG 2009 organized by Marcin Paprzycki from IBS PAN and SWPS in Warsaw (Poland), and Dana Petcu from the Western University of Timisoara (Romania) – Workshop on Parallel Computational Biology - PBC 2009 organized by David A. Bader from the Georgia Institute of Technology in Atlanta (USA), Denis Trystram from ID-IMAG in Grenoble (France), Alexandros Stamatakis ˙ from the TU M¨ unchen (Germany), and Jaroslaw Zola from the Iowa State University (USA)
Preface
VII
– Minisymposium on Applications of Parallel Computations in Industry and ˇ Engineering organized by Raimondas Ciegis from the Vilnius Gediminas ˇ Technical University (Lithuania), and Julius Zilinskas from the Institute of Mathematics and Informatics in Vilnius (Lithuania) – The Second Minisymposium on Interval Analysis organized by Vladik Kreinovich from the University of Texas at El Paso (USA), Pawel University of Technology (Poland), Sewastjanow from the Czestochowa Bartlomiej J. Kubica from the Warsaw University of Technology (Poland), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark) – Workshop on Complex Collective Systems organized by Pawel Topa and Jaroslaw Was from the AGH University of Science and Technology in Cracow (Poland)
Tutorials The PPAM 2009 meeting began with four tutorials: – GPUs, OpenCL and Scientific Computing, by Robert Strzodka from the Max-Planck-Institut f¨ ur Informatik (Germany), Dominik Behr from AMD (USA), and Dominik G¨ oddeke from the University of Dortmund (Germany) – FPGA Programming for Scientific Computing, by Magnus Peterson from the Synective Labs (Sweden) – Programming the Cell Broadband Engine, by Maciej Remiszewski from IBM (Poland), and Maciej Cytowski from the University of Warsaw (Poland) – New Data Structures Are Necessary and Sufficient for Dense Linear Algebra Factorization Algorithms, by Fred Gustavson from the the IBM T.J. Watson Research Center (USA), and Jerzy Wa´sniewski from the Technical University of Denmark (Denmark)
Best Poster Award The PPAM Best Poster Award is given to the best poster on display at the PPAM conferences, and was first awarded at PPAM 2009. This award is bestowed by the Program Committee members to the presenting author(s) of the best poster. The selection criteria are based on the scientific content and on the quality of the poster presentation. The PPAM 2009 winner was Tomasz Olas University of Technology, who presented the poster “Parfrom the Czestochowa allel Adaptive Finite Element Package with Dynamic Load Balancing for 3D Thermomechanical Problems.”
New Topics at PPAM 2009 GPU Computing: The recent advances in the hardware, functionality, and programmability of graphics processors (GPUs) have greatly increased their appeal
VIII
Preface
as add-on co-processors for general-purpose computing. With the involvement of the largest processor manufacturers and the strong interest from researchers of various disciplines, this approach has moved from a research niche to a forwardlooking technique for heterogeneous parallel computing. Scientific and industry researchers are constantly finding new applications for GPUs in a wide variety of areas, including image and video processing, molecular dynamics, seismic simulation, computational biology and chemistry, fluid dynamics, weather forecast, computational finance, and many others. GPU hardware has evolved over many years from graphics pipelines with many heterogeneous fixed-function components over partially programmable architectures towards a more and more homogeneous general purpose design, although some fixed-function hardware has remained because of its efficiency. The general-purpose computing on GPU (GPGPU) revolution started with programmable shaders; later, NVIDIA Compute Unified Device Architecture (CUDA) and to a smaller extent AMD Brook+ brought GPUs into the mainstream of parallel computing. The great advantage of CUDA is that it defines an abstraction which presents the underlying hardware architecture as a sea of hundreds of fine-grained computational units with synchronization primitives on multiple levels. With OpenCL there is now also a vendor-independent high-level parallel programming language and an API that offers the same type of hardware abstraction. GPU are very versatile accelerators because besides the high hardware parallelism they also feature a high bandwidth connection to dedicated device memory. The latency problem of DRAM is tackled via a sophisticated thread scheduling and switching mechanism on-chip that continues the processing of the next thread as soon as the previous stalls on a data read. These characteristics make GPUs suitable for both compute- and data-intensive parallel processing. The PPAM 2009 conference recognized the great impact of GPUs by including in its scientific program two major related events: a minisymposium on GPU Computing, and a full day tutorial on “GPUs, OpenCL and Scientific Computing.” The minisymposium received 18 submissions, of which 10 were accepted (55%). The contributions were organized in three sessions. The first group was related to Numerics, and comprised the following papers: “Finite Element Numerical Integration on GPUs,”“ Reduction to Condensed Forms for Symmetric Eigenvalue Problems on Multi-core Architectures,”“On Parallelizing the MRRR Algorithm for Data-Parallel Coprocessors,” and “A Fast GPU Implementation for Solving Sparse Ill-Posed Linear Equation Systems.” The second session dealt with Applications. The papers presented were: “Simulations of the Electrical Activity in the Heart with Graphic Processing Units,”“Stream Processing on GPUs Using Distributed Multimedia Middleware,” and “A GPU Approach to the Simulation of Spatio–temporal Dynamics in Ultrasonic Resonators.” Finally, a third session about General GPU Computing included presentations of three papers: “Fast In-Place Sorting with CUDA Based on Bitonic Sort,” “Parallel Minimax
Preface
IX
Tree Searching on GPU,” and “Modeling and Optimizing the Power Performance of Large Matrices Multiplication on Multi-core and GPU Platform with CUDA.” The tutorial covered a wide variety of GPU topics and also offered handson examples of OpenCL programming that any paticipant could experiment with on their laptop. The morning sessions discussed the basics of GPU architecture, ready-to-use libraries and OpenCL. The afternoon session went in depth on OpenCL and scientific computing on GPUs. All slides are available at http://gpgpu.org/ppam2009. Complex Collective Systems: Collective aspects of complex systems are attracting an increasing community of researchers working in different fields and dealing with theoretical aspects as well as practical applications. In particular, analyzing local interactions and simple rules makes it possible to model complex phenomena efficiently. Collective systems approaches show great promise in establishing scientific methods that could successfully be applied across a variety of application fields. Many studies in complex collective systems science follow either a cellular automata (CA) method or an agent-based approach. Hybridization between these two complementary approaches gives a promising perspective. The majority of work presented during the workshop on complex collective systems represents the hybrid approach. We can distinguish four groups of subjects presented during the workshop. The first group was modeling of pedestrian dynamics: Armin Seyfried from the Juelich Supercomputing Center presented actual challenges in pedestrian dynamics modeling. Another important issue of crowd modeling was also taken into account during the workshop: modeling of stop-and-go waves (Andrea Portz and Armin Seyfried), calibration of pedestrian stream models (Wolfram Klein, Gerta K¨ oster and Andreas Meister), parallel design patterns in a pedestrian simulation (Sarah Clayton), floor fields models based on CA (Ekaterina Kirik, Tat’yana Yurgel’yan and Dmitriy Krouglov), and discrete potential field construction (Konrad Kulakowski and Jaroslaw Was). The second group dealt with models of car traffic: a fuzzy cellular model of traffic (Bartlomiej Placzek), and an adaptive time gap car-following model (Antoine Tordeux and Pascal Bouvry). The third group included work connected with cryptography based on cellular automata: weakness analysis of a key stream generator (Frederic Pinel and Pascal Bouvry), and properties of safe CA-based S-Boxes (Miroslaw Szaban and Franciszek Seredy´ nski). The fourth group dealt with various applications in a field of complex collective systems: frustration and collectivity in spatial networks (Anna Ma´ nkaKraso´ n, Krzysztof Kulakowski), lava flow hazard modeling (Maria Vittoria Avolio, Donato D’Ambrosio, Valeria Lupiano, Rocco Rongo and William Spataro), FPGA realization of a CA-based epidemic processor (Pavlos Progias, Emmanouela Vardaki and Georgios Sirakoulis)
X
Preface
Acknowledgements The organizers are indebted to the PPAM 2009 sponsors, whose support was vital to the success of the conference. The main sponsor was the Intel Corporation. The other sponsors were: Hewlett-Packard Company, Microsoft Corporation, IBM Corporation, Action S.A., and AMD. We thank to all members of the International Program Committee and additional reviewers for their diligent work in refereeing the submitted papers. Finally, we thank all of the local organizers from the Czestochowa University of Technology and Wroclaw University of Technology who helped us to run the event very smoothly. We are especially indebted to Gra˙zyna Kolakowska, Urszula Kroczewska, L ukasz Kuczy´ nski, and University of Technology; and to Jerzy Marcin Wo´zniak from the Czestochowa ´ atek, and Jan Kwiatkowski from the Wroclaw University of Technology. Swi
PPAM 2011 We hope that this volume will be useful to you. We would like everyone who reads it to feel invited to the next conference, PPAM 2011, which will be held September 11–14, 2011, in Toru´ n, a city in northern Poland where the great astronomer Nicolaus Copernicus was born. February 2010
Roman Wyrzykowski Jack Dongarra Konrad Karczewski Jerzy Wa´sniewski
Organization
Program Committee Jan Weglarz Roman Wyrzykowski Boleslaw Szyma´ nski Peter Arbenz Piotr Bala David A. Bader Michael Bader Mark Baker Radim Blaheta Jacek Bla˙zewicz Leszek Borzemski Pascal Bouvry Tadeusz Burczy´ nski Jerzy Brzezi´ nski Marian Bubak ˇ Raimondas Ciegis Andrea Clematis Zbigniew Czech Jack Dongarra Maciej Drozdowski Erik Elmroth Anne C. Elster Mariusz Flasi´ nski Maria Ganzha Jacek Gondzio Andrzej Go´sci´ nski Laura Grigori Frederic Guinand Jos´e R. Herrero Ladislav Hluchy Ondrej Jakl Emmanuel Jeannot Grzegorz Kamieniarz Alexey Kalinov Ayse Kiper
Pozna´ n University of Technology, Poland Honorary Chair University of Technology, Poland Czestochowa Chair Rensselaer Polytechnic Institute, USA Vice-Chair ETH, Zurich, Switzerland N. Copernicus University, Poland Georgia Institute of Technology, USA TU M¨ unchen, Germany University of Reading, UK Institute of Geonics, Czech Academy of Sciences Pozna´ n University of Technology, Poland Wroclaw University of Technology, Poland University of Luxembourg Silesia University of Technology, Poland Pozna´ n University of Technology, Poland Institute of Computer Science, AGH, Poland Vilnius Gediminas Tech. University, Lithuania IMATI-CNR, Italy Silesia University of Technology, Poland University of Tennessee and ORNL, USA Pozna´ n University of Technology, Poland Umea University, Sweden NTNU, Trondheim, Norway Jagiellonian University, Poland IBS PAN, Warsaw, Poland University of Edinburgh, Scotland, UK Deakin University, Australia INRIA, France Universit´e du Havre, France Universitat Politecnica de Catalunya, Barcelona, Spain Slovak Academy of Sciences, Bratislava Institute of Geonics, Czech Academy of Sciences INRIA, France A. Mickiewicz University, Pozna´ n, Poland Cadence Design System, Russia Middle East Technical University, Turkey
XII
Organization
Jacek Kitowski Jozef Korbicz Stanislaw Kozielski Dieter Kranzlmueller
Institute of Computer Science, AGH, Poland University of Zielona G´ ora, Poland Silesia University of Technology, Poland Ludwig Maximillian University, Munich, and Leibniz Supercomputing Centre, Germany Henryk Krawczyk Gda´ nsk University of Technology, Poland Piotr Krzy˙zanowski University of Warsaw, Poland Jan Kwiatkowski Wroclaw University of Technology, Poland Giulliano Laccetti University of Naples, Italy Marco Lapegna University of Naples, Italy Alexey Lastovetsky University College Dublin, Ireland Vyacheslav I. Maksimov Ural Branch, Russian Academy of Sciences Victor E. Malyshkin Siberian Branch, Russian Academy of Sciences Tomas Margalef Universitat Autonoma de Barcelona, Spain Ami Marowka Shenkar College of Engineering and Design, Israel Norbert Meyer PSNC, Pozna´ n, Poland Jarek Nabrzyski University of Notre Dame, USA Marcin Paprzycki IBS PAN and SWPS, Warsaw, Poland Dana Petcu Western University of Timisoara, Romania Enrique S. Quintana-Ort´ı Universitat Jaime I, Spain Yves Robert Ecole Normale Superieure de Lyon, France Jacek Rokicki Warsaw University of Technology, Poland University of Technology, Poland Leszek Rutkowski Czestochowa Franciszek Seredy´ nski Polish Academy of Sciences and Polish-Japanese Institute of Information Technology, Warsaw, Poland Robert Schaefer Institute of Computer Science, AGH, Poland Jurij Silc Jozef Stefan Institute, Slovenia Peter M.A. Sloot University of Amsterdam, The Netherlands Masha Sosonkina Ames Laboratory and Iowa State University, USA Leonel Sousa Technical University Lisbon, Portugal Maciej Stroi´ nski PSNC, Pozna´ n, Poland Domenico Talia University of Calabria, Italy Andrei Tchernykh CICESE, Ensenada, Mexico Carsten Trinitis TU M¨ unchen, Germany Roman Trobec Jozef Stefan Institute, Slovenia Denis Trystram ID-IMAG, Grenoble, France Marek Tudruj Polish Academy of Sciences and Polish-Japanese Institute of Information Technology, Warsaw, Poland Pavel Tvrdik Czech Technical University, Prague Jens Volkert Johannes Kepler University, Linz, Austria Jerzy Wa´sniewski Technical University of Denmark Bogdan Wiszniewski Gda´ nsk University of Technology, Poland Ramin Yahyapour University of Dortmund, Germany Jianping Zhu University of Texas at Arlington, USA
Table of Contents – Part I
Parallel/Distributed Architectures and Mobile Computing R R Evaluating Performance of New Quad-Core IntelXeon 5500 Family Processors for HPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pawel Gepner, David L. Fraser, and Michal F. Kowalik
Interval Wavelength Assignmentin All-Optical Star Networks . . . . . . . . . . Robert Janczewski, Anna Malafiejska, and Michal Malafiejski Graphs Partitioning: An Optimal MIMD Queueless Routing for BPC-Permutations on Hypercubes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jean-Pierre Jung and Ibrahima Sakho Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks . . . . . Marcin Seredynski, Tomasz Ignac, and Pascal Bouvry
1 11
21 31
Numerical Algorithms and Parallel Numerics On the Performance of a New Parallel Algorithm for Large-Scale Simulations of Nonlinear Partial Differential Equations . . . . . . . . . . . . . . . ´ Juan A. Acebr´ on, Angel Rodr´ıguez-Rozas, and Renato Spigler Partial Data Replication as a Strategy for Parallel Computing of the Multilevel Discrete Wavelet Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liesner Acevedo, Victor M. Garcia, Antonio M. Vidal, and Pedro Alonso Dynamic Load Balancing for Adaptive Parallel Flow Problems . . . . . . . . . Stanislaw Gepner, Jerzy Majewski, and Jacek Rokicki A Balancing Domain Decomposition Method for a Discretization of a Plate Problem on Nonmatching Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leszek Marcinkowski Application Specific Processors for the Autoregressive Signal Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anatolij Sergiyenko, Oleg Maslennikow, Piotr Ratuszniak, Natalia Maslennikowa, and Adam Tomas A Parallel Non-square Tiled Algorithm for Solving a Kind of BVP for Second-Order ODEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Stpiczy´ nski
41
51
61
70
80
87
XIV
Table of Contents – Part I
Graph Grammar Based Petri Nets Model of Concurrency for Self-adaptive hp-Finite Element Method with Rectangular Elements . . . . Arkadiusz Szymczak and Maciej Paszy´ nski
95
Numerical Solution of the Time and Rigidity Dependent Three Dimensional Second Order Partial Differential Equation . . . . . . . . . . . . . . . Anna Wawrzynczak and Michael V. Alania
105
Hardware Implementation of the Exponent Based Computational Core for an Exchange-Correlation Potential Matrix Generation . . . . . . . . . . . . . Maciej Wielgosz, Ernest Jamro, and Kazimierz Wiatr
115
Parallel Implementation of Conjugate Gradient Method on Graphics Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Wozniak, Tomasz Olas, and Roman Wyrzykowski
125
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eugeniusz Zieniuk and Agnieszka Boltuc
136
Paralel and Distributed Non-numerical Algorithms Implementing a Parallel Simulated Annealing Algorithm . . . . . . . . . . . . . . Zbigniew J. Czech, Wojciech Mikanik, and Rafal Skinderowicz
146
Parallel Computing Scheme for Graph Grammar-Based Syntactic Pattern Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mariusz Flasi´ nski, Janusz Jurek, and Szymon My´sli´ nski
156
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Gorawski
166
Parallel Longest Increasing Subsequences in Scalable Time and Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Krusche and Alexander Tiskin
176
A Scalable Parallel Union-Find Algorithm for Distributed Memory Computers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fredrik Manne and Md. Mostofa Ali Patwary
186
Tools and Environments for Parallel/Distributed/Grid Computing Extracting Both Affine and Non-linear Synchronization-Free Slices in Program Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wlodzimierz Bielecki and Marek Palkowski
196
Table of Contents – Part I
XV
A Flexible Checkpoint/Restart Model in Distributed Systems . . . . . . . . . . Mohamed-Slim Bouguerra, Thierry Gautier, Denis Trystram, and Jean-Marc Vincent
206
A Formal Approach to Replica Consistency in Directory Service . . . . . . . Jerzy Brzezi´ nski, Cezary Sobaniec, and Dariusz Wawrzyniak
216
Software Security in the Model for Service Oriented Architecture Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grzegorz Kolaczek and Adam Wasilewski
226
Automatic Program Parallelization for Multicore Processors . . . . . . . . . . . Jan Kwiatkowski and Radoslaw Iwaszyn
236
Request Distribution in Hybrid Processing Environments . . . . . . . . . . . . . Jan Kwiatkowski, Mariusz Fras, Marcin Pawlik, and Dariusz Konieczny
246
Vine Toolkit - Grid-Enabled Portal Solution for Community Driven Computing Workflows with Meta-Scheduling Capabilities . . . . . . . . . . . . . Dawid Szejnfeld, Piotr Domagalski, Piotr Dziubecki, Piotr Kopta, Michal Krysinski, Tomasz Kuczynski, Krzysztof Kurowski, Bogdan Ludwiczak, Jaroslaw Nabrzyski, Tomasz Piontek, Dominik Tarnawczyk, Krzysztof Witkowski, and Malgorzata Wolniewicz
256
Applications of Parallel/Distributed Computing GEM – A Platform for Advanced Mathematical Geosimulations . . . . . . . . Radim Blaheta, Ondˇrej Jakl, Roman Kohut, and Jiˇr´ı Star´y Accelerating the MilkyWay@Home Volunteer Computing Project with GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Travis Desell, Anthony Waters, Malik Magdon-Ismail, Boleslaw K. Szymanski, Carlos A. Varela, Matthew Newby, Heidi Newberg, Andreas Przystawik, and David Anderson Vascular Network Modeling - Improved Parallel Implementation on Computing Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Jurczuk, Marek Kr¸etowski, and Johanne B´ezy-Wendling Parallel Adaptive Finite Element Package with Dynamic Load Balancing for 3D Thermo-Mechanical Problems . . . . . . . . . . . . . . . . . . . . . . Tomasz Olas, Robert Le´sniak, Roman Wyrzykowski, and Pawel Gepner Parallel Implementation of Multidimensional Scaling Algorithm Based on Particle Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Piotr Pawliczek and Witold Dzwinel
266
276
289
299
312
XVI
Table of Contents – Part I
Particle Model of Tumor Growth and Its Parallel Implementation . . . . . . Rafal Wcislo and Witold Dzwinel
322
Applied Mathematics and Neural Networks Modular Neuro-Fuzzy Systems Based on Generalized Parametric Triangular Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Korytkowski and Rafal Scherer Application of Stacked Methods to Part-of-Speech Tagging of Polish . . . . Marcin Kuta, Wojciech W´ ojcik, Michal Wrzeszcz, and Jacek Kitowski Computationally Efficient Nonlinear Predictive Control Based on State-Space Neural Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maciej L awry´ nczuk Relational Type-2 Interval Fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . Rafal Scherer and Janusz T. Starczewski Properties of Polynomial Bases Used in a Line-Surface Intersection Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gun Srijuntongsiri and Stephen A. Vavasis
332 340
350 360
369
Minisymposium on GPU Computing A GPU Approach to the Simulation of Spatio–temporal Dynamics in Ultrasonic Resonators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pedro Alonso–Jord´ a, Isabel P´erez–Arjona, and Victor J. S´ anchez–Morcillo Reduction to Condensed Forms for Symmetric Eigenvalue Problems on Multi-core Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paolo Bientinesi, Francisco D. Igual, Daniel Kressner, and Enrique S. Quintana-Ort´ı On Parallelizing the MRRR Algorithm for Data-Parallel Coprocessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christian Lessig and Paolo Bientinesi
379
387
396
Fast In-Place Sorting with CUDA Based on Bitonic Sort . . . . . . . . . . . . . . Hagen Peters, Ole Schulz-Hildebrandt, and Norbert Luttenberger
403
Finite Element Numerical Integration on GPUs . . . . . . . . . . . . . . . . . . . . . . Przemyslaw Plaszewski, Pawel Maciol, and Krzysztof Bana´s
411
Modeling and Optimizing the Power Performance of Large Matrices Multiplication on Multi-core and GPU Platform with CUDA . . . . . . . . . . Da Qi Ren and Reiji Suda
421
Table of Contents – Part I
Stream Processing on GPUs Using Distributed Multimedia Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Repplinger and Philipp Slusallek Simulations of the Electrical Activity in the Heart with Graphic Processing Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernardo M. Rocha, Fernando O. Campos, Gernot Plank, Rodrigo W. dos Santos, Manfred Liebmann, and Gundolf Haase Parallel Minimax Tree Searching on GPU . . . . . . . . . . . . . . . . . . . . . . . . . . . Kamil Rocki and Reiji Suda A Fast GPU Implementation for Solving Sparse Ill-Posed Linear Equation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florian Stock and Andreas Koch
XVII
429
439
449
457
The Second Minisymposium on Cell/B.E. Technologies Monte Carlo Simulations of Spin Glass Systems on the Cell Broadband Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francesco Belletti, Marco Guidetti, Andrea Maiorano, Filippo Mantovani, Sebastiano Fabio Schifano, and Raffaele Tripiccione
467
Montgomery Multiplication on the Cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joppe W. Bos and Marcelo E. Kaihara
477
An Exploration of CUDA and CBEA for Einstein@Home . . . . . . . . . . . . . Jens Breitbart and Gaurav Khanna
486
Introducing the Semi-stencil Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ra´ ul de la Cruz, Mauricio Araya-Polo, and Jos´e Mar´ıa Cela
496
Astronomical Period Searching on the Cell Broadband Engine . . . . . . . . . Maciej Cytowski, Maciej Remiszewski, and Igor Soszy´ nski
507
Finite Element Numerical Integration on PowerXCell Processors . . . . . . . Filip Kru˙zel and Krzysztof Bana´s
517
The Implementation of Regional Atmospheric Model Numerical Algorithms for CBEA-Based Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dmitry Mikushin and Victor Stepanenko
525
Adaptation of Double-Precision Matrix Multiplication to the Cell Broadband Engine Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Rojek and L ukasz Szustak
535
XVIII
Table of Contents – Part I
Optimization of FDTD Computations in a Streaming Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adam Smyk and Marek Tudruj
547
Workshop on Memory Issues on Multi- and Manycore Platforms An Orthogonal Matching Pursuit Algorithm for Image Denoising on the Cell Broadband Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dominik Bartuschat, Markus St¨ urmer, and Harald K¨ ostler
557
A Blocking Strategy on Multicore Architectures for Dynamically Adaptive PDE Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wolfgang Eckhardt and Tobias Weinzierl
567
Affinity-On-Next-Touch: An Extension to the Linux Kernel for NUMA Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stefan Lankes, Boris Bierbaum, and Thomas Bemmerl
576
Multi–CMP Module System Based on a Look-Ahead Configured Global Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eryk Laskowski, L ukasz Ma´sko, and Marek Tudruj
586
Empirical Analysis of Parallelism Overheads on CMPs . . . . . . . . . . . . . . . . Ami Marowka
596
An Implementation of Parallel 3-D FFT with 2-D Decomposition on a Massively Parallel Cluster of Multi-Core Processors . . . . . . . . . . . . . . . . . . Daisuke Takahashi
606
Introducing a Performance Model for Bandwidth-Limited Loop Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Treibig and Georg Hager
615
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
625
Table of Contents – Part II
Workshop on Scheduling for Parallel Computing (SPC 2009) Fully Polynomial Time Approximation Schemes for Scheduling Divisible Loads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joanna Berli´ nska Semi-online Preemptive Scheduling: Study of Special Cases . . . . . . . . . . . . Tom´ aˇs Ebenlendr Fast Multi-objective Reschulding of Grid Jobs by Heuristics and Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wilfried Jakob, Alexander Quinte, Karl-Uwe Stucky, and Wolfgang S¨ uß
1
11
21
Comparison of Program Task Scheduling Algorithms for Dynamic SMP Clusters with Communication on the Fly . . . . . . . . . . . . . . . . . . . . . . . . . . . . L ukasz Ma´sko, Marek Tudruj, Gregory Mounie, and Denis Trystram
31
Study on GEO Metaheuristic for Solving Multiprocessor Scheduling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Piotr Switalski and Franciszek Seredynski
42
Online Scheduling of Parallel Jobs on Hypercubes: Maximizing the Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ondˇrej Zaj´ıˇcek, Jiˇr´ı Sgall, and Tom´ aˇs Ebenlendr
52
The Third Workshop on Language-Based Parallel Programming Models (WLPP 2009) Verification of Causality Requirements in Java Memory Model Is Undecidable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matko Botinˇcan, Paola Glavan, and Davor Runje
62
A Team Object for CoArray Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert W. Numrich
68
On the Definition of Service Abstractions for Parallel Computing . . . . . . Herv´e Paulino
74
XX
Table of Contents – Part II
The Second Workshop on Performance Evaluation of Parallel Applications on Large-Scale Systems Performance Debugging of Parallel Compression on Multicore Machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Borkowski Energy Considerations for Divisible Load Processing . . . . . . . . . . . . . . . . . . Maciej Drozdowski Deskilling HPL: Using an Evolutionary Algorithm to Automate Cluster Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dominic Dunlop, S´ebastien Varrette, and Pascal Bouvry Monitoring of SLA Parameters within VO for the SOA Paradigm . . . . . . Wlodzimierz Funika, Bartosz Kryza, Renata Slota, Jacek Kitowski, Kornel Skalkowski, Jakub Sendor, and Dariusz Krol
82 92
102 115
A Role-Based Approach to Self-healing in Autonomous Monitoring Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wlodzimierz Funika and Piotr P¸egiel
125
Parallel Performance Evaluation of MIC(0) Preconditioning Algorithm for Voxel μFE Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ivan Lirkov, Yavor Vutov, Marcin Paprzycki, and Maria Ganzha
135
Parallel HAVEGE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alin Suciu, Tudor Carean, Andre Seznec, and Kinga Marton
145
The Fourth Grid Applications and Middleware Workshop (GAMW 2009) UNICORE Virtual Organizations System . . . . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Benedyczak, Marcin Lewandowski, Aleksander Nowi´ nski, and Piotr Bala Application of ADMIRE Data Mining and Integration Technologies in Environmental Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marek Ciglan, Ondrej Habala, Viet Tran, Ladislav Hluchy, Martin Kremler, and Martin Gera
155
165
Performance Based Matchmaking on Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrea Clematis, Angelo Corana, Daniele D’Agostino, Antonella Galizia, and Alfonso Quarati
174
Replica Management for National Data Storage . . . . . . . . . . . . . . . . . . . . . . Renata Slota, Darin Nikolow, Marcin Kuta, Mariusz Kapanowski, Kornel Skalkowski, Marek Pogoda, and Jacek Kitowski
184
Table of Contents – Part II
Churn Tolerant Virtual Organization File System for Grids . . . . . . . . . . . . Leif Lindb¨ ack, Vladimir Vlassov, Shahab Mokarizadeh, and Gabriele Violino
XXI
194
The Fourth Workshop on Large Scale Computations on Grids (LaSCoG 2009) Quasi-random Approach in the Grid Application SALUTE . . . . . . . . . . . . Emanouil Atanassov, Aneta Karaivanova, and Todor Gurov
204
Mobile Agents for Management of Native Applications in GRID . . . . . . . Rocco Aversa, Beniamino Di Martino, Renato Donini, and Salvatore Venticinque
214
Leveraging Complex Event Processing for Grid Monitoring . . . . . . . . . . . . Bartosz Balis, Bartosz Kowalewski, and Marian Bubak
224
Designing Execution Control in Programs with Global Application States Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Janusz Borkowski and Marek Tudruj
234
Distributed MIND - A New Processing Model Based on Mobile Interactive Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Magdalena Godlewska and Bogdan Wiszniewski
244
A Framework for Observing Dynamics of Agent-Based Computations . . . Jaroslaw Kawecki and Maciej Smolka HyCube: A DHT Routing System Based on a Hierarchical Hypercube Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Artur Olszak
250
260
Workshop on Parallel Computational Biology (PBC 2009) Accuracy and Performance of Single versus Double Precision Arithmetics for Maximum Likelihood Phylogeny Reconstruction . . . . . . . Simon A. Berger and Alexandros Stamatakis
270
Automated Design of Assemblable, Modular, Synthetic Chromosomes . . . Sarah M. Richardson, Brian S. Olson, Jessica S. Dymond, Randal Burns, Srinivasan Chandrasegaran, Jef D. Boeke, Amarda Shehu, and Joel S. Bader
280
GPU Parallelization of Algebraic Dynamic Programming . . . . . . . . . . . . . . Peter Steffen, Robert Giegerich, and Mathieu Giraud
290
Parallel Extreme Ray and Pathway Computation . . . . . . . . . . . . . . . . . . . . Marco Terzer and J¨ org Stelling
300
XXII
Table of Contents – Part II
Minisymposium on Applications of Parallel Computation in Industry and Engineering Parallelized Transient Elastic Wave Propagation in Orthotropic Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Arbenz, J¨ urg Bryner, and Christine Tobler
310
Parallel Numerical Solver for Modelling of Electromagnetic Properties of Thin Conductive Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ ˇ ˇ Raimondas Ciegis, Zilvinas Kancleris, and Gediminas Slekas
320
Numerical Health Check of Industrial Simulation Codes from HPC Environments to New Hardware Technologies . . . . . . . . . . . . . . . . . . . . . . . . Christophe Denis
330
Application of Parallel Technologies to Modeling Lithosphere Dynamics and Seismicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Boris Digas, Lidiya Melnikova, and Valerii Rozenberg
340
AMG for Linear Systems in Engine Flow Simulations . . . . . . . . . . . . . . . . . Maximilian Emans Parallel Implementation of a Steady State Thermal and Hydraulic Analysis of Pipe Networks in OpenMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mykhaylo Fedorov High-Performance Ocean Color Monte Carlo Simulation in the Geo-info Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tamito Kajiyama, Davide D’Alimonte, Jos´e C. Cunha, and Giuseppe Zibordi EULAG Model for Multiscale Flows – Towards the Petascale Generation of Mesoscale Numerical Weather Prediction . . . . . . . . . . . . . . . Zbigniew P. Piotrowski, Marcin J. Kurowski, Bogdan Rosa, and Michal Z. Ziemianski
350
360
370
380
Parallel Implementation of Particle Tracking and Collision in a Turbulent Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bogdan Rosa and Lian-Ping Wang
388
A Distributed Multilevel Ant-Colony Approach for Finite Element Mesh Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ˇ Katerina Taˇskova, Peter Koroˇsec, and Jurij Silc
398
Minisymposium on Interval Analysis Toward Definition of Systematic Criteria for the Comparison of Verified Solvers for Initial Value Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ekaterina Auer and Andreas Rauh
408
Table of Contents – Part II
Fuzzy Solution of Interval Nonlinear Equations . . . . . . . . . . . . . . . . . . . . . . Ludmila Dymova
XXIII
418
Solving Systems of Interval Linear Equations with Use of Modified Interval Division Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ludmila Dymova, Mariusz Pilarek, and Roman Wyrzykowski
427
Remarks on Algorithms Implemented in Some C++ Libraries for Floating-Point Conversions and Interval Arithmetic . . . . . . . . . . . . . . . . . . Malgorzata A. Jankowska
436
An Interval Method for Seeking the Nash Equilibria of Non-Cooperative Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bartlomiej Jacek Kubica and Adam Wo´zniak
446
From Gauging Accuracy of Quantity Estimates to Gauging Accuracy and Resolution of Measuring Physical Fields . . . . . . . . . . . . . . . . . . . . . . . . . Vladik Kreinovich and Irina Perfilieva
456
A New Method for Normalization of Interval Weights . . . . . . . . . . . . . . . . . Pavel Sevastjanov, Pavel Bartosiewicz, and Kamil Tkacz
466
A Global Optimization Method for Solving Parametric Linear Systems Whose Input Data Are Rational Functions of Interval Parameters . . . . . . Iwona Skalna
475
Direct Method for Solving Parametric Interval Linear Systems with Non-affine Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iwona Skalna
485
Workshop on Complex Collective Systems Evaluating Lava Flow Hazard at Mount Etna (Italy) by a Cellular Automata Based Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maria Vittoria Avolio, Donato D’Ambrosio, Valeria Lupiano, Rocco Rongo, and William Spataro
495
Application of CoSMoS Parallel Design Patterns to a Pedestrian Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah Clayton, Neil Urquhard, and Jon Kerridge
505
Artificial Intelligence of Virtual People in CA FF Pedestrian Dynamics Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ekaterina Kirik, Tat’yana Yurgel’yan, and Dmitriy Krouglov
513
Towards the Calibration of Pedestrian Stream Models . . . . . . . . . . . . . . . . Wolfram Klein, Gerta K¨ oster, and Andreas Meister
521
XXIV
Table of Contents – Part II
Two Concurrent Algorithms of Discrete Potential Field Construction . . . Konrad Kulakowski and Jaroslaw Was
529
Frustration and Collectivity in Spatial Networks . . . . . . . . . . . . . . . . . . . . . Anna Ma´ nka-Kraso´ n and Krzysztof Kulakowski
539
Weakness Analysis of a Key Stream Generator Based on Cellular Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fr´ed´eric Pinel and Pascal Bouvry
547
Fuzzy Cellular Model for On-line Traffic Simulation . . . . . . . . . . . . . . . . . . Bartlomiej Placzek
553
Modeling Stop-and-Go Waves in Pedestrian Dynamics . . . . . . . . . . . . . . . . Andrea Portz and Armin Seyfried
561
FPGA Realization of a Cellular Automata Based Epidemic Processor . . . Pavlos Progias, Emmanouela Vardaki, and Georgios Ch. Sirakoulis
569
Empirical Results for Pedestrian Dynamics at Bottlenecks . . . . . . . . . . . . . Armin Seyfried and Andreas Schadschneider
575
Properties of Safe Cellular Automata-Based S-Boxes . . . . . . . . . . . . . . . . . . Miroslaw Szaban and Franciszek Seredynski
585
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
593
Evaluating Performance of New Quad-Core R R IntelXeon 5500 Family Processors for HPC Pawel Gepner, David L. Fraser, and Michal F. Kowalik Intel Corporation {pawel.gepner,david.l.fraser,michal.f.kowalik}@intel.com
Abstract. In this paper we take a look at what the new Quad-Core Intel Xeon Processor code name Nehalem brings to high performance computing. We compare Intel Xeon 5400 series based system with a server utilizing his successor the new Intel Xeon X5560. We compare both CPU generations utilizing dual socket platforms using a number of HPC benchmarks. The results clearly prove that the new Intel Xeon processor 5500 family provide significant performance advantage on typical HPC workloads and demonstrate to be a right choice for many of HPC installations. Keywords: HPC, multi-core processors, quad-core processors, parallel processing, benchmarks.
1
Introduction
The new 5500 family Quad-Core Intel Xeon processor is the first Intel server CPU with an integrated memory controller (IMC) as well as the first Xeon processor with a Quick Path Interconnect (QPI) interface. Nehalem based products are different from the previous generation products and it was clearly intended not only to scale across all the different product lines, but to be optimized for all the different product segments and market needs from mobile via desktop to server. This new modular design strategy is called at Intel core and uncore approach. Basically Nehalem products are defined in two vectors: one specifies the explicit core functionality universal for all the Nehalem family members. The second is focused on uncore elements and is highly optimized to best fit the specific market needs and segment specifics: this includes number of cores, number of QPI links or memory interfaces and many others uncore components which might be completely different form one family member to another. Core and unncore components can be designed and validated separately. The Quad-Core Intel Xeon processor 5500 family redefines not only every feature of the microprocessor micro-architecture but requires fundamental modification in the system architecture. Systems based on Xeon 5500 family introduce a completely new model of memory hierarchy as well interconnect for connecting processors and other components. In addition to all of the micro-architecture and system innovations Intel Xeon 5500 processors have been designed with all the R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 1–10, 2010. c Springer-Verlag Berlin Heidelberg 2010
2
P. Gepner, D.L. Fraser, and M.F. Kowalik
advantages of 45-nanometer (nm) Hi-k metal gate silicon technology. This process technology uses a new combination of Hi-k gate dielectrics and conductive materials to improve transistor properties such as reducing electrical leakage, chip size, power consumption, and manufacturing costs[1].
2
Processor Microarchitecture and System Changes
The microarchitecture (or computer organization) is mainly a lower level structure that governs a large number of details, hidden in the programming model. It describes the constituent parts of the processor and how these interconnect and interoperate to implement the architectural specification. This means that the new microarchitecture enhances the end-user benefits when compared to the previous generations, achieving a new level of performance and changes the Instruction Set Architecture in silicon, including cache memory design, execution units, and pipelining[2]. Modifications made inside the Nehalem core increase compute resources and deliver higher performance, enhanced parallelism, an improved macrofiusion mechanism, modified Loop Stream Detector (LSD), a new branch prediction mechanism, add a new set of SSE 4.2 instructions and we bring in a new cache hierarchy. In addition to all of the above we re-introduce the Simultaneous MultiThreading (SMT) techniques pioneered on previous generations of the NetBurst R microarchitecture based Xeonprocessor. Increasing parallelism at a core level and extending the microperation concurrency requires a change in size and structure of the out of order window. Nehalem’s out of order buffer (ROB) grew from 96 entries in previous generation of the Xeon family to 128 entries on Nehalem. The Reservation Station, which is used to dispatch operations for execution units, grew from 32 to 36 entries. To track all allocations of load and store, the load buffer extended in size from 32 to 48 entries and the store buffer from 20 to 32. A new macrofusion mechanism identifies more macrofusion opportunities to further increase performance and power efficiency. Nehalem decodes CMP or TEST and conditional branch Jcc as a single microperation (CMP+Jcc), the same as it preprocessor did plus it extends the case for the following branch conditions: —JL/JNGE —JGE/JNL —JLE/JNG —JG/JNLE This increases the decode bandwidth and reduces the microperation count and effectively makes the machine wider. Intel Xeon 5400 family only support macrofusion in 32-bit mode, the Intel Xeon 5500 family supports macrofusion in both 32-bit and 64-bit modes. The Loop Stream Detector identifies software loops and takes advantage of this behavior by removing instruction fetch limitations and disabling unneeded blocks of logic. During a loop the processor decodes the same instructions over
R R Evaluating Performance of New Quad-Core IntelXeon 5500
3
and over and makes the same branch predictions over and over, these units can be disabled to increase higher performance and saving power. The Loop R R Stream Detector was first introduced in the IntelXeon 5300 family and it also operates as a cache for the instruction fetch unit. The Nehalem architecture improves on this concept by moving the Loop Stream Detector closer to the decode stage and keeps macroperations instead of instructions, theoretically this R family. is a very similar approach to the trace cache we used on the Pentium4 Nehalem has two levels of branch target buffer (BTB) these are designed to increase performance and power efficiency. The first and second level BTBs actually use different prediction algorithms and also different history files. These new and improved branch predictors are optimized not only for the best possible branch prediction accuracy but also to work with SMT. Simultaneous Multi-Threading returns to Intel Xeon family after first being implemented in the NetBurst microarchitecture based Xeon. Because an SMTenabled CPU requires substantial memory bandwidth it is likely that SMT will bring even more end-user benefit than Hyper- Threading did. However even for some workloads, including HPC applications SMT may need to be disabled to get the best possible performance[3]. Nehalem supports the same SSE 4.2 instructions as the Xeon 5400 family but in addition to the 47 instructions previously supported, 7 new instructions have been added. The SSE 4.2 instruction set includes instructions for the Vectorizing Compiler and Media Accelerator. The new instructions should improve the performance of audio, video, image editing applications, video encoders, 3-D applications, and games. The last major change to the core microarchitecture for Nehalem is its new cache hierarchy. Nehalem’s cache hierarchy has been extended to three levels, L1 and L2 staying fairly small and private to each core, while the L3 cache is much larger and shared between all cores. Each core in Nehalem has a private 32KB first level L1 Instruction and 32kB Data Cache. In addition the unified 256KB L2 cache, is 8 way associative and provides extremely fast access to data and instructions, the latency is typically smaller than 11 clock cycles. The first members of the Intel Xeon 5500 family have 8MB L3 cache, however additional products may be introduced with different cache sizes depending on the market segment requirements and platform specifics. The L3 Cache is organised as a 16 way set associative, inclusive and shared cache. The cache size can also be different, as a result of the uncore design philosophy. The L3 cache clock is decoupled from the cores frequency, so theoretically the clock of the cache can be different from the frequency of the cores. Nehalem’s L3 load to use latency is less than 35 cycles and is implement as an inclusive cache model, this approach guaranties coherency in the most efficient way. Any cache access misses in the L3 has a guarantee that data is not present it in any of the L2 or L1 caches of the cores. The core valid bits mechanism limits unnecessary snoops of the cores during hit data and only checks the indentified core where it has a possibility to find a private copy of this cache line which might be modified in L1 or L2.
4
P. Gepner, D.L. Fraser, and M.F. Kowalik
This method allows the L3 cache to protect each of the cores from expensive coherency traffic. The Nehalem microarchitecture changes all these pieces and also redefines system architecture. The most ground-breaking changes are the integrated memory controller and Quick Path Interconnect. The memory controller is optimized per R R 5500 market segment, the same analogy as the L3 cache. The initial IntelXeon family products support 3 memory channels per socket of DDR3-1333Mhz both registered and un-registered DIMMs. Each channel of memory can operate independently and handles request in an out-of-order fashion. The local memory latency for Nehalem is 60 ns. The memory controller has been designed for low latency and optimized for both local and remote memory latency. Nehalem delivers huge reduction in local memory latency comparing to previous generation Front Side Bus based Xeon processors, even remote memory latency is a fast 93 ns. Of course effective memory latency is dependent on the application and Operating System implementation of the Non Uniform Memory Architecture. Linux is a NUMA ready operating system, so too is Windows Vista. The biggest improvement comparing to previous generation based system is memory bandwidth. Total peak bandwidth for a single socket is 32GB/s and typical HPC two socket configuration gives 64GB/s. The QuickPath Interconnect (QPI) is a packet-based, high bandwidth, low latency point to point interconnect that operates at up to 6.4GT/s. This new point-to-point interconnect provides socket to socket connections and socket to chipset connections thus enabling system architects to build scalable solutions. Each link is a 20 bit wide interface using differential signaling. The QPI packet is 80 bits long but, only 64 bits are dedicated for data, the rest is used for flow control. This means that each link provides a transfer rate of 12.8GB/s in each direction and 25.6GB/s in total as links are bi-directional. The QPI links are uncore components and depending on the CPU implementation might have a different number of links. The first occurrence of Nehalem dedicated for DP servers (Intel Xeon 5500 family) has two links but the desktop version e.g. (Intel Core i7) has only one. All the above microarchitecture innovations as well as system enhancements increase the system performance and should benefit high performance computing. The exact detail of those innovations is described further in this paper. During all tests we have used single system performance on standard HPC workloads. For all of the tests where we have been comparing two systems: Intel Xeon processor 5400 (Harpertown) family based platform versus Intel Xeon processor 5500 family (Nehalem-EP) family based system. Intel Xeon processor 5400 family based platform: Platform configured with two Quad-Core Intel Xeon processors E5472 3.00GHz, 2x6MB L2 cache, 1600MHz FSB, 16GB memory (8x2GB FBDIMM 800MHz), RedHat Enterprise Linux Server 5 on x86 64. Intel Xeon processor 5500 family based platform: Pre-production platform Green City (Tylersburg B0 stepping) with two Quad-Core Intel Xeon processors X5560 (Nehalem-EP) 2.80GHz with 8M L3 cache, QPI -6.4 GT/s, 24
R R Evaluating Performance of New Quad-Core IntelXeon 5500
5
GB memory (6x4GB DDR3, 1066 MHz), RedHat Enterprise Linux Server 5 on x86 64. Those configurations are the typical ones suggested by vendors as typical HPC high end platform. Difference of memory size is mainly driven by dissimilar memory type (FBDIMM vs. DDR3) and different memory architecture subsystem (FSB vs. Integrated Memory Controller).
3
Processor Performance
The main focus of this section is to present a comparison of two generations R R processors. A popular benchmark well-suited of the Quad-Core IntelXeon for parallel, core-limited workloads is the Linpack HPL benchmark. Linpack is a floating-point benchmark that solves a dense system of linear equations in parallel. The metric produced is Giga-FLOPS or billions of floating point operations per second. Linpack performs operations called LU Factorization. These are highly parallel and store most of their working data set on processor cache. It makes relatively few references to memory for the amount of computation it performs. The processor operations it does perform are predominantly 64-bit floating-point vector operations and these use SSE instructions [4]. This benchmark is used to determine the world’s fastest computers published at the website http://www.top500.org/.
Fig. 1. LINPACK: Dense Floating-Point Operations
In both cases each processor core has 3 functional units that are capable of generating 128-bit results per clock. In this case we may assume that a single processor core does two 64 bit floating-point ADD instructions and two 64 bit floating-point MUL instructions per clock. The theoretical performance, calculated is the product of MUL and ADD executed in each clock, multiplied by frequency giving the following results. For Quad-Core Intel Xeon processor E5472 this gives 3GHz x 4 operations per clock x 4 cores = 48 GFLOPS. For
6
P. Gepner, D.L. Fraser, and M.F. Kowalik
Intel Xeon X5560 theoretical performance we have = 2.8GHz x 4 operations per clock x 4 cores = 44.8 GFLOPS. This is theoretical performance only, and does not fully reflect the real life scenario of running Linpack, in fact this level of theoretical performance disadvantage has also been observed in the Linpack benchmarking scenario. All the innovations of Nehalem do not compensate for the slower clock it has at these initial launch frequencies when purely considering theoretical performance or CPU bounded workloads. The Intel Xeon processor E5472 benefits from a higher CPU clock on the Linpack application. Using LINPACK HPL we see 6% performance disadvantage between the sysR R tem based on the Quad-Core IntelXeon processor E5472 and new Quad-Core Intel Xeon processor X5560 as Fig. 1 shows. It indicates that as we have expected based on the theoretical performance the Intel Xeon 5500 family will not realize a performance improvement on tasks that are heavily dependent on CPU clock frequency.
4
Memory Performance
Theory describes memory performance as a combination of the two following elements, latency and throughput. These components can have an impact on the different workloads, and they are applicable to different usage models. Latency explains how long it takes to chase a chain of pointers through memory. Only a single chain is tracked at a time. Each chain stores only one pointer in a cache line and each cache line is randomly selected from a pool of memory. The pool of memory simulates the working environment of an application. When the memory pool is small enough to be placed inside cache, the benchmark measures the latency required to fetch data from cache. By changing the size of the memory pool we can measure the latency of any specific level of cache, or to main memory, by simply creating the pool bigger than all levels of cache [5]. We measured latency using a 3.0 GHz Quad-Core Intel Xeon processor E5472 and a 2.8 GHz Quad-Core Intel Xeon processor X5560. The results of this experiment are shown in Figure 2. The Intel Xeon processor X5560 with its new cache structure and integrated memory controller significantly improves latency and data movement. The L1 cache stays on the same level even taking slightly longer to fetch data because of the lower clock frequency of the Intel Xeon X5560 vs. Intel Xeon E5472. Also the new hierarchy of cache with a smaller dedicated L2 256KB and shared L3 8MB makes the picture less clear. Based on the preliminary results we have achieved it seems that LMBENCH does not count L3 at all or hides it in the memory latency. Such a huge reduction in memory latency of 48% has been achieved simply by the local memory allocation as the LMBENCH does not measure the NUMA aspects of the processor. The remote memory allocation latency will not be so greatly improved in fact it only increases by 5% versus Xeon E5472. The improvement made on latency reduction for local memory access significantly helps random memory access where the NUMA element is partly considered and overall reduction is 22%.
R R Evaluating Performance of New Quad-Core IntelXeon 5500
7
Fig. 2. Memory Latency
Fig. 3. Stream benchmark - memory throughput improvement
The second component of the memory performance is the throughput for sequential memory accesses. The benchmark we have used to measure the throughput is the Stream benchmark. The Stream benchmark is a synthetic benchmark program, written in standard Fortran 77 developed by John McCalpin. It estimates, both memory reads and memory writes (in contrast to the standard usage for bcopy). It measures the performance of four long vector operations. These operations are: COPY: a(i) = b(i) SCALE: a(i) = q*b(i) SUM: a(i) = b(i) + c(i) TRIAD: a(i) = b(i) + q*c(i)
8
P. Gepner, D.L. Fraser, and M.F. Kowalik
These operations are representative of long vector operations and the array sizes are defined in that way so that each array is larger than the cache of the processors that are going to be tested. This gives us an indication of how effective the memory subsystem is excluding caches. As Fig. 3 shows we see significant R R processor memory bandwidth improvement with the Quad-Core IntelXeon X5560 and this is mainly due to the Integrated Memory controller, different memory type and memory arrangement. This enormous throughput improvement versus the previous generation Quad Core Intel Xeon based system will be reflected in all memory intensive applications not only Stream but also other HPC workloads.
5
Application Performance
Linpack and Stream, which are synthetic benchmarks, used during memory performance tests, measure the performance of specific subsystems and do not explain the full spectrum of system capability. Typical HPC applications use by nature much more than a single subsystem and their behavior is much more sophisticated. So to get a better understanding of how the Quad-Core R R processor X5560 based platform benefits real HPC applications IntelXeon we have selected several real examples. These application and benchmarks represent a broad spectrum of HPC workloads and seem to be a typical representation of a testing suite for this class of calculation. Similar benchmarks were used in publication [1] therefore benchmark descriptions are similar. Abaqus from Simula is a commercial software package for finite element analysis. This is general-purpose solver using a traditional implicit integration scheme to solve finite element analyses. The product is popular with academic and re search institutions due to the wide material modeling capability, and the ability to be easy customized. Amber is a package of molecular simulation programs. The workload measures the number of problems solved per day (PS) using eight standard molecular dynamic simulations. See http://amber.ch.ic.ac.uk/amber8.bench1.html for more information. Fluent is a commercial engineering application used to model computational fluid dynamics. The benchmark consists of 9 standard workloads organized into small, medium and large models. These comparisons use all but the largest of the models which do not fit into the 8GB of memory available on the platforms. The Rating, the default Fluent metric, was used in calculating the ratio of the platforms by taking a geometric mean of the 8 workload ratings measured. GAMESS from Iowa State University, Inc - a quantum chemistry program widely used to calculate energies, geometries, frequencies and properties of molecular systems http://www.msg.ameslab.gov/GAMESS/GAMESS.html. Gaussian from Gaussian, Inc - a quantum chemistry program widely used to calculate energies, geometries, frequencies and properties of molecular systems http://www.gaussian.com.
R R Evaluating Performance of New Quad-Core IntelXeon 5500
9
GROMACS (Groningen Machine for Chemical Simulations) from Groningen University is molecular dynamics program used to calculate energies, geometries, frequencies and properties of molecular systems http://www.gromacs.org. LS-DYNA is a commercial engineering application used in finite element analysis such as a car collision. The workload used in these comparisons is called 3 Vehicle Collision and is publicly available from http://www.topcrunch.org/. The metric for the benchmark is elapsed time in seconds. Monte Carlo from Oxford Center for Computational Finance (OCCF) - financial simulation engine using Monte Carlo technique http://www.occf.ox.ac.uk/. PamCrash from ESI Group - an explicit finite-element program well suited for crash simulations. For more information go to http://www.esi-group.com/ /SimulationSoftware/NumericalSimulation/index.html. Star-CD is a suite of test cases, selected to demonstrate the versatility and robustness of STAR-CD in computational fluid dynamic solutions. The metric produced is elapsed seconds converted to jobs per day. For more information go to http://www.cd-adapco.com/products/STAR-CD.
Fig. 4. Platform comparison across HPC selected workloads
All these selected workloads have been tested on two dual socket HPC optiR R mized platforms. The Quad-Core IntelXeon processor E5472 based platform has been used as the baseline to illustrate the improvement that the new platform is going to bring in different HPC workloads. As we can see the Quad-Core Intel Xeon processor X5560 based platform shows significant performance improvement up to 133%. The new miroarchitecture helps especially in the data intensive applications and workloads where data movement plays an important role. If the task is more CPU intensive the difference is around 5-15% for most of these workloads. For reference we have also placed the best published results of the AMD two sockets platform utilizing two Quad-Core AMD Opteron 2354 (Barcelona 2.2GHz) processors with 16 GB RAM (8x2 GB, 667MHz DDR2). When looking at a broad range of highly applicable workloads the Quad-Core Intel Xeon processor E5560 based platform demonstrates clear leadership.
10
6
P. Gepner, D.L. Fraser, and M.F. Kowalik
Conclusion
R R The new Quad-Core IntelXeon X5560 processor brings a new level of performance to HPC, improving significantly on applications where its preprocessors were not the most powerful. The new microarchitecture enhancements and system level modifications benefit data intensive types of HPC applications by up to as much as 130%. Processor intensive tasks will not achieve the same performance improvement as memory banded workloads, but they continue to stay on the same high level. Both generations of tested CPUs deliver the same theoretical performance when running the same clock and keep within the same thermal envelope but as shown deliver an improvement in performance of HPC workloads. Also the platform level modifications like QPI make the new platform more balanced from an I/O perspective and open new possibilities for more superior interconnect solutions. The Quad-Core Intel Xeon 5500 series is the first instance of the Nehalem generation and new additions to this family will bring even more functionality, scalability and performance and will become a compelling choice for many of the new HPC installations.
References 1. Gepner, P., Fraser, D.L., Kowalik, M.F.: Multi-Core Processors: Second generation Quad-Core Intel Xeon processors bring 45nm technology and a new level of performance to HPC applications. In: ICCS 2008, pp. 417–426 (2008) 2. Smith, J.E., Sohi, G.S.: The Microarchitecture of superscalar processors. Proc. IEEE 83, 1609–1624 (1995) 3. Eggers, S.J., Emer, J.S., Levy, H.M., Lo, J.L., Stamm, R.L., Tullsen, D.M.: Simultaneous multithreading: A platform for next generation processors. Proc. IEEE 17, 12–19 (1997) 4. Dongarra, J., Luszczek, P., Petitet, A.: Linpack Benchmark: Past, Present, and Future, http://www.cs.utk.edu/luszczek/pubs/hplpaper.pdf 5. Gepner, P., Kowalik, M.F.: Multi-Core Processors: New Way to Achieve High System Performance. PARELEC 2006, pp. 9–13 (2006)
Interval Wavelength Assignment in All-Optical Star Networks Robert Janczewski, Anna Małafiejska, and Michał Małafiejski Gdańsk University of Technology, Poland Algorithms and System Modeling Department {robert,aniam,mima}@kaims.pl
Abstract. In the paper we consider a new problem of wavelength assigment for multicasts in the optical star networks. We are given a star network in which nodes from a set V are connected to the central node with optical fibres. The central node redirects the incoming signal from a single node on a particular wavelength from a given set of wavelengths to some of the other nodes. The aim is to minimize the total number of used wavelengths, which means that the overall cost of the transmission is minimized (i.e. wavelength conversion or optoelectronic conversion is minimized). This problem can be modelled by a p-fiber coloring of some labelled digraph, where colors assigned to arcs of the digraph correspond to the wavelengths. In the paper we assume the set of all wavelengths of the incoming signals to a particular node forms an interval, i.e. a consecutive set of numbers. We analysed the problem of one-multicast transmission (per node). We constructed polynomial time algorithms for some special classes of graphs: complete k-partite graphs, trees and subcubic bipartite graphs. Keywords: WDM optical networks, multicasting, interval coloring, star arboricity.
1
Introduction
The considerations in this paper are motivated by the multicasting communication in a multifiber WDM (wavelength-division multiplexing) all-optical networks, which was recently considered in papers [3,6,7,22]. By the multifiber WDM network we mean the network where nodes are connected with parallel fibers. The problem is formulated as follows: we are given an all-optical star network with a set of nodes V , which are connected to the central node by optical fibers. Each node v ∈ V sends a set of at most q multicasts to some other nodes S1 (v), . . . , Sq (v), where Si (v) ⊂ V . The transmission through the central node is using WDM, i.e. different signals may be sent at the same time through the same fiber but on different wavelengths. First step of every multicast transmission from a node v is sending a message to the central node on the particular wavelength from given set of wavelengths. In the next step, the central node redirects the message to each node of Si (v) using one of these wavelengths. The R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 11–20, 2010. c Springer-Verlag Berlin Heidelberg 2010
12
R. Janczewski, A. Małafiejska, and M. Małafiejski
goal is to minimize the total number of wavelengths during the simultaneous transmission of all multicasts in the network. This problem can be modelled by arc coloring of labelled (multi)digraph with some special requirements on the colors [3,7]. If we assume there is only one fiber between the central node and each node from V , and there is only one multicast per node (we define such a network as one-multicast one-fiber network) the problem reduces to the (directed) star arboricity (coloring) problem [1,2,15] and incidence coloring problem [8]. In this paper we assume the set of wavelengths of incoming signals forms an interval, i.e. a consecutive set of numbers, which may mean the multiplies of some base wavelength. Model and problem definition. In the general case, every node from set V of n nodes is connected to the central node with p optical fibres, such a network is sometimes called the star network. The multicast transmission in this network can be modelled by a (multi)digraph D with vertex set V and with labelled arc sets leaving v corresponding to multicasts S1 (v), . . . , Sq (v), i.e. every set Ai (v) = {vw : w ∈ Si (v)} of arcs leaving v is labelled with i (Fig. 1).
Fig. 1. S1 (v) = {a, b}, S2 (v) = {a, c}
We define a p-fiber coloring as a function assigning positive integers (colors, wavelengths) to the arcs such that at each vertex v and for every color c, inarc(v, c) + outlab(v, c) ≤ p, where inarc(v, c) is the number of arcs entering v and colored with c, and outlab(v, c) = |{i : ∃e ∈ Ai (v) and e is colored with c}|. In other words, outlab(v, c) is the number of labels of arcs leaving v and colored with c. For a given p-fiber coloring of digraph D we can easily construct a proper assignment of fibers in a network corresponding to D as follows: we assign a different fiber corresponding to every arc entering v and colored with c, and to each set of arcs leaving v, colored with c and of the same label. The correspondence between colors and wavelengths is shown in Fig. 1. Hence, the problem of simultaneous transmission of all multicasts in the p-fiber network with minimum number of wavelengths is equivalent to the p-fiber coloring of arcs of the digraph D with a minimum number of colors. Now, we formulate a decision version of the problem of p-fiber wavelength assigment for q-multicasts in optical star networks:
Interval Wavelength Assignment in All-Optical Star Networks
13
The (p, q)-WAM problem Instance: A digraph D with at most q labels on arcs, a positive integer k. Question: Is there a p-fiber coloring of D with at most k colors? A comprehensive survey of the (p, q)-WAM problem one can find in [3]. The problem (p, 1)-WAM was considered in [16] (i.e. for multitrees). In the paper we consider the (1, 1)-WAM problem, i.e. the fundamental case, where the fiber is unique and each node sends only one multicast. In this case, the p-fiber coloring of multidigraph reduces to the coloring of arcs of digraph satysfying only two conditions: (c1) any two arcs entering the same vertex have different colors, (c2) any arc entering and any arc leaving the same vertex have different colors. This coloring is well-studied [3,7] and arises from the problem of partitioning of a set of arcs into the smallest number of forests of directed stars, this problem is called the directed star arboricity problem [1,2,15]. Moreover, we focus our attention to the symmetrical communication, i.e. every transmission from v to w implies the communication between w and v, in this case the digraph modeling the communication between nodes is a simple graph. This problem can be described as (1, 1)-WAM problem with symmetric multicasts and can be reduced to the incidence coloring of graphs [8,10,11,20,23] (for details see Section 2). Previous results and our contribution. In this paper we study the WAM problem under the assumption that the set of colors assinged to arcs entering a vertex forms an interval, i.e. a consecutive set of numbers. This corresponds to having a consecutive wavelengths on the link between the central node and the destination node. This problem is important for traffic grooming in WDM networks, where wavelengths could be groomed into wavebands [22]. The interval (p, q)-WAM problem is a new concept arising from well-studied model of interval-edge coloring [4,12,13,14]. In [17] the authors introduced the problem of interval incidence coloring modeling message passing in networks. The authors proposed some bounds on the interval incidence coloring number (IIC) and studied this problem for some classes of graphs (paths, cycles, stars, fans, wheels and complete graphs). In this paper we construct polynomial time algorithms for trees, subcubic bipartite graphs, i.e. bipartite graphs with the degree of vertex at most three, and complete k-partite graphs. Outline of the paper. In Section 2 we introduce definitions, prove some useful bounds for the interval incidence chromatic number, and construct polynomial time algorithm for complete k-partite graphs. In Section 3 we construct polynomial time algorithms for biparite graphs with maximum degree at most 3, i.e.
14
R. Janczewski, A. Małafiejska, and M. Małafiejski
subcubic bipartite graphs. In Section 4 we construct polynomial time algorithm for trees using dynamic methods.
2
Definitions and Simple Properties
In the following we consider connected simple graphs only, and use standard notations in graph theory. Incidence coloring. For a given simple graph G = (V, E), we define an incidence as a pair (v, e), where vertex v ∈ V (G) is one of the ends of edge e ∈ E(G) (we say v is incident with e). Let us define a set of incidences I(G) = {(v, e) : v ∈ V (G) ∧ e ∈ E(G) ∧ v ∈ e}. We say that two incidences (v, e) and (w, f ) are adjacent if one of the following holds: (i) v = w, e = f, (ii) e = f , v = w, (iii) e = {v, w}, f = {w, u} and v = u. By an incidence coloring of G we mean a function c : I(G) → N such that c((v, e)) = c((w, f )) for any adjacent incidences (v, e) and (w, f ). In the following we use simplified notation c(v, e) instead of formal c((v, e)). Interval incidence coloring. A finite subset A of N is an interval if and only if it contains all numbers between min A and max A. For a given incidence coloring c of graph G let Ac (v) = {c(v, e) : v ∈ e ∧ e ∈ E(G)}. By an interval incidence coloring of graph G we mean an incidence coloring c of G such that for each vertex v ∈ V (G) set Ac (v) is an interval. The interval incidence coloring number of G, denoted by IIC(G), is the smallest number of colors in an interval incidence coloring. The interval incidence coloring of G using IIC(G) colors is said to be minimal. Symmetric (1, 1)-WAM problem. For a given symmetric digraph D consider (1, 1)-WAM problem and observe that we can ommit the unique label on arcs. Every two symmetric arcs uv and vu between vertices u and v we can replace by one edge e = {u, v} and define two incidences (v, e) and (u, e), which corresponds to arcs uv and vu, respectively. Hence, the coloring of arcs of digraphs satisfying conditions (c1) and (c2) is equivalent to the incidence coloring of graphs. 2.1
Bounds for the Interval Incidence Coloring Number
The IIC number can be bounded by maximum degree Δ(G) and its chromatic number χ(G) as follows. Theorem 1. For a given non-empty graph G we have Δ(G) + 1 ≤ IIC(G) ≤ χ(G) · Δ(G).
Interval Wavelength Assignment in All-Optical Star Networks
15
Proof. The lefthand inequality holds because for any interval incidence coloring c, each vertex v ∈ V (G) and its neighbour u we have c(u, {u, v}) = c(v, {u, v}). To prove the righthand inequality, we divide vertex set into χ(G) indepedent sets denoted by I1 , I2 , . . . , Iχ(G) . For any v ∈ Ii we can assign {c(v, e) : (v, e) ∈ I(G)} ⊆ {1 + (i − 1) · Δ, . . . , i · Δ} in such a way that Ac (v) is an interval. It is easy to see that any two adjacent incidences (v, {v, u}) and (u, e) will receive different colors. Theorem 2. For every k ≥ 2 and for every complete k-partite graph G = Kp1 ,p2 ,...,pk holds the following lower bound IIC(G) ≥ 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j}. Proof. Let c be an interval incidence coloring of G that uses IIC(G) colors. Let v be a vertex of G such that min Ac (v) = min c(I(G)). Let u be a neighbour of v such that c(v, {u, v}) = max Ac (v). Then Ac (u) ∩ Ac (v) = ∅ and IIC(G) ≥ |Ac (u)| + |Ac (v)| = deg(u) + deg(v) ≥ 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j}. 2.2
IIC Number for Some Class of Graphs
In the Table 1 we give exact values of the interval incidence coloring number for some known classes of graphs. Table 1. The value of IIC for some classes of graphs Graph’s family paths Pn , n ≥ 5 cycles Cn stars Sn 2-stars Sn2 k-stars Snk , k ≥ 3 wheels and fans complete graphs complete bipartite Kp,q complete k-partite Kp1 ,p2 ,...,pk 2n − max{pq bipartite torus regular bipartite
2.3
IIC(G)
Ref.
Δ+2 Δ+2 Δ+1 Δ+1 2Δ Δ+3 2Δ p+q + pr : 1 ≤ q < r ≤ k} 2Δ 2Δ
[17] [17] [17] [17] [21] [17] [17] [21] [21] [18] [18]
IIC Number for Complete k-Partite Graphs
Theorem 3. A complete k-partite graph Kp1 ,p2 ,...,pk can be colored optimally with 2n − max{pi + pj : i, j = 1, . . . , k ∧ i = j} colors in O(m + n) time. Proof. By Theorem 2, it suffices to construct interval incidence coloring of G = = j} colors. Kp1 ,p2 ,...,pk that uses exactly 2n − max{pi + pj : i, j = 1, . . . , k ∧ i
16
R. Janczewski, A. Małafiejska, and M. Małafiejski
Let I1 , I2 , . . . , Ik be partitions of G. Without loss of generality we can assume that pi = |Ii | (1 ≤ i ≤ k), p1 ≤ p2 ≤ . . . ≤ pk and Ii = {v1i , v2i , . . . , vpi i }. We construct the desired interval incidence coloring of G as follows: p1 + p2 + . . . + pk−1 + l k > i, i i k c(vj , {vj , vl }) = n + p1 + p2 + . . . + pk−1 + l otherwise. We leave it to the reader to verify that the above formula defines interval inci= j} colors. dence coloring that uses 2n − max{pi + pj : i, j = 1, . . . , k ∧ i
3
Polynomial Time Algorithms for Subcubic Bipartite Graphs
Cycles and paths were considered in Section 2.2, and there are only a few connected graphs with IIC ≤ 3. Let G be a subcubic bipartite graph G with Δ(G) = 3. By Theorem 1 the IIC(G) is between 4 and 6. In this section we will construct different algorithms for interval incidence coloring of subcubic bipartite graphs using 4, 5 and 6 colors. 3.1
Interval Incidence Coloring with 4 Colors
Suppose IIC(G) = 4. We prove that graph G satisfies the following properties: (i) each vertex v of degree 3 has at most one adjacent vertex of degree 3, (ii) each vertex v of degree 3 has at least one adjacent vertex of degree 1, (iii) on every path between two non-adjacent vertices of degree 3 there are at least two vertices of degree 2. The property (i) follows from the fact, that in {1, 2, 3, 4} one can find only two intervals of length 3: {1, 2, 3} and {2, 3, 4}, and in each of them there is only one element not belonging to the second one. To prove the property (ii) observe that if Ac (v) = {1, 2, 3} (Ac (v) = {2, 3, 4}), then the vertex u for which the incidence (v, {v, u}) is colored by 3 (by 2, respectively) must be a leaf. Suppose, contrary to the property (iii), that on the path between 3-degree vertices v1 , v2 there is only one vertex u with degree 2. Assume that c(u, {u, v1 }) = 1, c(u, {u, v2 }) = 2. This cleary implies 2 ∈ / Ac (v2 ) and max Ac (v2 ) ≥ 5, a contradiction. The same reasoning applies to the cases where Ac (u) = {2, 3} or Ac (u) = {3, 4}, so the property (iii) holds. We propose the algorithm using 4 colors for interval incidence coloring of graph G which satisfies properties (i) − (iii). First, color vertices of G using two colors a and b, which represent intervals containing 1 and 4, respectively. Next, for every 3-degree vertex v colored with a (or b, respectively), let • c(v, {v, u}) = 1 (or 4 respectively), if deg(u) = 3, • c(v, {v, u}) = 2 (or 3 respectively), if deg(u) = 2, or if deg(u) = 1 and there is no u ∈ N (v) such that deg(u) = 2,
Interval Wavelength Assignment in All-Optical Star Networks
17
• c(v, {v, u}) = 3 (or 2 respectively), if deg(u) = 1. For every 2-degree vertex v colored by a which represents an interval {1, 2} (or b, respectively, which represents an interval {3, 4}), we have to assign c(v, {v, u}) = 1 (respectively 4) if deg(u) = 3. Finally, we color all pending incidences (i.e. at leafs). By the above, the problem of interval incidence coloring of a graph using 4 colors is equivalent to veryfing properties (i) − (iii), hence we get Theorem 4. The problem of interval incidence coloring with 4 colors can be solved for subcubic bipartite graphs in linear time. 3.2
Interval Incidence Coloring with 5 Colors
For a given subcubic bipartite graph G, if IIC(G) ≤ 5 then the following algorithm for interval incidence coloring problem uses 5 colors, otherwise returns F ALSE: (i) Color vertices of G using a and b. (ii) For every v ∈ V (G) color incidences (v, {v, u}), for all u ∈ N (v) as follows: • if deg(v) = 3 and v is colored with a (b, respectively), then let c(v, {v, u}) ∈ {1, 2, 3} (c(v, {v, u}) ∈ {3, 4, 5}, respectively) in such a way that if deg(u) = 3 then c(v, {v, u}) = 3. If for every u ∈ N (v) deg(u) = 3, then return F ALSE, • if deg(v) = 2 and v is colored by a (b, respectively), then let c(v, {v, u}) ∈ {1, 2} (c(v, {v, u}) ∈ {4, 5}, respectively) in any way, • if deg(v) = 1 and v is colored by a (b respectively), then let c(v, {v, u}) = 1 (c(v, {v, u}) = 5 respectively).
Fig. 2. Interval incidence coloring of subcubic bipartite graphs with 5 colors
Consider any v ∈ V (G), if deg(v) ≤ 2, then because G is bipartite, we have Ac (v) ∩ Ac (u) = ∅, for any u ∈ N (v). The same observation holds, if deg(v) = 3 and there exists u ∈ N (v) such that deg(u) = 3, as showed in Figure 2. If for all u ∈ N (v), deg(u) = 3, then there is no interval incidence coloring of G with 5
18
R. Janczewski, A. Małafiejska, and M. Małafiejski
colors and algorithm returns F ALSE. In the other case we get proper incidence coloring c of G with 5 colors. It is easy to see that c is interval coloring, hence holds the following Theorem 5. The problem of interval incidence coloring with 5 colors can be solved for subcubic bipartite graphs in linear time. 3.3
Interval Incidence Coloring with 6 Colors
Suppose G is a subcubic bipartite graph such that IIC(G) > 5. Then we use the same algorithm as in the proof of Theorem 1 for the upper bound, which is exactly 6 in this case. Hence combining all the algorithms we have Theorem 6. Finding a minimal interval incidence coloring of subcubic bipartite graph can be done in linear time.
4
Polynomial Time Dynamic Algorithm for Trees
In this chapter one can find a polynomial-time algorithm for interval incidence coloring of trees. Given a tree T , we take any leaf r as a root of T and direct tree T from the leaves different from r to the root r. It means that we direct every edge e ∈ E(T ) in such a way that for every v ∈ V (T ), v = r, there is exactly one edge leaving v. We will color the incidences of T using the bottom-up technique in accordance to the defined direction. For every edge e = (v, u) ∈ E(T ) we build a matrix A of size 2Δ × 2Δ in such a way that A(i, j) is the maximum of already used colors of incidences in some minimal coloring of subtree Tv with the root v, assuming that the incidence (v, e) is colored by i and the incidence (u, e) is colored by j. The main idea of the algorithm for finding IIC(T ) is as follows: (i) For each edge e ∈ E(T ), beginning from the bottom, find a matrix A. (ii) Return IIC(T ) = mini,j=1,2,...,2Δ A(i, j), where A is a matrix for r. Constructing consecutive incidence coloring of T with IIC(T ) colors is possible by using additional structures for remembering chosen pair of colors for adjacent incidences while building a matrix A. 4.1
Constructing a Matrix A
We will assume that A(i, i) = ∞ for every i = 1, 2, . . . , 2Δ. Let’s consider an edge e = (v, u) directed from v to u. If v is a leaf, then A(i, j) = max{i, j} for i = j. In the other case, let v1 , v2 , . . . , vp be the neighbours of v different from u, p = deg(v) − 1. According to our assumption, there is a matrix Ai built for every edge (vi , v), i = 1, . . . , p. We use the following algorithm for constructing a matrix A for e = (v, u):
Interval Wavelength Assignment in All-Optical Star Networks
19
Algorithm MATRIX(e = (v, u)) for each pair of colors of incidences (v, e), (u, e), denoted by a, b, a =b for each k = 0, 1, . . . , deg(v) − 1 let mk = ∞ let Ik = {a − deg(v) + 1 + k, . . . , a + k} if b ∈ / Ik then construct bipartite graph G with partitions V1 , V2 and with weights of edges as follows: V1 = {v, v1 , v2 , . . . , vp }, V2 = Ik E(G) = {{vi , j} : i = 1, . . . , p ∧ j ∈ Ik } ∪ {v, a} w : E(G) → N w({vi , j}) = minl∈{1,...,2Δ}\Ik Ai (l, j) w({v, a}) = max{a, b} remember mk = MATCHING(G) end for A(a, b) = mink mk end for The procedure MATCHING is used for finding a perfect matching M of G that minimalizes the maximum of w(e), e ∈ E(M ). Algorithm MATCHING(G) for each i = 1, 2, . . . , 2Δ G = G \ {e ∈ E(G) : w(e) > i} if there is a perfect matching of G then return i end for To speed up the algorithm MATCHING, we can use bisection method instead of the loop for i = 1, 2, . . . , 2Δ. Using the Hopcroft-Karp algorithm for finding perfect matching in a graph G we obtain time complexity of MATCHING es5 timated by O(log Δ · Δ 2 ). The time complexity √ of the MATRIX algoritm for a given edge e ∈ E(T ) is estimated by O(log Δ ΔΔ5 ). Using the MATRIX algorithm O(n) times leads to the resulting algorithm for finding IIC(T ) with time 1 complexity estimated by O(nΔ5 2 log Δ).
5
Final Remarks
Theorem 7. Interval incidence coloring of planar graphs of degree 4 is NPC. The proof of this theorem can be found in [19]. An interesting question is the complexity of the interval incidence coloring of bipartite graphs.
20
R. Janczewski, A. Małafiejska, and M. Małafiejski
References 1. Algor, I., Alon, N.: The star arboricity of graph. Discrete Mathematics 75, 11–22 (1989) 2. Alon, N., McDiarmid, C., Reed, B.: Star arboricity. Combinatorica 12, 375–380 (1992) 3. Amini, O., Havet, F., Huc, F., Thomasse, S.: WDM and directed star arboricity. In: INRIA, France, pp. 1–20 (2007) 4. Asratian, A., Kamalian, R.: Investigation on interval edge-colorings of graphs. J. Combin. Theory. Ser. B 62, 34–43 (1994) 5. Beaquier, B., Bermond, J.C., Gargano, L., Hell, P., Perennes, S., Vacarro, U.: Graph problems arising from wavelength-routing in all-optical networks. In: Proc. of WOCS 1997. IEEE, Los Alamitos (1997) 6. Brandt, R., Gonzalez, T.F.: Multicasting using WDM in Multifiber Optical Star Networks. In: Proc. of the 15th IASTED International Conference on Parallel and Distributed Computing and Systems PDCS 2003, Canada, pp. 56–61 (2003) 7. Brandt, R., Gonzalez, T.F.: Wavelength assignment in multifiber optical star networks under the multicasting communication mode. Journal of Interconnection Networks 6, 383–405 (2005) 8. Brualdi, R.A., Massey, J.Q.: Incidence and strong edge colorings of graphs. Discrete Mathematics 122, 51–58 (1993) 9. Chen, D.L., Liu, X.K., Wang, S.D.: The incidence chromatic number and the incidence coloring conjecture of graphs. Math. Econom. 15(3), 47–51 (1998) 10. Chen, D.L., Pang, S.C., Wang, S.D.: The incidence coloring number of Halin graphs and outerplanar graphs. Discrete Mathematics 256, 397–405 (2002) 11. Dolama, M.H., Sopena, E., Zhu, X.: Incidence coloring of k-degenerated graphs. Discrete Mathematics 283, 121–128 (2004) 12. Giaro, K.: Interval edge-coloring Contemporary Mathematics. In: Kubale, M. (ed.) Graph Colorings, pp. 105–121. AMS (2004) 13. Giaro, K., Kubale, M., Małafiejski, M.: Compact scheduling in open shop with zero-one time operations. INFOR 37, 37–47 (1999) 14. Giaro, K., Kubale, M., Małafiejski, M.: Consecutive colorings of the edges of general graphs. Discrete Mathemathics 236, 131–143 (2001) 15. Guiduli, B.: On incidence coloring and star arboricity of graphs. Discrete Mathematics 163, 275–278 (1997) 16. Janczewski, R., Małafiejski, M.: Incidence coloring of multitrees. Zesz. Nauk. Pol. Gd. 13, 465–472 (2007) (in Polish) 17. Janczewski, R., Małafiejska, A., Małafiejski, M.: Interval incidence coloring of graphs. Zesz. Nauk. Pol. Gd. 13, 481–488 (2007) (in Polish) 18. Janczewski, R., Małafiejska, A., Małafiejski, M.: Interval incidence coloring of graphs. Rap. Tech. WETI 12, 1–16 (2008) (in Polish) 19. Janczewski, R., Małafiejska, A., Małafiejski, M.: The complexity of interval incidence coloring of graphs. Rap. Tech. WETI 16, 1–17 (2008) (in Polish) 20. Li, X., Tu, J.: NP-completeness of 4-incidence colorability of semi-cubic graphs. Discrete Mathematics 308, 1334–1340 (2008) 21. Małafiejska, A.: Interval coloring of graphs MSc Thesis, University of Gdańsk, pp. 1–70 (2008) 22. Modiano, E., Lin, P.J.: Traffic grooming in WDM networks. IEEE Communications Magazine 39(7), 124–129 (2001) 23. Shiu, W.C., Lam, P.C.B., Chen, D.-L.: Note on incidence coloring for some cubic graphs. Discrete Mathematics 252, 259–266 (2002)
Graphs Partitioning: An Optimal MIMD Queueless Routing for BPC-Permutations on Hypercubes Jean-Pierre Jung and Ibrahima Sakho Universit´e de Metz - UFR MIM - Dpt. d’Informatique Ile du Saulcy BP 80794 - 57012 Metz Cedex 01, France {jpjung,sakho}@univ-metz.fr
Abstract. Bit-Permute-Complement permutations (BPC) constitute the subclass of particular permutations which have gained the more attention in the search of optimal routing of permutations on hypercubes. The reason of this attention comes from the fact that they care permutations for general-purpose computing like matrix transposing, vector reversal, bit shuffling and perfect shuffling. In this paper we revisit the optimal routing of BPC problem on hypercubes under MIMD queueless communication model through a new paradigm which takes advantage of their topology: the so-called graphs partitioning. We prove that BPC are partitionable in any dimension of the hypercube and that the resulting permutations are also BPC. It follows that any BPC on n-dimensional hypercube is routable in at most n steps of data exchanges, each one realizing the partition of the hypercube. Keywords: Interconnection network, hypercube, BPC permutations, MIMD queueless routing, perfect matching of bipartite graph, graph partitioning.
1
Introduction
The processors interconnection network (IN) is the heart of the message passing parallel computers. Indeed, the performance of such computers depends greatly on the performance of their IN whose essential criteria are the scalability for massive parallelism, the capability of deadlock-free routing on shortest paths, the capability of simulating others IN and the management facility. In the research for IN which fulfil these criteria, hypercubes constitute a very attractive alternative. The incremental construction of hypercubes confers to them interesting mathematical properties which allow meeting most of performance criteria. On the contrary of most IN, they allow exponential increase of the number of the nodes while the diameter increases linearly. They admit implicit routing functions which alleviate the effort that requires their management in avoiding the computation of huge routing tables and which route on shortest paths. Finally most of the popular IN, rings, 2D wrap-around grids, trees, can be almost isometrically embed in hypercubes so guaranteeing their simulation capability. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 21–30, 2010. c Springer-Verlag Berlin Heidelberg 2010
22
J.-P. Jung and I. Sakho
For all these reasons, several commercial parallel machines have been built over the years and several theoretical and practical research works have also been done on different aspects of their use as IN. Among the theoretical research, one of the most challenging is the optimal rearrangeability of hypercubes that is, their capability for optimal routing any permutation such that the routes of each routing step constitute a one-to-one correspondence of adjacent nodes. BPC constitute the subclass of particular permutations which have gained the more attention in search of optimal permutations routing in hypercubes because they care permutations for general-purpose computing like matrix transposing, vector reversal, bit shuffling and perfect shuffling. In this paper we address the problem of the optimal routing of BPC. We revisit the problem through a new paradigm the so called graphs partitioning to take advantage of the recursive structure of the hypercubes topology. The remainder of the paper is organised in five sections. Section 2 gives the problem formulation and some basic definitions relative to hypercubes, permutations and routing. Section 3 presents the related works. Section 4 introduces the mathematical foundation used to develop the proposed routing algorithm. Section 5 characterizes partitionable BPC and proposes a routing algorithm. Section 6 concludes the paper and presents some perspectives.
2 2.1
Problem Formulation Definitions
Definition 1. An n-dimensional hypercube, nD-hypercube for short, is a graph H (n) = (V, E) where V is a set of 2n nodes u = (un−1 , un−2 , . . . , u0 ) such that ui ∈ {0, 1} and E is the set of arcs (u, v) ∈ V × V such that there is only one dimension i for which ui = vi . It is well known that for n > 0 an nD hypercube is obtained by interconnecting two (n− 1)D-hypercubes in any dimension 0 ≤ i ≤ n− 1. So any hypercube H (n) (n) (n) can be viewed as anyone of the n couples (H0,i , H1,i ) of (n − 1)D-hypercubes obtained by restricting H (n) to its nodes u such that ui = 0 (resp. 1). Figure 1 illustrates such a view.
5
4
6 2
14
7 3
13 9
8
1
0
12
15
10 11
Fig. 1. 4D-hypercube viewed as the interconnection of two 3D-hypercubes in the dimension 3
Optimal Queueless Routing for BPC on Hypercubes
23
Definition 2. A permutation on a nD-hypercube H (n) is an automorphism on V ; that is a one-to-one application π which associates to each node u = (un−1 , un−2 , . . ., u0 ) of H (n) one and only one node π(u) = (πn−1 (u),πn−2 (u),. . ., π0 (u)). It is represented by the sequence π = (π(u); u = 0, 1, . . ., 2n − 1). Definition 3. A permutation π in a nD-hypercube is a BPC if for any i, there is j such that for any u, πi (u) = uj or πi (u) = uj , the complement of uj . 2.2
MIMD Queueless Routing
Let π be a permutation on an nD-hypercube network with bidirectional links. Let’s consider a set of 2n messages of the same size each one located on one node u and destined for the node π(u). Routing the permutation π on the hypercube under MIMD queueless communication model consists in conveying all the messages to their destination such that each node holds one and only one message. Clearly, such a routing consists in a sequence of global and synchronous exchanges of messages between neighbourhood nodes such that no more than one message is located at each node after each exchange step.
3
Related Works
Optimal routing of permutations on nD-hypercubes is one of the most challenging open problems in the theory of IN. For an arbitrary permutation, it is well known, from e-cube routing [1], that such a routing requires at least n exchange steps. To specify its complexity, it has been extensively studied under several communication models and routing paradigms. Szimansky in [2] considers the offline routing in circuit-switched and packet switched commutation models under all-port MIMD communication model. Under the circuit-switched hypothesis he proves that, for n ≤ 3, any hypercube is rearrangeable. He also conjectured that routing can be made on the shortest paths, conjecture for which a counterexample has been given by Lubiw in [3]. Under packet-switched hypothesis he also shows that routing can be made in 2n − 1 steps, result which has been then improved to 2n − 3 by Shen et al in [4] in assuming that each link is used at most twice. Under the single port MIMD communication model, Zhang in [5] proposes a routing in O(n) steps on a spanning tree of the hypercube. While the above results considered offline routing and clearly established communication models, others works like the ones by Hwang et al [6,7] considered online oblivious routing under buffered all port MIMD communication models. They prove that in using local information, n steps routing is possible for n ≤ 12. The better routings under the previous models are due to V¨ockling. In [8], he √ proves on one side that deterministic offline routing can be done in n + O( n log n) steps on the other side that online oblivious randomized routing can be made in n + O(n/ log n) steps but in buffered all port MIMD model.
24
J.-P. Jung and I. Sakho
For the more restrictive single-port, MIMD queueless communication model the works can be classified in routing of arbitrary and particular permutations among which the BPC. For arbitrary permutations, the personal communication of Coperman to Ramras and the works of Ramras [9] constitute certainly the leading ones. Indeed, while Coperman gives the computational proof that any arbitrary permutation in the 3D-hypercube can be routed in 3 steps. Ramras proves that if a permutation can be routed in r steps in rD-hypercubes, then for n ≥ r arbitrary permutations can be routed in 2n − r steps. Thus, it follows that arbitrary permutations can be routed in 2n − 3 steps improving so the 2n − 1 routing steps of Gu and Tamaki [10]. Recently, Laing and Krumme in [11] have introduced an approach based on the concept of k-separability that is the possibility to partition a permutation after k routing steps into 2k permutations on disjoints (n − k)D-hypercubes. For BPC, Nassimi and Sahni in [12] prove that under SIMD communication models BPC can be routed in n steps. Johnson and Ho present algorithms for matrix transposition in all-port MIMD [13] and all-to-all personalized [14] communication models. Recently, M´elatiaga and Tchuent´e have developed an n-steps parallel routing algorithm.
4
Mathematical Foundation
Hypercubes partitioning is the foundation of our routing algorithm. It is based on the computation of bipartite graphs perfect matching. 4.1
Definitions
Definition 4. A bipartite graph is a graph whose the nodes set V = V1 ∪ V2 and the edges set E is constituted of edges (u, v) such that u ∈ V1 and v ∈ V2 . Definition 5. The adjacency matrix of a bipartite graph is the |V1 |×|V2 | matrix M whose each component M [u, v] = 1 (resp. 0) if (u, v) ∈ (resp. ∈) / E. Let’s observe that, as an nD-hypercube can be viewed as the interconnection of two (n−1)D-hypercubes in anyone of its n dimensions, its adjacency matrix M is anyone of the n 2 × 2 block matrix M (i) whose block M (i) [x, x] (resp. M (i) [x, x]) (n) is the adjacency matrix of the (n − 1)D-hypercube Hi,x (resp. interconnection (n)
(n)
graph of the nodes of Hi,x to the ones of Hi,x ). Table 1 illustrates such a matrix. Definition 6. A matching of a bipartite graph G is an one-to-one correspondence Γ which associates to each node u of a subset of V1 a node Γ (u) of V2 such that the edges (u, Γ (u)) belong to E and are two-by-two non adjacent. A matching Γ is said to be maximum (resp. perfect, maximal) if its cardinal |Γ | is maximum (resp. = |V1 | = |V2 |, maximal). The main result about the computation of maximum and perfect matching is due to C. Berge [15].
Optimal Queueless Routing for BPC on Hypercubes
25
Table 1. 4D-hypercube adjacency matrix view according to dimension i = 2 2 0 1 2 3 8 9 10 11 4 5 6 7 12 13 14 15
0 1 1 1
1 1 1 1
2 1 1 1
3
8 1
1 1 1
1 1
10 11
4 1
1
5
6
1
1
12 13 14 15
1 1
1 1
7
1 1
1 1 1
1
9
1
1 1 1
1 1 1 1
1
1 1 1 1 1 1
1 1 1
1 1 1
1
1 1 1
1 1 1 1
1 1
1 1
1 1 1 1 1
1 1
1
1
1 1 1
1 1 1
1 1 1
Theorem 1 (of C. Berge). A matching Γ of a bipartite graph G is maximum if and only if there is no path with extremities nodes not saturated by Γ which alternates edges belonging and edges not belonging to Γ . The implementations of this theorem have lead to several algorithms. We will use the Neiman algorithm [16] which proceeds by distinguishing one and only one “1” by row and by column in the adjacency matrix of the bipartite graph. In the sequel, the non distinguished “1” will be double crossed. 4.2
Characterization of Partitionable Permutations
Given a permutation π on H (n) , for x = 0, 1 and 0 ≤ i ≤ n − 1, let n min be its minimal number of routing steps, Gx,i the bipartite graph (Sx,i , Dx,i , E), where Sx,i = {u ∈ H (n) : πi (u) = x} and Dx,i = {u ∈ H (n) : ui = x} and Γ x,i a maximum matching of Gx,i . Definition 7. A permutation π on H (n) is partitionable in a dimension i if there is a permutation Γ = (Γ 0,i , Γ 1,i ) on H (n) routable in one step and which leads to the distinct permutation α (resp. β) on a distinct (n − 1)D-hypercube, routable in n min − 1 steps and which associates π(u) to each Γ (u) such that Γi (u) = 0 (resp. 1). Actually, the partionability of a permutation on H (n) is a guaranty that it can be decomposed, after at most one exchange step, in two independent permutations on two disjoint (n − 1)D hypercubes. Now let’s characterize the partitionable permutations. Lemma 1. A necessary and sufficient condition for a permutation on an nDhypercube to be partitionable is that there is a dimension i such that for any x, the bipartite graph Gx,i = (Sx,i , Dx,i , E) admits a perfect matching.
26
J.-P. Jung and I. Sakho
Proof. It is straightforward. It comes from Definition 4.
Lemma 2. A necessary and sufficient condition for the bipartite graph Gx,i = (Sx,i , Dx,i , E) to admit no perfect matching is that there is v ∈ Dx,i which cannot match with any node u ∈ Sx,i . Proof. The condition is obviously sufficient. To prove that it is also necessary it suffices to prove that the adjacency matrix, say N (i,x) , of Gx,i has at least one null column. To do this let’s first observe that this adjacency matrix is a mix of rows of M (i) [x, x] and M (i) [x, x]. Then by suited rows and columns permutations it can be rewritten in the form of table 2a where m = 2n−1 , A and B, I and O are respectively, non null, identity and null matrices. Secondly let’s observe that, by construction: (a) there is a perfect matching between the rows of I and the columns of B, (b) there is a matching of the rows of A with the columns of A and B which saturates the rows of A, (c) any perfect matching of Gx,i contains the perfect matching between the rows of I and the columns of B. Now let’s suppose that Gx,i does admit no perfect matching and consider any of its maximum matching. From the above properties, necessarily the rows and the columns of A are not perfectly matched, but they are maximally matched. Then from the Neiman algorithm, by permuting the rows and the columns of A, it can be decomposed as shown in table 2b where B’ results from a permutation of the rows of B, A1 contains in its diagonal a maximal matching of the rows and the columns of A, while A2 and A3 are non null matrices which do contain no matching. From the theorem of Berge, there is no alternated path whose initial extremity, a row of A3 , and the terminal extremity, a column of A2 are not saturated. In others words any attempt of building an alternated path from a row of A3 to a column of A2 whose extremities are not saturated will fail while, according to (b) it should be a success. Then necessarily A2 is null and N (i,x) contains at least one null column. Table 2. Block decomposition of N (i,x) according to a maximum matching a u
v1
…
vk
vk
vm
…
1
…
A
B
u uk+1 um
y1…
…yk
yk+1
…
ym
1
k
…
b
O
I
x … … xk xk+1 …
A1
A2
A3
O
B’
O
I
xm
Proposition 1. A necessary and sufficient condition for a permutation on a nD-hypercube to be partitionable is that there is a dimension i such that for any x, the adjacency matrix N (i,x) of the bipartite graph Gx,i does contain no null column. Proof. It is straightforward. It comes from the two previous lemmas.
Optimal Queueless Routing for BPC on Hypercubes
5
27
Optimal Routing of BPC in nD-Hypercubes
In this section we deal with the declination of the above characterization of partitionable permutations for BPC. 5.1
Characterization of Partitionable BPC
Lemma 3. If π is a BPC on H (n) then for any dimension i, Sx,i = Dx,j (resp. Dx,j ) if πi (u) = uj (resp. uj ) for any u ∈ H (n) . Proof. By definition, for any i, there is a dimension j such that πi (u) = uj or πi (u) = uj for any u. As Sx,i = {u ∈ H (n) : πi (u) = x}, if πi (u) = uj then Sx,i = {u ∈ H (n) : uj = x} = Dx,j and if πi (u) = uj then Sx,i = {u ∈ H (n) : uj = x} = {u ∈ H (n) : uj = x} = Dx,j . Proposition 2. Any BPC on H (n) is partitionable in any dimension. Proof. According to proposition 1, we have to prove that for any dimension i, the adjacency matrices N (i,x) contains no null column. Let v be a component of Dx,i . By definition, v = (vn−1 , . . . , vi+1 , x, vi−1 , . . ., vj+1 , vj , vj−1 , . . ., v0 ). According to Lemma 3, we distinguish two cases. Case 1. Sx,i = Dx,j . As Dx,j = {u ∈ H (n) : u = (un−1 , . . . , ui+1 , ui , ui−1 , . . ., uj+1 , x, uj−1 , . . ., u0 )}, then u = (vn−1 , . . . , vi+1 , x, vi−1 , . . ., vj+1 , x, vj−1 , . . ., v0 ) ∈ Dx,i . If vj = x then u = v and N (i,x) [u, v] = 1. If vj = x then u and v differ on only one bit, so they are adjacent nodes and N (i,x) [u, v] = 1. Case 2. Sx,i = Dx,j . In this case too, by a similar reasoning we prove that N (i,x) [u, v] = 1. 5.2
Partitioning Strategy of BPC
In order to determine a partitioning strategy, let’s examine the structure of N (i,x) . Before, let’s remark that for given i and x, N (i,x) is constituted of the adjacency matrices of the bipartite graphs ((Sx,i ∩ Dy,i , Dx,i , E), y = x, x). Let π be a BPC on H (n) , i and j such that πi (u) = uj or πi (u) = uj for any u and k = j (resp. n − 1, j − 1) if j < (resp. =, >)i. We have the following results. Lemma 4. Sx,i ∩ Dy,i is the sequence of the odd (resp. even) rank subsequences if πi (u) = uj (resp. uj ) of the longest sequence of disjoint 2k -terms subsequences of Dy,i in ascending (resp. descending) order if x = 0 (resp. 1) of their addresses. Proof. It comes from the fact that in the binary code table of the first 2n natural integers taken in the ascending order, the ith bit alternates 2i -length sequences of 0 and 1. Indeed, as Sx,i ∩Dy,i = Dz,j ∩Dy,i = {u ∈ H (n) : uj = z and ui = y}, from Lemma 3, the result follows in comparing the ith and j th bits of the binary code table.
28
J.-P. Jung and I. Sakho
Lemma 5. N (i,x) is the 2n−k × 2n−k block matrix of 2k × 2k blocks whose each block [I, J], for 0 ≤ I, J ≤ 2n−(k+1) − 1 is such that: N (i,x) [I, J] = M (i) [(2n−(k+1) −1)x+(−1)x 2I, (2n−(k+1) −1)x+(−1)x J] (resp. M (i) [(2n−(k+1) − 1)x + (−1)x (2I + 1), (2n−(k+1) − 1)x + (−1)xJ]) if πi (u) = uj (resp. πi (u) = uj ). Proof. It is straightforward. Indeed it is a corollary of Lemma 4. Table 3 illustrates from table 1 such adjacency matrices. Table 3. N (2,x) matrices for x = 0 and 1 of a BPC π on a 4D-hypercube such that π2 (u) = u1
0 1 8 9 4 5 12 13
0 1 1 1
1 1 1 1
2 1
3
8 1
1 1 1
9 1 1 1
10 11
1 1
1 1 1 1
Proposition 3. Let Γ be a partition ⎧ ⎨u Γ (u) = u ⊕ 2j ⎩ u ⊕ 2i
15 14 7 6 11 10 3 2
15 14 13 12 1 1 1 1 1 1 1 1 1 1
7 1 1 1
6
5
1 1 1
1
4
1
1 1
of π in dimension i. if (πi (u) = ui , u ∈ H (n) ) else if πi (u) = uj otherwise.
Proof. Let Γ be a perfect matching of Gx,i . We distinguish the following three cases. Case 1. πi (u) = ui , for any u ∈ H (n) . From Lemma 5, N (i,x) coincides with M (i) [x, x] and then admits the one-to-one correspondence (Γ (u) = u, u ∈ Dx,i ) as a perfect matching. Case 2. πi (u) = ui , for any u ∈ H (n) . From Lemma 5, N (i,x) coincides with M (i) [x, x] which is an identity matrix and then admits only the one-to-one correspondence (Γ (u) = u ⊕ 2i , u ∈ Dx,i ) as a perfect matching. Case 3. πi (u) = uj or πi (u) = uj , for any u ∈ H (n) . From Lemma 5, any Γ consists in a perfect matching of N (i,x) blocks belonging to M (i) [x, x] and M (i) [x, x]. Then from Case 2, (Γ (u) = u ⊕ 2i , u ∈ Dx,i ). According to Neiman algorithm, the only “1” residual components of N (i,x) are then the ones of the 2k × 2k identity matrix of M (i) [x, x]. As each of these residual “1” corresponds to the interconnection between the nodes whose addresses differ only on their j th bit, it follows that (Γ (u) = u ⊕ 2j , u ∈ Dx,i ). Γ is illustrated in Table 3 by the non-double-crossed “1”, the double-crossed ones being forbidden according to the Neiman algorithm.
Optimal Queueless Routing for BPC on Hypercubes
5.3
29
Optimal Routing of BPC
The previous subsection specified a way for the first step routing of a BPC on an nD-hypercube: partition the BPC in two distinct permutations α and β on two disjoint (n − 1)D-hypercubes. Now, we have to decide how to route each one of these permutations. In this order, let’s recall that α (resp. β) is the permutation which associates π(u) to node Γ (u) such that Γi (u) = 0 (resp. 1) the node. Proposition 4. Any permutation which results from a partition of a BPC on an nD-hypercube is also a BPC on an (n − 1)D-hypercube. Proof. We have to prove that for any λ ∈ {α, β}, ∀r, ∃s such that λr (Γ (u)) = Γs (u) or Γs (u) for any Γ (u). This is the case for r = i, the partition dimension. Thus let’s consider the permutation α and a dimension r = i. From the definition of α, ∀Γ (u) such that Γi (u) = 0, αr (Γ (u)) = πr (u). As π is a BPC, ∃s such that πr (u) = us or πr (u) = us . In others words, αr (Γ (u)) = us or αr (Γ (u)) = us . We then distinguish the following three cases. Case 1. πi (u) = ui . From Proposition 3, Γ (u) = u for any u. Then αr (Γ (u)) = us = Γs (u) or αr (Γ (u)) = us = Γs (u). Case 2. πi (u) = uj . From Proposition 3, Γ (u) = u ⊕ 2k , where k = i or j for any u. Then αr (Γ (u)) = (Γ (u) ⊕ 2k )s = Γs (u) (resp. Γs (u)) if s = i and s =j (resp. s = i or s = j) or αr (Γ (u)) = (Γ (u) ⊕ 2k )s = Γs (u) (resp. Γs (u)) if s = i and s = j (resp. s = i or s = j). Case 3. πi (u) = uj . The reasoning is similar to the Case 2 one. From the above study, routing a BPC π on an nD-hypercube under queueless communication model consists in recursive partitioning in two different BPC on two disjoint (n − 1)D-hypercubes . At each partition step, each node has either to conserve or to move its message in one of the dimensions i or j for which πi (u) = uj or πi (u) = uj .
6
Conclusion and Perspectives
This paper has addressed the problem of the optimal MIMD queueless routing of BPC on nD-hypercubes. The partitioning paradigm is the framework of the proposed routing algorithm. It consists in decomposing recursively a permutation in independent permutations on disjoint hypercubes. We have proved that any BPC can be partitioned in any dimension of the hypercube and that the resulting permutations are also BPC. Then any BPC can be routed in at most n data exchange steps. The resulting routing algorithm, for each message, consists in a simple comparison of two bits of its source and destination addresses. Clearly, the proposed routing algorithm looks like a self routing one. Unfortunately, these bits must have the same weight for all the nodes u and then require a consensual choice. Thus one of our future works will concern the BPC self routing. Beyond BPC there are others subclasses of permutations which are also subject to great attention. It is for instance the Linearly Transformed and Complemented, Omega and Inverse Omega permutations. So we also plan to test the applicability of the partitioning paradigm to these classes of permutations.
30
J.-P. Jung and I. Sakho
Acknowledgement. The conversion of this work to LATEX according to the LNCS format has been kindly realized with help of Frank Holzwarth, Springer DE.
References 1. Draper, J.T., Ghosh, J.: Multipath e-cube algorithms (MECA) for adaptive wormhole routing and broadcasting in k-ary n-cubes. In: Proceedings of International Parallel Processing Symposium, pp. 407–410 (1992) 2. Szymanski, T.: On the permutation capability of a circuit switched hypercube. In: Proceedings of the 1989 International Conference on Parallel Processing, pp. I-103–I-110 (1989) 3. Lubiw, A.: Counter example to a conjecture of Szymanski on hypercube routing. Informations Processing Letters 35, 57–61 (1990) 4. Shen, X., Hu, Q., Liang, W.: Realization of arbitrary permutations on a hypercube. Informations Processing Letters 51(5), 237–243 (1994) 5. Zhang, L.: Optimal bounds for matching routing on trees. In: Proceedings of the 8th Annual ACM-SIAM Symposium on Discrete Algorithms, New Orleans, pp. 445–453 (1997) 6. Hwang, F., Yao, Y., Grammatikakis, M.: A d-move local permutation routing for d-cube. Discrete Applied Mathematics 72, 199–207 (1997) 7. Hwang, F., Yao, Y., Dasgupta, B.: Some permutation routing algorithms for low dimensional hypercubes. Theoretical Computer Science 270, 111–124 (2002) 8. V¨ ocking, B.: Almost optimal permutation routing on hypercubes. In: Proceedings of the 33rd Annual ACM- Symposium on Theory of Computing, pp. 530–539. ACM Press, New York (2001) 9. Ramras, M.: Routing permutations on a graph. Networks 23, 391–398 (1993) 10. Gu, Q.-P., Tamaki, H.: Routing a permutation in the hypercube by two sets of edge disjoint paths. J.of Parallel and Distr. Computing 44, 147–152 (1997) 11. Laing, A.K., Krumme, D.W.: Optimal Permutation Routing for Low-dimensional Hypercubes. EEECS Tufts University (July 11, 2003) 12. Nassimi, D., Sahni, S.: Optimal BPC permutations on a cube connected SIMD computer. IEEE Trans. Comput. C-31(4), 338–341 (1982) 13. Johnsson, S.L., Ho, C.T.: Algorithms for matrix transposition for boolean n-cube configured ensemble architectures. SIAM J. Matrix Appl. 9(3), 419–454 (1988) 14. Johnsson, S.L., Ho, C.T.: Optimal communication channel utilization for matrix transpose and related permutations on boolean cubes. Disc. Appl. Math. 53(1-3), 251–274 (1994) 15. Berge, C.: Graphes, 3`eme ´edition, Dunod, Paris (1983) 16. Neiman, V.I.: Structures et commandes des r´eseau18 sans blocage, Annales des T´el´ecom (1969)
Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks Marcin Seredynski1 , Tomasz Ignac2 , and Pascal Bouvry2 1
University of Luxembourg, Interdisciplinary Centre for Security, Reliability and Trust, 6, rue Coudenhove Kalergi, L-1359, Luxembourg, Luxembourg
[email protected] 2 University of Luxembourg, Faculty of Sciences, Technology and Communication
[email protected]
Abstract. A wireless mobile ad hoc network consists of a number of devices that form a temporary network that operates without a support of any fixed infrastructure. Its distributed nature, lack of a single authority, and limited battery resources may lead participating devices to be reluctant to packet forwarding, which is a key assumption of such a network. This paper demonstrates how cooperation can be reached by means of a new trust-based probabilistic packet relaying scheme. Its most significant properties are relativity of the evaluation of trust and the use of activity as one of the trust measures. Computational experiments demonstrated that the more selfish nodes were present in the network, the more reasonable was to use strict strategies by the remaining nodes, which in turns led to a good protection of the network from free-riders. Keywords: Wireless ad hoc networks, trust-based cooperation.
1
Introduction
A correct operation of wireless mobile ad hoc network (MANET) requires its participants to cooperate on packet relaying. Nodes reluctant to this duty are referred to as selfish nodes or free-riders. Most of devices in MANET rely on battery, therefore the risk of a selfish behaviour driven by the battery conservation need is very high. As shown in the literature, MANET composed of nodes that belong to different authorities can suffer from free-riding [1,2,3]. The problem of selfish behaviour can be solved if nodes relay packets only on behalf of cooperative nodes. This requires network participants to use a local trust or reputation systems in order to collect information regarding behaviour of other nodes. The main difference between the two systems is that in the former a node evaluates a subjective view of the entities trustworthiness, while in the latter the view of the whole community is included [4]. Several packet relaying approaches have been proposed in the literature [5,6,7,8,9]. However, in these works it was typically assumed that all nodes follow the same deterministic relaying approach. Since in R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 31–40, 2010. c Springer-Verlag Berlin Heidelberg 2010
32
M. Seredynski, T. Ignac, and P. Bouvry
civilian MANET nodes belong to different authorities such an assumption might be difficult to attain. In this paper a different approach is introduced: firstly a new probabilistic relaying scheme is defined and next it is examined in various scenarios, in which nodes choose the relaying strategies that differ in its level of cooperation. Similarly to the approach introduced in [10] an activity trust measure is used as a key component to discourage nodes from strategically re-joining the network with a new identity (whitewashing action [11]). A new approach for the trust system is introduced: network participants are not classified as selfish or cooperative by some fixed standards, but rather they are compared against each other, so that one can view the system as an evaluation of subjective ranking of nodes. The next section introduces the new model of a probabilistic packet relaying. This is followed by Section 3, which contains simulation results. Last section concludes the paper.
2
Probabilistic Relaying Model
Each time an intermediate node receives a routing request (RREQ) it checks trustworthiness of the original sender of the message (source node). The action whether to relay or discard the packet originated by a node with a given trust is specified by a probabilistic relaying strategy. 2.1
Local Trust Evaluation System
In this work it is assumed that nodes use only a trust system. A source routing protocol is assumed to be used, which means that a list of intermediate nodes is included in the packet’s header. The information regarding behaviour of other nodes (referred later to as trust data) is gathered only by nodes directly participating in the communication session. The communication session involves a sender, several forwarders (nodes that relay packets) and a destination node. The trust data collection is performed in the following way: nodes are equipped with a watchdog mechanism (WD) [2] that enables them to check if a packet was successfully delivered to the destination. A node that asks another node to relay a packet verifies by means of a passive acknowledgment [12] whether the node has actually accepted the RREQ. As an example, let us assume that node n1 sends a message to node n4 via intermediate nodes n2 and n3 , and eventually the message is discarded by node n3 . This event is recorded by the WD mechanism of node n2 , which next informs node n1 about selfish behaviour of n3 . As a result, trust system of node n1 is updated with the two events - “packet relayed by n2 ” and “packet discarded by n3 ”, while the trust system of n2 is updated with the event “packet discarded by n3 ”. The detailed analysis of the performance of the WD can be found in [2]. The trust data are classified into two types referred to as personal and general trust data. The former takes considers status of packets originated by a node itself, while in the latter the status of packets originated by other nodes is taken into account. In consequence, node in a sender role collects personal trust data, while forwarders collect general trust data. The evaluation
Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks
33
of trust of node j (source of the packet) by node i asked to relay the packet is based on two characteristics, namely relative forwarding rate (denoted by f rj|i ) and relative activity (denoted by acj|i ). Later in the text, relative forwarding rate and relative activity will be refered to as forwarding rate and activity. The notation j|i means that node j is under the evaluation of node i. Both values result from WD-based observations made by node i during its participation in the network. Only personal trust data is considered, however if such data are unavailable then general trust data is taken into account. Forwarding rate is defined over the context of ratio of packets relayed to packets discarded by a node, while activity over node’s degree of participation to the forwarding duties. It is assumed that acj|i , f rj|i ∈ [0, 1]. In the next step node i evaluates trust of node j, which is a number from the interval [0, 1]. Formally, the trust (evj|i ) is defined as a function of forwarding rate and activity: evj|i = fi (f rj|i , acj|i ), where fi : [0, 1] ⊗ [0, 1] → [0, 1]. Nodes might have different criteria of the trust evaluation, hence every function f is indexed with an index of an evaluating node. Node i observes two characteristics of j, namely req accj|i and req dscj|i , which are respectively numbers of packets relayed and discarded by j. Then, the node computes the RREQ acceptance ratio (rarj|i ) of node j: rarj|i =
req accj|i . req accj|i + req dscj|i
The values f rj|i and acj|i are functions of rarj|i and req accj|i respectively. Obviously, such functions can be defined in many various ways. However, in this paper the following definitions are proposed: 1 · rark|i < rarj|i , Oi − 1
(1)
1 · req acck|i < req accj|i , Oi − 1
(2)
f rj|i =
k∈Oi
acj|i =
k∈Oi
where Statement =
1 0
if Statement is true, if Statement is false.
Oi denotes a set of all nodes observed by node i and Oi stands for its size. The forwarding rate value defined by the equation (1) is a fraction of nodes that have lower forwarding rate than node j. A similar interpretation holds for the activity. Finally, the evj|i is computed using the following formula: evj|i = w1 · f rj|i + w2 · acj|i ,
(3)
34
M. Seredynski, T. Ignac, and P. Bouvry
where w1 , w2 are weights summing to one. In this paper the following assignment was made w1 = w1 = 12 . However, demanding on certain factors one may want to promote nodes with higher forwarding rate value or with higher activity by changing values of w1 and w2 . Based on histograms of rarj|i and req accj|i , it is possible to approximate density functions of distributions of RREQ acceptance ratio and number of relayed packets (denoted by rari (x) and req acci (x) respectively). Please note, that the function rari (x) is defined on the interval [0, 1] and req acci (x) may be considered on the semi-axis [0, +∞]. Now, equations (1) and (2) can be replaced by: rarj|i f rj|i = rari (x)dx, (4) 0
acj|i =
req accj|i
req acci (x)dx.
(5)
0
In large networks, it may be impossible to collect informations about all nodes. In such a case, one can take an advantage of the statistical approach. Even, if there is no information about node j, there still exists possibility to evaluate it in a reasonable way. A natural choice in this case is to compute expected values of forwarding rate and activity:
1
x · rari (x)dx,
f rj|i = 0
acj|i = 2.2
∞
x · req acci (x)dx.
0
Decision-Making: Probabilistic Relaying Strategy
The next step for a node that received the packet for relaying is to decide whether to accept or reject the RREQ originated by a node with a given trust. The decision is based on the relaying strategy which is a function of evj|i (trust), and is denoted by si (evj|i ). A probabilistic strategy is used, i.e., si (evj|i ) is a probability that i will relay a packet originated by j. This implies that: si : [0, 1] → [0, 1]. In the experiments presented in the next section simple strategies were used. The aim of these experiments was to demonstrate that the proposed approach efficiently enforces cooperation between nodes in a network. However, if one wants to build a stable, large network there will be a need to use more complex strategies. The reason for this is that, the choice of a strategy is motivated by many factors, such as battery level, number of RREQ or computational resource of network devices. Moreover, a high variability of conditions in MANET is assumed. Hence, nodes will be forced to adapt their strategies in a dynamic way
Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks
35
due to changes of the environment. Experimental results suggested that when choosing a strategy one must take into account a level of cooperation of other nodes. This is another parameter that should be incorporated into the process of the strategy selection. The scope of our future work is to construct an AI module which will be responsible for the strategy management. A dynamic environment of MANET implies that one must be able to describe an evolution of a node’s strategy as a stochastic process. Hence, it would be effectively if all strategies of a node were members of the same family of probability distributions. A natural candidate is the beta distribution, which is defined on the interval [0, 1] and is characterized by two parameters, a and b. For such a distribution the density function is defined by the formula: da,b (x) = 1 0
xa−1 (1 − x)b−1 ua−1 (1 − u)b−1 du
,
for x ∈ [0, 1].
(6)
The strategy is defined as a cumulative distribution function related to da,b (x), namely: evj|i da,b (x)dx. (7) si (evj|i ) = 0
Parameters a and b determine the shape of si (x). The advantage of the beta distribution is that changing two parameters enables to cover a large set of strategies differing in their level of cooperativeness. Such a definition needs an additional assumption that a strategy is a nondecreasing function. Let us present an example of a parameter that will strongly affect the choice of a strategy - the distribution of a RREQ sent to node i. We introduce an auxiliary function ri (x) : [0, 1] → [0, +∞]. The value ri (x) is an average number of RREQ in a time unit received from nodes with the trust equal to x. The ri (x) is computed by node i out of the past RREQ received by the node. The ri (x) may be one of the key parameters used for selecting a strategy in a situation of limited power resource or channel capacity. For example, one can compute the expected value of packets relayed by i per a unit of time (with a given strategy si (x)): 1 Ki = ri (x)si (x)dx. (8) 0
However, the choice based only on Ki may be not the optimal one. There may be a need to compute other parameters of the ri (x), for instance a standard deviation, before choosing a strategy. This problem will be considered in a forthcoming paper.
3 3.1
Computational Experiments Model of the Network
The network was composed of a finite population of N nodes, while time was divided into R rounds. In each round every node initiated a single communication
36
M. Seredynski, T. Ignac, and P. Bouvry
session exactly once (acted as a sender), thus each round was composed of N communication sessions. Since dynamics of a typical MANET expressed in terms of mobility and connectivity of the nodes are unpredictable [13], intermediate and destination nodes were chosen randomly. The trust data were also used by nodes for originating its own packets. Each network participant used a path rating mechanism in order to avoid distrusted nodes. If a source node had more than one path available to the destination it would choose the one with the best rating. The rating was calculated as an arithmetic mean of trust of all nodes belonging to the route (trust of an unknown node was set to 0.3). The simulation of the network was defined by the following algorithm: 1. Specify i (source node) as i := 1, N as a number of nodes participating in the network and R as a number of rounds; 2. Randomly select node j (destination of the packet) and intermediate nodes, forming several possible paths from node i to j; 3. For each available path calculate arithmetic mean of the trust of all nodes belonging to the path and choose the path with best rating; 4. Let node i := 1 originate a packet (initiate a communication session), which is next processed by intermediate nodes according to their strategies; 5. As soon as the communication session is completed update the trust data as described in Section 2.1; 6. If i < N , then choose the next node (i := i + 1) and go to step 2. Else go to step 7; 7. If r < R, then r := r + 1 and go to step 1 (next round). Else stop the simulation. 3.2
Description of Experiments
A general form of a relaying strategy evaluated in the experiments was as follows: ⎧ p1 for evj ∈ [0, 0.4] ⎪ ⎪ ⎪ ⎪ for evj ∈ (0.4, 0.7] ⎨ p2 s(evj ) = p3 for evj ∈ (0.7, 1] ⎪ ⎪ for an unknown node in rounds 1 − 50 p ⎪ 4 ⎪ ⎩ p5 for an unknown node in rounds 51 − 600. Three types strategies were defined: a tolerant strategy (T), a strict strategy (S), and a non-cooperative strategy (N). Their relaying probabilities are shown in Table 1. The non-cooperative strategy forwarded small amount of packets in order to make it more difficult to be discovered. The T and S-type strategies were more cooperative (probability p4 set to 1) towards unknown nodes in the initial period (between rounds 1 and 50), while after that period these nodes turned to be non-cooperative (probability p5 set to 0.1). The reason for the non-cooperative behaviour after certain period was that this discouraged nodes from changing identity in order to take advantage of cooperative approach towards unknown nodes. Four sets of scenarios differing in the number of nodes using selfish strategies (represented by N strategy) were
Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks
37
Table 1. Forwarding probabilities of the T, S and N-type strategies p1 p2 p3 p4 p5
Tolerant strategy (T) Strict strategy (S) Non-cooperative strategy (N) 0.4 0 0 0.9 0.7 0.1 1 1 0.15 1 1 0.1 0.3 0.1 0
analyzed. In the first set (s1.1 -s1.3 ) selfish strategies were not present at all, in the second set (s2.1 -s2.3 ) 20% of nodes used the N strategy, in the third (s3.1 -s3.3 ) 40%, while in the last one (s4.1 -s4.3 ) 60%. In each set three different distributions of the remaining T and S-types strategies were defined (s∗.1 with a dominance of T strategies, s∗.2 with equal number of both strategies and s∗.3 with dominance of S strategies). Dominance was defined as a situation, in which the dominating strategy was used by around 90% of remaining nodes. The exact settings of scenarios are shown in Table 2. Table 2. Composition of strategies used by nodes in tested scenarios s1.1 #N 0 # T 54 #S 6
s1.2 0 30 30
s1.3 0 6 54
s2.1 12 43 5
s2.2 12 24 24
s2.3 12 5 43
s3.1 24 32 4
s3.2 24 18 18
s3.3 24 4 32
s4.1 36 22 2
s4.2 36 12 12
s4.3 36 2 22
The following performance characteristics of nodes using a given type of strategy were used: a throughput (tp) defined as a ratio of successful message delivery, a number of packets relayed (npr) by a node and a relaying rate (rr) specifying a ratio of packets relayed to packets discarded. Each scenario was repeated 50 times. The values of performance characteristics of each type of strategy were calculated as an arithmetic mean over the values of all runs. The evaluation of trust (ev) was performed according to equations (1)-(3). The total number of all nodes (N ) was equal to 60, while the number of rounds in the NTS (R) was set to 600. The path length was set from 1 up to 5 hops with the following probabilities: one hop - 0.1, two hops - 0.3 and three to five hops - 0.2. The number of available paths from a source to a given destination ranged from 1 up to 4 (each value with equal probability). 3.3
Results
The results of all four sets of scenarios are shown in Table 3. The average throughput ranged from 0.19 (s4.3 ) to 0.67 (s4.1 ), meaning that 19 to 67% of messages were successfully delivered without being re-send. As one might expect, these figures were mainly related to the number of N strategies present in the network.
38
M. Seredynski, T. Ignac, and P. Bouvry
Table 3. Performance of T, S and N-type strategies in the four sets of scenarios: throughputs (tp), numbers of packets relayed (npr) and relaying rates (rr) of nodes using one type of strategy vs. nodes using other type of strategy
s1.1 s1.2 s1.3 s2.1 s2.2 s2.3 s3.1 s3.2 s3.3 s4.1 s4.2 s4.3
overall tp/npr tp by T 0.67 0.69/873 0.56 0.64/821 0.45 0.62/758 0.5 0.58/779 0.43 0.55/740 0.35 0.52/691 0.36 0.47/679 0.31 0.44/650 0.26 0.41/621 0.25 0.35/581 0.22 0.34/569 0.19 0.33/558
tp/npr by S 0.54/623 0.48/580 0.43/516 0.47/550 0.43/511 0.38/459 0.4/462 0.36/425 0.32/385 0.31/366 0.29/339 0.27/318
tp/npr rr: T vs. N, by N S vs. N 0.25/35 0.42, 0.15 0.2/28 0.43, 0.16 0.14/16 0.43, 0.16 0.21/28 0.43, 0.15 0.18/24 0.43, 0.16 0.15/20 0.44, 0.16 0.19/21 0.45, 0.16 0.16/18 0.45, 0.16 0.13/15 0.45, 0.17
rr: T vs. T T vs. S 0.84, 0.73 0.86, 0.75 0.88, 0.77 0.86, 0.75 0.89, 0.78 0.9, 0.81 0.9, 0.8 0.91, 0.82 0.91, 0.83 0.93, 0.86 0.93, 0.86 0.95, 0.87
rr: S vs. T S vs. S 0.68, 0.51 0.72, 0.54 0.76, 0.58 0.71, 0.55 0.75, 0.57 0.79, 0.61 0.76, 0.59 0.78, 0.61 0.8, 0.63 0.81, 0.67 0.82, 0.67 0.84, 0.7
In the first set of experiments, where nodes did not use non-cooperative strategies the throughput was between 0.45 and 0.67. The worse performance was observed in set 4 (0.19 to 0.25), which was due a dominance of N strategies (used by 60% of nodes). Differences between three variants of scenarios within a given set were related to the proportion of T to S strategies. As T-type strategies were generous in their contribution to packet forwarding, their presence improved the performance of nodes using all other types of strategies (including the selfish ones). Nodes using selfish strategies (N strategies) managed to send from 13% (s4.3 ) up to 25% (s2.1 ) of packets, which was far below the values obtained by nodes using other types of strategies. The T strategy scored the best performance (throughput from 0.33 to 0.69), while S came second with throughput ranging from 0.27 up to 0.54. These values were positively correlated with participation to packet relying. Nodes that used T strategy forwarded on average from 558 (s4.3 ) to 873 (s1.1 ) packets each, while nodes following S strategy relayed 318 (s4.3 ) up to 623 (s1.1 ) packets. On the other hand, selfish users almost did not participate to forwarding duties at all (forwarding 15 to 35 packets). Since all strategies were trust-dependent the relaying rates depended on type of node requesting forwarding service. Nodes using S strategies were the most demanding ones in terms of trust requirements regarding source nodes. Nodes using these strategies prevented selfish nodes from obtaining good throughput, as relaying rates concerning nodes using N strategies were around 0.15. On the other hand, these nodes were by far more cooperative towards nodes that used T strategies (relaying rates ranging from 0.68 to 0.84) and S strategies (0.45-0.63). Nodes that used T strategies were even more cooperative with relaying rates being in a range of 0.84-0.95 towards nodes using T strategies and 0.73-0.87 towards S strategies. These nodes were also quite cooperative towards selfish nodes (relaying rates around 0.43), but one has to keep in mind that as a path is typically
Probabilistic Packet Relaying in Wireless Mobile Ad Hoc Networks
39
composed of more than one intermediate hop, such rate is not enough to obtain a good throughput. This was demonstrated by scenarios s2.1 , s3.1 and s4.1 , where selfish nodes were accompanied mostly by nodes using T strategies. In these scenarios, nodes using N strategies managed to successfully send at most 25% of packets. As in MANET strategy selection process is done independently by each node the question is what strategies would be most probably chosen by network participants - the more or the less cooperative ones? Such a decision would most probably be driven by two factors: state of battery resources and a level of performance that a node would like to obtain from the network. The better the battery is and the more expectations from a network a node has, the more likely it is going to use more cooperative strategy. As demonstrated by the four sets of experiments, being more cooperative results in a better level of service received from the network (defined here as throughput). But as more nodes are cooperative, the better performance is obtained by selfish nodes, so one might expect, that nodes might be tempted to switch to selfish strategies. However, as demonstrated by the experiments this is not very likely to happen. For instance, in s1.1 (no selfish nodes present) the S nodes forwarded 623 packets and obtained a throughput equal to 0.54, while T nodes forwarded 40% more packets and obtained a throughput equal to 0.69. In such a case a decision whether to use more or less cooperative strategy was a matter of preferences and battery resources. However, in a large presence of selfish nodes (s4.1 -s4.3 ) T nodes forwarded 59-75% more packets then S nodes, and obtained only slightly higher throughput. It means, that in such a situation (large presence of selfish nodes) it is more reasonable to use a less cooperative strategy. As a result, it is very unlikely, that selfish nodes might have an occasion to free-ride.
4
Conclusions
In this paper a new trust-based probabilistic scheme for packet relaying was introduced. One of its key features is the use of the notion of activity as a basis for the relaying strategy. It promotes identity persistence as nodes are discouraged from a whitewashing action because new identity means that the activity status is reset to zero. Additionally, the relativity of the evaluation of trust enables the trust system to adapt the ratings to the actual networking conditions. Thus, one can view the output of the system as an evaluation of subjective ranking of nodes. Such an approach also helps to deal with uncertainty related to the watchdog-based data collection mechanism. False or inaccurate data occasionally provided by the mechanism affect all nodes in the same way, thus these errors will not affect the ranking of nodes. In consequence, this uncertainty, as long as remains reasonable, will not create any significant threat to the trust system. The computational experiments demonstrated a substantial degree of resistance to free-riding, as in a case of a large number of selfish nodes, the stricter strategies proved to be a better choice. However, the experiments were performed under simplified assumptions and selected strategies were far from
40
M. Seredynski, T. Ignac, and P. Bouvry
the ideal. The scope of our future work is to construct a model of MANET in which additional complex factors will be taken into account.
Acknowledgments This work has been partially founded by C08/IS/21 TITAN Project (CORE programme) financed by National Research Fund of Luxembourg.
References 1. Buttyan, L., Hubaux, J.P.: Nuglets: a virtual currency to stimulate cooperation in self-organized mobile ad hoc networks. Technical Report DSC/2001/001, Swiss Federal Institute of Technology (2001) 2. Marti, S., Giuli, T., Lai, K., Baker, M.: Mitigating routing misbehavior in mobile ad hoc networks. In: Proc. ACM/IEEE 6th International Conference on Mobile Computing and Networking (MobiCom 2000), pp. 255–265 (2000) 3. Michiardi, P., Molva, R.: Simulation-based analysis of security exposures in mobile ad hoc networks. In: Proc. European Wireless Conference (2002) 4. Josang, A., Ismail, R., Boyd, C.: A survey of trust and reputation systems for online service provision. Decision Support Systems 43(2), 618–644 (2007) 5. Buchegger, S., Boudec, J.Y.L.: Performance analysis of the confidant protocol. In: Proc. 3rd International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc 2002), pp. 226–236 (2002) 6. Buchegger, S., Boudec, J.Y.L.: The effect of rumor spreading in reputation systems for mobile ad-hoc networks. In: Proc. Workshop on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt 2003), pp. 131–140 (2003) 7. Buchegger, S., Boudec, J.Y.L.: Self-policing mobile ad hoc networks by reputation systems. IEEE Communications Magazine, Special Topic on Advances in SelfOrganizing Networks 43(7), 101–107 (2005) 8. Giordano, S., Urpi, A.: Self-organized and cooperative ad hoc networking. In: Basagni, S., Conti, M., Giordano, S., Stojmenovic, I. (eds.) Mobile Ad Hoc Networking, pp. 355–371. Wiley, IEEE Press (2004) 9. Michiardi, P., Molva, R.: Core: A collaborative reputation mechanism to enforce node cooperation in mobile ad hoc networks. In: Proc. 6th Conference on Security Communications, and Multimedia (CMS 2002), pp. 107–121 (2002) 10. Seredynski, M., Bouvry, P., Klopotek, M.A.: Analysis of distributed packet forwarding strategies in ad hoc networks. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 68–77. Springer, Heidelberg (2008) 11. Feldman, M., Papadimitriou, C., Chuang, J., Stoica, I.: Free-riding and whitewashing in peer-to-peer systems. IEEE Journal on Selected Areas in Communications 24(5), 1010–1019 (2006) 12. Jubin, J., Turnow, J.D.: The darpa packet radio network protocols. Proceedings of the IEEE 75(1), 21–32 (1987) 13. Fischer, D., Basin, D., Engel, T.: Topology dynamics and routing for predictable mobile networks. In: Proc. The 16th IEEE International Conference on Network Protocols (ICNP 2008). IEEE, Los Alamitos (2008)
On the Performance of a New Parallel Algorithm for Large-Scale Simulations of Nonlinear Partial Differential Equations ´ Juan A. Acebr´on1 , Angel Rodr´ıguez-Rozas1, and Renato Spigler2 1
2
Center for Mathematics and its Applications, Department of Mathematics, Instituto Superior T´ecnico, Av. Rovisco Pais 1049-001 Lisboa, Portugal
[email protected],
[email protected] Dipartimento di Matematica, Universit` a “Roma Tre”, Largo S.L. Murialdo 1, 00146 Rome, Italy
[email protected]
Abstract. A new parallel numerical algorithm based on generating suitable random trees has been developed for solving nonlinear parabolic partial differential equations. This algorithm is suited for current high performance supercomputers, showing a remarkable performance and arbitrary scalability. While classical techniques based on a deterministic domain decomposition exhibits strong limitations when increasing the size of the problem (mainly due to the intercommunication overhead), probabilistic methods allow us to exploit massively parallel architectures since the problem can be fully decoupled. Some examples have been run on a high performance computer, being scalability and performance carefully analyzed. Large-scale simulations confirmed that computational time decreases proportionally to the cube of the number of processors, whereas memory reduces quadratically.
1
Introduction
An efficient design of numerical parallel algorithms for large-scale simulations becomes crucial when solving realistic problems arising from Science and Engineering. Most of them are based on domain decomposition (DD) techniques [10], since it has been shown to be specially suited for parallel computers. Nevertheless, classical approaches based on domain decomposition usually suffer from process intercommunication and synchronization, and consequently the scalability of the algorithm turns out to be seriously degraded. Moreover, an additional excess of computation is often introduced when designing the algorithms in order to diminish the effects of the two previous issues. From the parallelism point of view, probabilistic methods based on Monte Carlo techniques offer a promising alternative to overcome these problems. The possibility of using parallel computers to solve efficiently certain partial differential equations (PDEs) based on its probabilistic representation was R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 41–50, 2010. c Springer-Verlag Berlin Heidelberg 2010
42
´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.
recently explored. In fact, in [1,2], an algorithm for numerically solving linear two-dimensional boundary-value problems for elliptic PDEs, exploiting the probabilistic representation of solutions, were introduced for the first time. This consists in a hybrid algorithm, which combines the classical domain decomposition method [12], and the probabilistic method, and was called “probabilistic domain decomposition method” (PDD for short). The probabilistic method was used merely to obtain only very few values of the solution at some points internal to the domain, and then interpolating on such points, thus obtaining continuous approximations of the sought solution on suitable interfaces. Known the solution at the interfaces, a full decoupling in arbitrarily many independent subdomains can be accomplished, and a classical local solver, arbitrarily chosen, used. This fact represents a definitely more advantageous circumstance, compared to what happens in any other existing deterministic domain decomposition method. However, in contrast with the linear PDEs, probabilistic representation for nonlinear problems only exists in very particular cases. In [5], we extended the PDD method to treat nonlinear parabolic one-dimensional problems, while in [6] it was conveniently generalized to deal with arbitrary space dimensions. In this paper, for the first time the performance of the PDD algorithm in terms of computational cost and the memory consumption will be carefully analyzed. To this purpose, large-scale simulations in a high performance supercomputer were runned, and the advantageous scalability properties of the algorithm verified. Since the local solver for the independent subproblems can be arbitrarily chosen, for this paper we chose a direct method based on LAPACK for solving the ensuing linear algebra problem. This is so because ScaLAPACK were selected to compare the performance of our PDD method, being ScaLAPACK a competitive, freely available and widely used parallel numerical library. Other alternatives based on iterative methods are worth to be investigated, and will be done in a future work. In the following, we review briefly the mathematical fundamentals underlying the probabilistic representation of the solution of nonlinear parabolic PDEs. More precisely, for simplicity let consider the following initial-boundary value problem for a nonlinear two-dimensional parabolic PDE, ∂2u ∂2u ∂u ∂u ∂u = Dx (x, y) 2 + Dy (x, y) 2 + Gx (x, y) + Gy (x, y) − cu + αi ui , ∂t ∂x ∂y ∂x ∂y i=2 m
in Ω = [−L, L] × [−L, L], t > 0, u(x, y, 0) = f (x, y), u(x, y, t)|x∈∂Ω = g(x, y, t),
(1)
m where x = (x, y) ∈ R2 , αi ∈ [0, 1], i=2 αi = 1, being f (x, y) and g(x, y, t), the initial condition and boundary data, respectively. The probabilistic representation for the solution of (1) was explicitly derived in [6] for the n-dimensional case, and requires generating suitable random trees with so many branches as the power of the nonlinearity found in (1). Such random trees play the role that a random path plays in the linear case, (cf. Feynman-Kac formula for linear parabolic PDE). Mathematically, the solution of (1) for a purely initial value
On the Performance of a New Parallel Algorithm
43
problem at (x, t) can be represented (see [11]) as follows k(ω)
u(x, t) = Ex [
f (xi (t, ω))].
(2)
i=1
Here k(ω) is the number of branches of the random tree starting at x, and reaching the final time t. xi (t, ω) is the position of the ith branch of the random tree, corresponding to the stochastic process solution of the stochastic differential equation: dβ = b(x, y, t) dt + σ(x, y, t) dW (t), (3) where W (t) represents the standard Brownian motion, and b(x, y, t) and σ(x, y, t), the corresponding drift and diffusion related to the coefficients of the elliptic operator in (1). The different possible realizations of the random trees are labeled by ω, and a suitable average Ex should be taken over all trees. In case of a initialboundary value problem, with the boundary data u(x, t)|x∈∂Ω = g(x, t), a similar representation holds, i.e., ⎤ ⎡ k(ω) u(x, t) = Ex ⎣ f (xi (t, ω))1[Si >τ∂Ωi ] + g(βi (τ∂Ωi ), τ∂Ωi )1[Si <τ∂Ωi ] ⎦ , (4) i=1
where τ∂Ωi is the first exit time out of the domain of the stochastic process β(·); Si a random time governed by the exponential distribution p(Si ) = c exp(−c Si ), and 1[Si <τΩi ] is the indicator function, being 1 or 0 depending whether τ∂Ωi is or is not greater than Si . For more mathematical details, we refer to the reader to [5] and [6]. In practice, the probabilistic representation in (2) works as follows: A stochastic process initially located in (x, 0) (root of the tree) is generated and evolves in time. Simultaneously, a random time S1 , picked up from the exponential probability density p(S) = c exp(−c S), is generated. If the random time S1 is less than t, a number of branches corresponding to the power of the nonlinearity are created. n S > t. WhenThis procedure is repeated successively until holds that i=1 i ever one of these possible branches reaches the final time, t, the initial value function, f , is evaluated at the position where the stochastic process associated to every branch was located. The solution is finally reconstructed multiplying all contributions coming from each branch. In practice, to achieve a reasonable small error in (4), a large number of random trees should be generated. More details on the numerical errors related with the probabilistic representation are found in [5] and [6]. For the purpose of illustration, in Fig. 1 a sketchy picture is shown. This corresponds to the case of a tree with two branches, one of them reaching the final time t, while the other hitting the boundary. The paper is organized as follows. In Section 2 the PDD algorithm is described, showing their different components, and stressing how they can be parallelized in practice. Here we also discuss the important feature of being a natural faulttolerant algorithm. In Section 3 some examples of nonlinear two-dimensional PDEs illustrate the PDD method, and a deep analysis of the computational cost and memory consumption is also undertaken. Finally, some conclusions are given.
44
´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.
Fig. 1. A picture illustrating a typical random tree in 2D
2
The PDD Method
As it was mentioned above, the core of algorithm requires evaluating the solution of (1) at some convenient given points using a probabilistic representation. The following parallelization strategies can be adopted: • Parallelizing by splitting in independent sets of points. Since the number of points where the solution is computed is larger than the number of processors p, computing such a solution can be assigned as a task to different processors. This can be seen as a coarse-grain parallelization and is the more convenient strategy. • Parallelizing for each point. Recall that for each point, the probabilistic representation requires generating a rather large number of random trees to obtain a sufficiently small statistical error. Since the number of random trees turns out to be often much larger than p, the task to generate only a few of them can be accomplished by different processors. This corresponds to a mid-grain parallelization. • Parallelizing for a given realization. Since every branch pertaining to a given tree is independent from each other, the task to compute only a few branches can be assigned to different processors. This can be seen as a fine-grain parallelization, being by far the most involved, provided that a strong intercommunication and synchronization among the processors is required. However, this level of parallelization offers an important advantage, since it allows to balance easily the overall computational load. We now briefly describe the three parts of the algorithm: Probabilistic part. This is the first step to be carried out, and consists in computing the solution of the PDE at a few suitable points by Monte Carlo, which is esentially a mesh-free method. In Fig. 2 a sketchy diagram is plotted, illustrating how the algorithm works in practice for a two-dimensional case. Here the solution is obtained probabilistically at a few points pertaining to some “interfaces” conveniently chosen inside the space-time domain D := Ω × [0, T ], with Ω ⊂ Rn . Such interfaces divide the domain into p subdomains, Ωi , from i = 1, ..., p, being assigned to different processors, pi , i = 1, ..., p. For instance, in R2 this is done
On the Performance of a New Parallel Algorithm
45
by splitting the domain in slices where the interfaces are parallel to the y-axis, as shown in Fig. 2. Since evaluating by Monte Carlo is computationally costly, only few points are chosen in each plane (y, t), and every processor computes the interfacial values at the corresponding interface located to the right (see Fig. 2). Since each processor shares its interface with its immediately closer neighbor, the interfacial values must be transferred from the processor pi to the processor pi+1 , establishing a next neighbor intercommunication among processors.
Fig. 2. A sketchy diagram illustrating the main steps of the algorithm in 2D: The figure on the left shows how the domain decomposition is done in practice. The figure on the right shows the points where the solution is computed probabilistically; these are used afterwards as nodal points for interpolation.
Interpolation. Once the solution has been computed at few points on each interface, a second step consists in interpolating on such points, being used as nodal points, thus obtaining continuous approximations of interfacial values of the solution. For that purpose, a tensor product interpolation based on cubic spline [7] was used. The computational cost of the two dimensional case is negligible compared with the time spent in the other parts of the algorithm. The nodal points were uniformly distributed on each plane, and a not-a-knot condition was imposed. Local solver. The third and final step consists in computing the solution inside each subdomain, this task being assigned to different processors. This can be accomplished resorting to local solvers, which may use classical numerical schemes, such as finite differences or finite elements methods. Since the comparison with other freely available method was made using ScaLAPACK, LAPACK has been chosen for this purpose. Fault Tolerance-Related Issues We stress that the algorithm is naturally fault-tolerant, being so in any of the parts it is composed. Here, we describe the peculiarities of such features: • Probabilistic part. If a given processor pi fails when computing the solution probabilistically at a given point, either any other processor may continue the pending task or, in case of no free processors at that time, this task may be postponed to be executed later.
46
´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.
• Interpolation. Similarly as in the previous case, when a processor fails, either the subdomain corresponding to such a processor may be reassigned to a neighbor processor or the task may be postponed to be executed afterwards. Note that the failure merely affects to this processor. • Local solver. Let suppose a failure occurs at a certain time t = Tf , when solving locally the PDE by a classical method. Then, the solution obtained at Tf may be used as a new initial data, solving the problem starting now from Tf rather than t = 0.
3
Results and Performance
We now proceed with some numerical examples for two-dimensional problems aiming to illustrate the performance of the PDD algorithm. All simulations were carried out on the MareNostrum Supercomputer located at the Barcelona Supercomputing Center, using up to 1,024 processors. 3.1
Computational Cost and Memory Consumption
The space-time domain inside each subdomain corresponding to the PDD algorithm, as well as, the full domain to compare with the PDD, were discretized using the implicit Crank-Nicolson finite difference method. On the various uncoupled subdomains obtained by the PDD algorithm we used LAPACK as a local solver. In [6], a comparison with the results obtaining with ScaLAPACK was already carried out, showing serious limitations in terms of memory consumption. A striking difference in the performance of both methods was shown, the performance of the PDD method being notably superior to ScaLAPACK. Here we focus mainly on the strong scalability and performance of the PDD method. Being Nx and Ny the number of nodes in each dimension, the resulting matrix A ∈ RNx Ny ×Nx Ny of the linear algebra problem associated to the discretization of the domain is known to be banded, with a bandwidth BW = Nx . Since we parallelized the PDD algorithm assigning the task of solving the local linear system pertaining to each subdomain to different processors, the bandwidth of the associated matrix turns out to be smaller. The local solver for the PDD method, being based on LAPACK for solving banded linear systems (routines dgbtrf/dgbtrs), consists in a LU factorization followed by a forward/backward substitution. The memory consumption per processor, including an extra fill-in space, was also estimated in [6], and is given by MpP DD = 3
Nx2 Ny . p2
(5)
Recall that the computational cost for the LU factorization is known to be of order of Nx Ny BW 2 , while is Nx Ny BW for the forward/backward substitution, Nx Ny being the size of the square matrix and BW = Nx /p. Hence, the computational cost of the local solver for the PDD based on LAPACK is,
On the Performance of a New Parallel Algorithm
Nx Nx Ny N 2 Ny Nx Nx Ny CP DD ≈ αopt (4 + 1) + 3 x2 + p p p p p Nx2 Nx Ny Ny Nx Ny = 4αopt 2 + (αopt + 3) + p p p p
47
(6)
see [8,9,6]. The last two terms are related to the computational time spent to compute the matrix coefficients and the right-hand side, respectively. Here the parameter αopt < 1 was introduced to account for the computational advantages gained when using an optimized LAPACK library, after tuning conveniently the BLAS library for the specific hardware platform used. Typically, BLAS library can be easily tuned through the ATLAS package. To estimate experimentally the value of αopt when runned the simulations at the MareNostrum Supercomputer, we have compiled BLAS with and without ATLAS, and it turns out to be approximately αopt = 0.84. We stress that, when Nx >> p and Ny >> p, the computational cost of the PDD method decreases as p−3 for large p, while for ScaLAPACK it has been reported in [8,9] to decrease rather as p−1 . 3.2
Numerical Results
In this subsection we consider two numerical test problems. Example A. The problem is given by ∂u ∂2u ∂2u ∂u ∂u = (1+x4 ) 2 + (1+y 2 sin2 (y)) 2 + (2+sin(x)ey ) + (2+x2 cos(y)) ∂t ∂x ∂y ∂x ∂y 1 2 1 3 −u + u + u , in Ω = [−L, L] × [−L, L], 0 < t < T, u(x, y, t)|∂Ω = 0, (7) 2 2
πy with initial-condition u(x, y, 0) = cos2 πx cos2 2L . No analytical solution is 2L known for this problem, hence the numerical error was controlled comparing the solution obtained with the PDD method with that given by a finite difference method, with a very fine space-time mesh. Table 1. Example A: TT OT AL denotes the computational time spent in seconds by PDD. TM C , and TINT ERP correspond to the time spent by the Monte Carlo and the interpolation part, respectively; M emory denotes the total memory consumption. No. Processors 128 256 512 1024
TM C 445” 459” 463” 461”
TINT ERP <1” <1” <1” <1”
Memory 0.68 GBs 0.17 GBs 0.04 GBs 0.01 GBs
TT OT AL 12976” 3420” 1166” 595”
In Fig. 3, the pointwise numerical error made when solving the problem of Example A by PDD is shown. Here the value of the parameters were kept fixed to Δx = Δy = 10−2 , Δt = 10−3 and L = 1.
48
´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.
Fig. 3. Pointwise numerical error for the solution of problem ”Example A”, at t = 0.5, using two processors. Note that there is only one interface located at x = 0.
In Table 1 results from the Example A are shown, using up to 1,024 processors. Now, the parameters chosen for the local solver were Δx = Δy = 2.5 × 10−4 , T = 0.5, L = 1 and Δt = 10−3 . Note that the algorithm scales with the number of processors according to a factor of approximately equal to 4 initially with p = 128, when doubling the number of processors. This is so when the number of processors is not too large (p = 128), being the computational workload per processor sufficiently large for this case. However, a factor of 8 is theoretically expected from (6), since Nx /p >> 1 and Ny /p >> 1. An explanation of this fact could be found in the size of the problem, which is not sufficiently large, and because the optimized LAPACK is speeding up considerably the LU factorization. Therefore, the dominant asymptotic behavior is governed, rather, by the second term in (6). Only for the purpose of confirming that this is indeed the case, new simulations for a larger computational load are needed, such that now Nx /p >> 1/αopt and Ny /p >> 1/αopt . The required computational resources in terms of memory, however, largely exceed the available memory (the limits have already being reached). A simple way to circumvent this problem consists in increasing the bandwidth of the associated matrix, keeping almost constant the computational load. This can be done merely considering a rectangular domain instead, Ω = [−Lx , Lx ] × [−Ly , Ly ], with Lx > Ly . Other alternatives based on adopting an heterogeneous computational grid, or even using higher order numerical schemes are equally possible, and will be analyzed elsewhere. From the results above, it is observed that increasing the size of the problem by raising the number of nodes Nx along the x−dimension, results in a computational cost growing cubically with Nx , whereas the memory consumption grows quadratically. Nevertheless, the effect on the y-dimension is more contained, since both the computational cost and the memory consumption increase linearly with the number of nodes Ny . Therefore, to increase the computational workload appreciably it is more convenient to increase the size of the problem along the x−dimension. In fact, this is successfully illustrated in the next example.
On the Performance of a New Parallel Algorithm
49
Example B. Consider the following problem ∂u ∂u ∂2u ∂2u ∂u = (1 + x2 ) 2 + (1 + y 3 ) 2 + (2 + sin(x) ey ) + (2 + sin(x) cos(y)) ∂t ∂x ∂y ∂x ∂y 1 2 1 3 −u + u + u , in Ω = [−Lx , Lx ] × [−Ly , Ly ], 0 < t < T, u(x, y, t)|∂Ω = 0, (8) 2 2
πy and initial-condition u(x, y, 0) = cos2 πx cos2 2L . In Table 2, the results cor2L responding to such a rectangular domain are shown. We used the parameters: Lx = 200, Ly = 0.125, Δx = Δy = 5 × 10−3 , T = 0.5, and Δt = 10−3 . Note that the scaling factor has changed significantly, reaching even the value 5, when passing from 128 to 256 processors. Increasing further the number of processors, implies reducing the scaling factor. This is in good agreement with the theoretical estimates in (6), since, asymptotically, the dominant term changes accordingly to the number of processors involved as well. Table 2. Example B: TT OT AL denotes the computational time spent in seconds by PDD. TM C and TINT ERP correspond to the time spent by the Monte Carlo and the interpolation part, respectively; M emory denotes the total memory consumption. No. Processors 128 256 512 1024
4
TM C 67” 69” 71” 72”
TINT ERP <1” <1” <1” <1”
Memory 1.79 GBs 0.45 GBs 0.11 GBs 0.03 GBs
TT OT AL 54235” 10666” 2601” 748”
Conclusions and Future Work
The performance of a new parallel numerical algorithm for solving nonlinear parabolic partial differential equations based on a probabilistic method, has been investigated. Such a method allows to decouple the full problem into several independent subproblems, the computational cost being dramatically alleviated. Many different sources of parallelism for this algorithm can be explored efficiently, since intercommunication overhead among processors is almost nonexistent. Moreover, the algorithm appears to be fault-tolerant in a natural way. While the computational cost for traditional schemes scales linearly with the number of processors, the PDD algorithm scales quadratically or even cubically, provided that the workload per processor is sufficiently large. Some numerical examples are given, showing the excellent scalability properties of the PDD algorithm in large-scale simulations, using up to 1,024 processors on a high performance supercomputer. Summarizing, the proposed algorithm appears to be a promising alternative to the traditional parallel schemes to solve partial differential equations. Such an algorithm is characterized by desirable features, such as low intercommunication overhead, and fault-tolerance, allowing to exploit at best massively parallel supercomputers as well as Grid computing.
50
´ Rodr´ıguez-Rozas, and R. Spigler J.A. Acebr´ on, A.
Acknowledgments This work was supported by the Portuguese FCT. The authors thankfully acknowledge computer resources, technical expertise, and assistance provided by the Barcelona Supercomputing Center - Centro Nacional de Supercomputaci´ on.
References 1. Acebr´ on, J.A., Busico, M.P., Lanucara, P., Spigler, R.: Domain decomposition solution of elliptic boundary-value problems. SIAM J. Sci. Comput. 27(2), 440– 457 (2005) 2. Acebr´ on, J.A., Busico, M.P., Lanucara, P., Spigler, R.: Probabilistically induced domain decomposition methods for elliptic boundary-value problems. J. Comput. Phys. 210(2), 421–438 (2005) 3. Acebr´ on, J.A., Spigler, R.: Supercomputing applications to the numerical modeling of industrial and applied mathematics problems. J. Supercomputing 40, 67–80 (2007) 4. Acebr´ on, J.A., Spigler, R.: A fully scalable parallel algorithm for solving elliptic partial differential equations. In: Kermarrec, A.-M., Boug´e, L., Priol, T. (eds.) Euro-Par 2007. LNCS, vol. 4641, pp. 727–736. Springer, Heidelberg (2007) 5. Acebr´ on, J.A., Rodr´ıguez-Rozas, A., Spigler, R.: Efficient parallel solution of nonlinear parabolic partial differential equations by a probabilistic domain decomposition (2009) (submitted) 6. Acebr´ on, J.A., Rodr´ıguez-Rozas, A., Spigler, R.: Domain decomposition solution of nonlinear two-dimensional parabolic problems by random trees. J. Comput. Phys. 228(15), 5574–5591 (2009) 7. Antia, H.M.: Numerical methods for scientists and engineers. Tata McGraw-Hill, New Delhi (1995) 8. Arbenz, P., Cleary, A., Dongarra, J., Hegland, M.: A comparison of parallel solvers for diagonally dominant and general narrow-banded linear systems. Parallel and Distributed Computing Practices 2(4), 385–400 (1999) 9. Arbenz, P., Cleary, A., Dongarra, J., Hegland, M.: A comparison of parallel solvers for diagonally dominant and general narrow-banded linear systems II. In: EuroPar 1999 Parallel Processing, pp. 1078–1087. Springer, Berlin (1999) 10. Keyes, D.E.: Domain Decomposition Methods in the Mainstream of Computational Science. In: Proceedings of the 14th International Conference on Domain Decomposition Methods, pp. 79–93. UNAM Press, Mexico City (2003) 11. McKean, H.P.: Application of Brownian motion to the equation of KolmogorovPetrovskii-Piskunov. Comm. on Pure and Appl. Math. 28, 323–331 (1975) 12. Quarteroni, A., Valli, A.: Domain Decomposition Methods for Partial Differential Equations. Oxford Science Publications, Clarendon Press, Oxford (1999)
Partial Data Replication as a Strategy for Parallel Computing of the Multilevel Discrete Wavelet Transform Liesner Acevedo1,2 , Victor M. Garcia1 , Antonio M. Vidal1 , and Pedro Alonso1 1
Departamento de Sistemas Inform´ aticos y Computaci´ on Universidad Polit´ecnica de Valencia Camino de Vera s/n, 46022 Valencia, Espa˜ na {lacevedo,vmgarcia,avidal,palonso}@dsic.upv.es 2 Departamento de T´ecnicas de Programaci´ on Universidad de las Ciencias Informaticas Carretera a San Antonio de los Ba˜ nos Km 2 s/n, La Habana, Cuba
Abstract. In this paper we propose a strategy of partial data replication for efficient parallel computing of the Discrete Wavelet Transform in a distributed memory environment. The key is to avoid the communications needed between computation of different wavelet levels, by replicating part of the data and part of the computations, avoiding completely communications (except maybe at the setup phase). A similar idea was proposed in a paper by Chaver et al.; however, they proposed to replicate completely the data, which can require too much memory in each processor. In this work we have determined exactly how many data items shall be needed for each processor, in order to compute the DWT without extra communications and using only the memory strictly necessary.
1
Introduction
The Discrete Wavelet Transform (DWT) [1,2,3] is still a very recent field, so that it is the subject of active research. The number of applications which can benefit from this tool increases nearly every day, such as signal analysis [4], image compression [5], electromagnetism [6] and many others. There is also a growing interest in the use of the DWT in Numerical Linear algebra [7,8]. The DWT is a relatively simple operation that is applied to signals and data of any kind, and amounts to perform convolutions of certain digital filters with the data. It can be used with different goals: to obtain compressed representations of the data, to determine the frequency contents of the data, etc. Usually, the DWT is applied repeatedly over the data set; each new application of the DWT is called a level. Very often, several levels of the DWT are applied over very large data sets. This requires heavy computations and, in such cases, parallel computing may be needed to obtain reasonable computing times. There are many papers about the parallel computing of the DWT. Among these, we find specially relevant the paper by Nielsen and Hegland [9]. There, R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 51–60, 2010. c Springer-Verlag Berlin Heidelberg 2010
52
L. Acevedo et al.
they made a detailed study of the parallel computing of the DWT for vectors (DWT-1D) and for matrices (DWT-2D) in distributed memory computers. Their study was focused mainly in the 1-level case, establishing clearly the communications needed in this case. To compute the 1-level DWT, each processor should receive a portion of the data (which, for efficiency, should be composed of contiguous entries of the vector/matrix) and should compute its part of the DWT; to complete the calculation, each processor needs to receive some data from the “neighbor”processors. They also studied the 1-level 2D DWT, which involves as well a matrix (image) transposition. They proposed a strategy (The Communication-Efficient Distribution) to avoid the transposition step. Still, this is accomplished at the cost of some communications among processors. The k-level case can be obtained applying repeatedly the 1-level DWT, but as will be shown, new communications must be carried out prior to the computation of each level. In [10] a full replication strategy was proposed for the k-level DWT of a matrix. This amounts to copy the full matrix in every processor, splitting the computations among the processors. In this case, no communication would be needed (although some computations should be replicated as well). The main drawback of this strategy is that it requires a full copy of the matrix in each processor, so that there is a severe limit on the size of the matrices that can be processed in this way. In this paper, we have refined further the idea proposed in [10], determining exactly how many elements needs to hold each processor so that the parallel DWT can be computed without extra communications. We have obtained an exact result for this problem, proven in Proposition 2. The result was proven for the 1D case, but, since any higher dimension DWT can be computed as a sequence of 1D DWTs, our result can be used for DWTs of any dimension. The paper is structured as follows: Section 2 describes the DWT. The next Section describes the Parallel DWT, mainly for the 1D case, and includes our proposal, whose main result is the Proposition 2. Then, we discuss several considerations about the possible practical use of our proposal, and finally we give our conclusions.
2
Discrete Wavelet Transform
The theoretical background of the DWT is based on the concept of Multiresolution Analysis and is already quite well known. However, these concepts are not useful when considering the implementation details and the parallelization, so that we will refer to the interested reader to [2,11]. The DWT applied over a signal or vector is obtained through digital filters, L a lowpass filter G determined by the coefficients {gi }i=1 and a highpass filter H L determined by {hi }i=1 . Not any pair of highpass and lowpass filters are valid for a DWT; however, there are many such pairs. This means that the DWT is not a single, unique
Partial Data Replication as a Strategy for Parallel Computing
53
operation, since it depends on the filters. The length L of the filter is called the “order”of the wavelet. To obtain the DWT of a vector, both filters shall be applied (convolved) over the data vector; so, a single stage of the DWT (or a one-level DWT) over a vector v of length N = 2j is performed, first, convolving both filters with the vector v, obtaining two sequences of length N , and next, discarding every even element of the transformed sequences, and putting together the resulting sequences. These two steps are equivalent to the multiplication of the matrix WN = H by the data vector, where H (of dimension N2 × N ) has the following G structure: ⎛ ⎞ h1 h2 ... hL 0 · · · · · · · · · 0 ⎜ 0 0 h1 h2 ... hL 0 · · · 0 ⎟ ⎜ ⎟ (1) H=⎜ ⎟ . .. . . . . . . ⎝ 0 ··· ··· . . . . 0 ⎠ h3 h4 · · · hL 0 · · · 0 h1 h2 The matrix G has the same structure and dimension. The matrix-vector product would result in ⎛ ⎞ d1 ⎜ .. ⎟ ⎜ ⎟ ⎜ . ⎟ ⎜ H ·v dN/2 ⎟ ⎟ , WN · v = (2) =⎜ ⎜ ⎟ G·v ⎜ s1 ⎟ ⎜ . ⎟ ⎝ .. ⎠ sN/2 where the di are the high frequency (or “detail”) components and the si are the low frequency (or “smooth”) components. The k-Level DWT (or, simply, the DWT) is obtained applying k times the one-level DWT over the low frequency part of the data vector. The two-dimensional wavelet transform is applied to a matrix by applying 1-D transforms along each dimension. So, a one-level 2-D DWT applied over a matrix T0 ∈ n,n can be written as T˜0 = WN T0 WNT , with WN defined as above. In this case, we obtain the following partition: A1 B1 T˜0 = WN T0 WNT = , (3) C1 T1 where T1 = GT0 GT , A1 = HT0 H T , B1 = HT0 GT and C1 = GT0 H T . If, as usually happens, we want to apply more than one DWT level, at least two different possibilities appear. The so-called “Standard”DWT form of a matrix is obtained applying the k-level 1D-DWT over the columns, and then applying the k-level 1D DWT over the rows. The Non-Standard form [3] is obtained applying a one level DWT over the rows, and one over the columns (obtaining a partition T ) and another DWT by as in (3)). Then, another one-level DWT by rows (WN/2 columns (WN/2 ) are applied over the low-frequency submatrix T1 , and this is repeated k times, always over the low frequency submatrix.
54
3
L. Acevedo et al.
Data Distributions for the Parallel DWT
The parallel DWT is a relatively simple operation to implement in a distributed memory model (it becomes trivial in a shared memory model) [9]. Examining the 1D case, outlined above, it is clear that the DWT is just a matrix-vector product (although in a practical implementation the matrix Wn would never be built). The straightforward parallelization is just like in the matrix-vector product, distributing the data vector among the available processors in “chunks”of length N P , or as close as possible. Then, each processor performs its part of the DWT, although it will probably need data from other processors. Since the DWT is applied to contiguous components of vectors/matrices, a parallel DWT will be efficient only if the data assigned to any processor is made up of contiguous components of the vectors/matrices to be DWT-transformed. So, we will consider only this kind of “contiguity”data distributions. The parallel 2D DWT has been studied in different papers [9,12], but mainly for the single level case. In all cases, the distributions proposed were by blocks of columns, and the main efforts were oriented towards avoiding the transposition step. However, all the strategies assumed that there should be communications before the computation of each new level. In the next section we will discuss the procedure for parallel computing of a single DWT level. 3.1
Parallel Computation of the 1D-DWT, One Level
To understand how proceeds the computation of a single 1D-DWT level we can examine Fig. 1. This figure shows the data dependences in the computation of a vector of length 16 distributed across 2 processors using wavelet filters of length L = 4. Let us consider only the computation of the low frequency components (the high frequency components are computed exactly in the same way). To compute the low frequency components, the processor P0 computes the coefficients s10 , s12 , s13 , s14 , while the processor P1 computes the coefficients s15 , s16 , s17 , s18 . To carry out their tasks, the processor P0 needs to receive the values v9 and v10 to compute s14 while P1 needs to receive the values v1 and v2 . This idea was generalized by Nielsen and Hegland [9], who obtained the following basic result: Proposition 1. Consider the parallel computing of the 1D-DWT, applied over a vector of length N , and distributed over P processors (0, . . . , P − 1) in subvec. If the wavelet order (length of the filters) is L, to complete the tors of length N P computation of the 1D-DWT over the vector v, each processor must receive exactly L − 2 elements of the processor “neighbor”from the right. (except for Pp−1 , that must receive L − 2 elements from P0 ) Now the k-level DWT shall be discussed, including our proposal.
Partial Data Replication as a Strategy for Parallel Computing
55
Fig. 1. A 1D-DWT level over a vector v distributed over 2 processsors, wavelet filters with L = 4 (4 coefficients)
3.2
Parallel Computation of the 1D-DWT, Several Levels
The computation of several 1D-DWT levels can be done applying repeatedly the results of the former section, that is, prior to the computation of each level each processor sends and receives (L − 2) elements. Then, each processor computes its part of a new DWT level.
Fig. 2. Two 1D-DWT level over a vector v distributed over 2 processsors, wavelet filters with L = 4 (4 coefficients)
Figure 2 shows the data dependencies for the same case of Fig. 1. For two levels, the dependencies show that each processor (in this case) should receive 6 extra elements of the original vector to complete its task. It is clear that after each level there must be new communications, and the size of these communications grows with each new level. To avoid all these communications, in [10] it was proposed to replicate all the data in all the processors, so that instead of using communications each processor can compute the new elements that it needs. (This idea was proposed for the 2D-DWT of a matrix, but the same idea can be used for any dimension).
56
L. Acevedo et al.
In this way, all the communications (after the first distribution of the data) can be avoided. The extra computations are much less time consuming than the communications that it replaces. We call this a “full replication”technique. However, this means that all the processors must hold the full data. For very large sets of data, this may become a serious problem. Our proposal in this paper is to refine this idea, determining exactly how many elements will need each processor to complete its part of a k-level DWT without further communications, so that (hopefully) the memory needed by each processor is much less than with the “full replication”technique; we call our idea the “partial replication”technique. Considering a vector distributed over a set of processors, we want to determine the number NL,k of elements of the original vector (apart from those originally in the processor) that each processor will need to complete the k-level DWT using wavelet filters of length L. To prove our main result, we need first the following lemma: Lemma 1. Consider the parallel computing of k levels of the 1D-DWT, applied over a vector of length N , and distributed over P processors (0, . . . , P − 1) in . Let us assume that each processor has computed already subvectors of length N P the k-level DWT of its subvector (and, to do that, had already received all the NL,k elements needed from the original vector). For each processor to compute M additional elements of the k-level DWT without further communications, it must receive 2k M additional elements from the original vector Proof. Let us prove the lemma for k = 1 (one level). Let v be the original vector, and vdwt the vector transformed by a single DWT level. Let us consider a single processor P , which initially holds the subvector of v contained between the indexes [iini , iend ], and that has already computed its part of the vdwt vector, between the indexes [iwini , iwend ]. For this, processor P has received from another processor L − 2 extra elements, from v[iend + 1] to v[iend + L − 2]. It is desired that this processor computes as well some of the elements of vdwt “to the right”of those already computed: vdwt [iwend + 1] ,vdwt [iwend + 2], ... . We need to know how many extra elements of v must hold the processor P to compute these extra elements of the transformed vector vdwt . Starting by the element vdwt [iwend +1] , to compute this element the processor P would need the elements v[iend + 1],v[iend + 2], up to v[iend + L]. (Recall that the length of the filter is L). However, the first L − 2 elements of these had already been sent to P , so that the only new elements needed are v[iend + L − 1] and v[iend + L]. For the next element to be computed, vdwt [iwend + 2] , and taking the downsampling into account, the processor P needs v[iend + 3],v[iend + 4], up to v[iend + L + 2]. Again, it is two more elements from v to compute a new one of vdwt . It should be clear now that: For the processor P to compute M additional elements of the transformed vector vdwt it needs 2M additional elements from the original vector v.
Partial Data Replication as a Strategy for Parallel Computing
57
This has been proved for a single DWT level. However, the reasoning is exactly the same to go from level k − 1 to level k. Let us consider the case put forward in the lemma 1, when each processor has computed its part of the k-level DWT, and it is needed to compute M extra elements of the k-level DWT, located, as before, “to the right”of the already computed elements. Using the result just obtained, to compute these new M elements the processor needs 2M elements of the k-1 DWT vector; to compute these elements, 4M elements of the k-2 DWT will be needed. Repeating the process, we obtain that to compute the M elements of the k level DWT, 2k M elements of the original vector (level 0) will be needed, which proves the lemma. We can examine again Fig. 1 to check the lemma. We see that, to compute its part of the first level, the processor P0 had received the elements v9 and v10 . If we want that P0 computes as well s15 , it needs to have access to v9 , v10 , v11 and v12 ; but since v9 and v10 were already available, it only needs v11 and v12 . To compute also s16 , it would need another two elements (v13 and v14 ), and so on. Now, the number of elements that must be sent to each processor is easily obtained through the following proposition: Proposition 2. Consider the parallel computing of k levels of the 1D-DWT, using wavelet filters of length L, applied over a vector of length N , and distributed . To complete the task over P processors (0, . . . , P − 1) in subvectors of length N P without further communications, each processor should receive
(4) NL,k = 2k − 1 · (L − 2) additional elements from the original vector v, located “to the right”of those already available to processor P . Proof. We will prove this proposition using induction over the number of levels k. Let us consider first the base case, with k = 1. The base case is already proven using Proposition 1, where it states that the number of additional elements that each processor needs is
(5) L − 2 = NL,1 = 21 − 1 · (L − 2) . Let us suppose that (4) is true for k levels:
NL,k = 2k − 1 · (L − 2) ;
(6)
we will prove that the result is true for k + 1 levels:
NL,k+1 = 2k+1 − 1 · (L − 2) .
(7)
To compute the (k+1) DWT level, each processor must have computed previously the k-th level. Therefore, by
the induction hypothesis, each processor must have received NL,k = 2k − 1 · (L − 2) additional elements from the original vector.
58
L. Acevedo et al.
Apart from these NL,k elements from the original vector, each processor must receive some extra elements to complete the computation of the k+1 DWT level. These elements are easily determined, since the computation of the k+1 DWT level amounts to apply a 1 level DWT over the k-level DWT vector. So, using the base case, each processor must receive L − 2 additional elements from the k-level DWT vector. Therefore, to complete
the k+1 level DWT, each processor needs to have available NL,k = 2k − 1 · (L − 2) elements of the original vector, plus L − 2 additional elements from neighbor processor corresponding the k-level DWT vector. However, we want that each processor completes its task receiving only elements of the original vector. Using the lemma, we obtain that to compute these L − 2 elements from the k-level DWT vector, 2k · (L − 2) elements of the original vector v are needed. Then, we obtain easily that the number of elements of the original vector needed is:
k (8) 2 − 1 (L − 2) + 2k · (L − 2) = 2k+1 − 1 · (L − 2) . 3.3
Practical Use of Proposition 2
The practical interest of Proposition 2 resides in the elimination of communications when computing the parallel DWT in a distributed memory environment. Consider the following example; let us suppose that we want to compute the 8-level DWT of a vector of length, say 107 elements, distributed among 10 processors (P0 ,...,P9 ), using wavelet filters of length 4. Then, each processor must compute the 8-level DWT of a subvector of size 106 , but it needs some elements of the processor located ”to its right” (except the processsor P9 which will receive them from processor P0 ). There are three different procedures: 1. Repeated use of the 1-level DWT and of Proposition 1. Then, in the initial distribution each processor should receive 106 elements, plus another 2 from the processor holding the elements to the right. Then, after the computation of each level, each processor must receive another 2 elements (from the former level), so that there must be 8 collective communications, one per level. 2. Use of a full replication strategy. Then, no extra communication would be needed, but every processor needs to hold the full vector of 107 elements. This would be, somehow, equivalent to compute the 8-level DWT in a shared memory multiprocessor. 3. Partial replication strategy, using Proposition 2. In this case, each processor will need its subvector of 106 elements, plus (applying Proposition 2) (27 − 1) ∗ 2 = 254 extra elements in the initial distribution; with these extra elements (and performing the extra computations associated to these points) all the processors will be able to complete the 8-level DWT without extra communications. If the filter is longer (say, L=30), then (27 − 1) ∗ (30 − 2) = 3556; clearly, the number of extra elements to be initially sent to each processor is substantially
Partial Data Replication as a Strategy for Parallel Computing
59
higher (and the number of computations for these extra points). However, since all the communications after the first one are no longer needed, it is still probably a faster strategy. Other considerations about the practical use of Proposition 2: -This proposition has been proved for the 1D parallel DWT, but it can be used similarly for the DWT-2D or for higher dimension DWTs. For example, Fig. 3 displays a matrix distributed by blocks, where a certain processor holds the block A[r1 :r2 ,c1 :c2 ] . To compute two
DWT levels with L = 4 of that block, for each row the processor needs 22 − 1 · (4 − 2) = 6 extra elements on the right, and the same with the columns.
Fig. 3. Two 2D-DWT levels over a matrix wavelet filters with L = 4 (4 coefficients)
Some care must be taken to include the “diagonal”blocks, since these will be necessary too. - Even in the case when the number of elements given by the equation (4) is very high, still it would be possible to use (4) with a different approach. In such case, we might use (4) to perform communications only at selected levels. As an example, let us suppose that we want to compute 10 DWT levels using a filter with L = 24 of a large vector. We might choose between sending 210 − 1 · (L − 2) = 22506 elements right from the beginning, (where no more should take place), or send elements enough for 5 levels:
5 communications 2 − 1 · (L − 2) = 682, compute the first 5 levels, send again enough elements (again 682) for the final 5 levels, and finally compute the last 5 levels. Depending on the communication parameters of the system, we might determine an optimal strategy for each case. 3.4
Conclusions
The work presented allows the parallel computation af several DWT levels without any communication (except, maybe, in the setup phase). This idea, was already presented in [10], but there it was proposed replicating all the data. In this paper we have proved that it can be easily computed the exact number of extra elements needed. Therefore, these elements can be sent to each processor at the beginning of the computations, avoiding any extra computation.
60
L. Acevedo et al.
The practical impact of this proposal will depend on the application and on properties of the parallel computer where it is implemented. However, it is usually acknowledged that avoiding communications at the expense of a few extra computations will, in most cases, improve the execution times and the speed-up of the parallel DWT algorithms.
Acknowledgements Partially Supported by the Vice-rectorate for Research, Development and Innovation of UPV (Project 20080009) by the Generalitat Valenciana (Project 20080811), and by Spanish MICINN (Project TIN 2008-06570-C04-02).
References 1. Mallat, S.G.: A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Machine Intell. 2, 674–693 (1989) 2. Daubechies, I.: Ten Lectures on Wavelets. In: CBMS-NSF Regional Conference Series In Applied Mathematics, vol. 61. Society for Industrial and Applied Mathematics, Philadelphia (1992) 3. Beylkin, G., Coifman, R.R., Rokhlin, V.: Fast wavelet transforms and numerical algorithms I. Commun. Pure Appl. Math. XLIV, 141–183 (1991) 4. Chui, C.K.: Wavelets: A Mathematical Tool for Signal Analysis. In: SIAM Monographs on Mathematical Modeling and Computation. SIAM, Philadelphia (1997) 5. Starck, J.L., Murtagh, F., Bijaoui, A.: Image Processing and Data Analysis. In: The Multiscale Approach. Cambridge University Press, Cambridge (1998) 6. Sarkar, T.K., Salazar-Palma, M., Wicks, M.C.: Wavelet Applications in Engineering Electromagnetics. Artech House, Norwood (2002) 7. Chan, T.F., Chen, K.: On two variants of an algebraic wavelet preconditioner. SIAM J. Sci. Comput. 24, 260–283 (2002) 8. Chen, K.: Matrix Preconditioning Techniques and Applications (2005) 9. Nielsen, O.M., Hegland, M.A.: A Scalable Parallel 2D Wavelet Transform Algorithm. REPORT TR-CS-97-21, Australian National University (1997), http://cs.anu.edu.au/techreports 10. Chaver, D., Prieto, M., Pi˜ nuel, L., Tirado, F.: Parallel Wavelet for Large Scale Image Processing. In: Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IEEE IPDPS 2002), Florida, USA (April 2002) 11. Wickerhauser, M.V.: Adapted Wavelet Analysis from Theory to software. A.K. Peters, IEEE Press (1994) 12. Marino, F., Piuri, V., Swartzlander Jr., E.E.: A parallel implementation of the 2-D discrete wavelet transform without interprocessor communications. IEEE Transactions on Signal Processing 47(11), 3179–3184 (1999)
Dynamic Load Balancing for Adaptive Parallel Flow Problems Stanislaw Gepner, Jerzy Majewski, and Jacek Rokicki Warsaw University of Technology The Institute of Aeronautics and Applied Mechanics Nowowiejska 24, 00-665 Warsaw, Poland
[email protected],
[email protected],
[email protected] http://c-cfd.meil.pw.edu.pl
Abstract. Large scale computing requires parallelization in order to arrive at solution at reasonable time. Today parallelization is a standard in fluid problems simulation. On the other hand adaptation is a technique that allows for dynamic modification of the mesh as the need for locally higher resolution arises. Adaptation used during parallel simulation leads to unbalanced numerical load. This in turn decreases the efficiency of parallelization. Dynamic load balancing strategies should be applied in order to ensure proper parallelization efficiency. The paper presents the potential benefits of applying the dynamic load balancing to adaptive flow problems simulated in parallel environments. Keywords: dynamic load balancing, parallel cfd, adaptation, mesh refinement.
1
Introduction
The parallelization is the only effective approach to deal with modern scientific or engineering flow simulations. A common approach to parallelization is the one based on the domain decomposition. As it is a natural and relatively easy to implement approach (See Fig. 1 for an example of domain decomposition applied to a complex geometry). For domain decomposition based parallelization to be effective the partitioning into sub domains must satisfy certain conditions. Firstly the computational load assigned to sub domains must be equally distributed. Usually the computational cost is proportional to the amount of mesh entities assigned to partitions. So to ensure numerical load to be balanced, it is enough to equally divide mesh elements among partitions. Good quality partitioning requires also the volume of communication during calculation to be kept at its minimum. This can only be achieved by examining the topology of the mesh being partitioned, and minimizing the resulting edge cuts. For static applications where numerical load distribution and topology does not change during computation it is enough to perform partitioning only once, prior to the computation. Such use is often referred to as static load balancing. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 61–69, 2010. c Springer-Verlag Berlin Heidelberg 2010
62
S. Gepner, J. Majewski, and J. Rokicki
Fig. 1. Mesh of a fighter plane partitioned prior to computation. CoolFluid project test-case [4].
When computing for a certain flow features, such as shocks a technique of adaptation is an effective tool to limit the number of necessary cells [2]. It allows for high grid resolution in areas of importance, while applying a relatively coarse mesh in other regions. Thus lowering the overall numerical cost, while keeping the solution properly resolved. Figure 2 presents an example of adaptation used to capture a complex shock wave structure for Onera M6 wing geometry, together with the solution adapted mesh.
Fig. 2. Adapted surface mesh for Onera M6 wing geometry (left), and the resulting density field (right)
The use of adaptation during a parallel simulation leads to the numerical load change throughout the domain, making the initial load balancing ineffective. If
Dynamic Load Balancing for Adaptive Parallel Flow Problems
63
one wants to maintain a proper parallel efficiency a dynamic load balancing scheme has to be introduced. In addition to the condition fulfilled by a static mesh partitioner, a dynamic one has to satisfy some additional criteria. It should work well in parallel, be efficient in memory consumption and have a short runtime as not to negatively influence the overall performance. An important feature of a partitioner is its incrementality, which means that data migration resulting from repartition will be kept to a necessary minimum. The problem of dynamic balancing of numerical load was considered in [5,6,7,8,9,10,11], the present approach concentrates on evaluation of benefits of DLB in aerodynamically motivated applications. This paper presents results from implementation of the DLB algorithm into the in-house FLOW code [2] a parallel residual distribution flow solver.
2
The Measure of Partition Quality
To estimate the quality of load balance, let W be a total numerical load, equal to the total number of mesh entities. If p stands for the number of processor cores used, than the load considered optimal and assigned to an it’h processor would be: wiopt = W/p. The actual load assigned is wi , so a load balance indicator is assumed in the following way: β = max
0
wi wiopt
(1)
If defined in this way, the partitioning is of acceptable quality when load balance indicator is close to one. This means that the numerical load assigned to the most heavily loaded partition is close to the optimal one.
3
Parallel Mesh Refinement
In general adaptation for three dimensional complex geometries is a non trivial problem [2,3]. It is further so if it is to be performed in parallel. The following isotropic mesh refinement algorithm has been implemented into the FLOW solver: 1. An error indicator estimates error connected to each of the mesh cells (at the particular moment of the simulation iterative process). 2. All cells exceeding a chosen error threshold are marked for refinement. 3. Nodes are introduced on the edges of the marked cells. 4. New cells are created, and unnecessary cells are removed from the data structure. 5. New solution process is restarted at the adapted mesh. 6. The simulation is returned to point one, if the appropriate convergence criteria is not met.
64
S. Gepner, J. Majewski, and J. Rokicki
Fig. 3. Mesh refinement. Cells 1 and 2 have been marked to be refined, new nodes have been added to them.
The adaptation process is schematically illustrated in Fig. 3. When working in parallel there has to be additional step taken to ensure coherence in the regions of overlap. Fig. 4 presents mesh configuration resulting from a series of consecutive refinements.
Fig. 4. Computational mesh after fourth refinement steps
4
Repartitioning Phase
During the repartitioning phase a new, better balanced mesh decomposition is created. It is accomplished using one of many available partitioning tools. Subsequently the mesh entities are redistributed between processors according to the obtained coloring. 4.1
Mesh Partitioning Tool Graph Partitioner
To calculate mesh decomposition an outside tool is used. In this work the ParMETIS [1] graph partitioner was used. It is a parallel version of a well known METIS package. It is based on the multilevel approach, and is capable of computing graph partitions of good quality. Main factor in it’s selection was its parallel efficiency and ease of use.
Dynamic Load Balancing for Adaptive Parallel Flow Problems
65
Fig. 5. Partitions are being modified according to the previously calculated coloring. On the left is the original partitioning and coloring, the middle picture shows original partitioning with new coloring. On the right is the partitioning after elements have been sent / deleted.
4.2
Mesh Partitioning
To perform dynamic load balancing procedure, the following steps have to be undertaken: 1. Elements of the mesh being repartitioned have to be given unique numbering throughout the whole partitioned domain. This makes mesh elements easily distinguishable throughout the parallel environment. 2. Grid topology has to be described as a connectivity graph. When working with ParMETIS the so called distributed CSR format is used. 3. With graph connectivity ready it is possible for ParMETIS to calculate new mesh partitioning. 4. According to the calculated partitioning there are some data movement to be performed. During this phase mesh elements are selected for communication or deletion appropriately. This step is strongly dependent on the data structure used by the simulation software. Applying the new partitioning involves massive changes to the simulation data structure. Some mesh entities have to be send to other computing nodes while Table 1. Comparison of repartition time, graph partitioning time, average iteration time; case of 5078927 nodes No of cores 32 72 80
Simulation time 311500 s 157500 s 129500 s
Total repartition time 663 s 165 s 128 s
Partitioner Run time 5s 2s 3s
66
S. Gepner, J. Majewski, and J. Rokicki
Fig. 6. Impact of numerical imbalance on computation time. Meshes used for comparison, the convergence history for both cases. In the balanced case convergence time is reduced.
some must be removed. Also the received data have to be added to the data structure accessible to a given CPU. It is a process strongly dependent on the data structure used within the solver. Once the information is exchanged the solver is ready to restart calculations.
Dynamic Load Balancing for Adaptive Parallel Flow Problems
67
Fig. 7. Mesh after consecutive refinement. Dynamic load balancing has been applied on the left. The number next to the mesh denotes its load balance indicator (LBI).
Fig. 8. Convergence as a function of computational time, with and without load balancing used
68
S. Gepner, J. Majewski, and J. Rokicki
This process is illustrated on Fig. 5 where a three dimensional domain is being repartitioned. On the left shown is the initial partitioning. In the middle presented is the new coloring calculated during the graph partitioning phase plotted onto original partitioning. The rightmost picture shows the mesh after appropriate data migration has been carried out.
5
Impact of Dynamic Load Balancing on Runtime
The aim of using dynamic load balancing during parallel simulation is to ensure its proper parallel efficiency to be maintained during runtime. Figure 6 shows how computational time is influenced by the dis-balanced numerical load. The case for which load balance indicator is close to 2 (Fig. 6a) needs more time to converge than in the case of properly balanced one. Figure 7 presents the history of computational mesh after consecutive refinement steps, used during parallel flow simulation. On the right hand side is the case where load balancing has not been used. On the left side presented is the case with load balancing active. Next to each of the presented meshes noted is its load balance indicator. Graph on Fig. 8 shows convergence history for both of the cases respectively. The case for which load balancing is used has a load balance indicator close to one throughout the simulation process. While for the situation without load balancing, the LBI measure is increasing together with adaptation steps taken. This has an influence on convergence time. Numerical cost of rebalancing is relatively small in comparison to total simulation time. Table 1 shows comparison of CPU time required to perform the simulation. Compared is the time of a single iteration, time required to perform graph partitioning (ParMETIS run time) and a total time required to perform a full rebalancing step. It is worth noticing that total rebalancing time decreases larger the number of CPU’s. The case under investigation is a three dimensional simulation of the flow around Oner M6 wing. Mesh of 5078927 nodes (32622210 cells) is used. Results ware obtained by running the FLOW solver on the cluster containing 20 QuadCore AMD Opteron(tm) processors (4 cores per processor) with 2 GB of memory per core, connected by Fast Ethernet connection.
6
Conclusions and Further Work
The paper presented dynamic load balancing algorithm which is an effective tool for an adaptive engineering application run in parallel. It was shown that it has significant impact on the runtime and overall efficiency of a simulation. The rebalancing step itself is relatively cheap numerically even when using larger number of computing nodes.
Dynamic Load Balancing for Adaptive Parallel Flow Problems
69
Acknowledgments This work was supported by the FP6 EU ADIGMA project (Adaptive HigherOrder Variational Methods for Aerodynamic Applications in Industry), and by MNiSW, grant No: N N516 4280 33, Dynamic load balancing for adaptive engineering applications in parallel environments, 2007-2009.
References 1. Karypis, G., Schloegel, K., Kumar, V.: PARMETIS* Parallel Graph Partitioning and Sparse Matrix Ordering Library 2. Majewski, J.: Anisotropic solution-adaptive technique applied to simulation of steady and unsteady compressible flows. In: Proceedings of the International Conference on Computational Fluid Dynamics, Ghent, Belgium (2006) 3. Majewski, J.: Ph. D. Thesis Grid generation for flow problems in complex geometries (Rozprawa Doktorska, Generacja siatek dla symulacji przepyww w obszarach zoonej geometrii.) Warsaw University of Technology (2001) (in Polish) 4. COOLFluid project home page, http://coolfluidsrv.vki.ac.be/trac/coolfluid 5. Cybenko, G.: Dynamic load balancing for distributed memory multiprocessors. J. Parallel Distrib. Comput. 7, 279–301 (1989) 6. Devine, K., Boman, E., Heaphy, R., Hendrickson, B., Teresco, J., et al.: New challenges in dynamic load balancing. Applied Numerical Mathematics 52(2-3), 133– 152 (2005) 7. Touheed, N., Selwood, P., Jimack, P.K., Berzins, M.: A comparison of some dynamic load-balancing algorithms for a parallel adaptive flow solver. Parallel Computing 26(12), 1535–1554 (2000) 8. Karypis, G., Schloegel, K., Kumar, V.: Dynamic Repartitioning of Adaptively Refined Meshes. Department of Computer Science and Engineering (1998) 9. Schloegel, K., Karypis, G., Kumar, V.: Multilevel Diffusion Schemes for Repartitioning of Adaptive Meshes. Journal of Parallel and Distributed Computing 47(2), 109–124 (1997) 10. Walshaw, M., Cross, M.G.: Everett: Parallel dynamic graph partitioning for adaptive unstructured meshes. J. Parallel Distrib. Comput. 47, 102–108 (1997) 11. Hendrickson, B., Devine, K.: Dynamic load balancing in computational mechanics. Comp. Meth. AppL Mech. Engrg. 184, 485–500 (2000)
A Balancing Domain Decomposition Method for a Discretization of a Plate Problem on Nonmatching Grids Leszek Marcinkowski Faculty of Mathematics, Informatics, and Mechanics, University of Warsaw, Banacha 2, 02-097 Warszawa, Poland
[email protected]
Abstract. In this paper we present a balancing domain decomposition method for solving a discretization of a plate problem on nonmatching grids in 2D. The local discretizations are a Hsieh-Clough-Tocher macro finite elements. On the interfaces between adjacent subdomains two mortar conditions are imposed. The condition number of the preconditioned problem is almost optimal i.e. it is bounded poly-logarithmically with respect to the mesh parameters.
1
Introduction
Recently, there has been a large number of investigations devoted to domain decomposition methods for discretizations on nonmatching grids. Most of the methods for the discretizations of partial differential equations are on one global grid, however there is a growing interest in the discretizations with nonmatching local triangulations. These types of approach are important among others for mesh generations, problems with local singularities, contact problems, multiphysics simulations, and hp-adaptive methods. In our paper we consider a mortar element method which is based on coupling between nonmatching meshes between adjacent substructures by imposing integral conditions on the traces of finite element functions from these subdomains on the subdomain interface. For more details we refer to [1]. In subdomain we use a Hsieh-Clough-Tocher (HCT) macro elements. There many effective iterative solvers for the mortar discretization of second order problems, cf. e.g. [2], [3], [4], [5], [6] and references therein, but there is a more limited number of investigations devoted to nonconforming discretizations of plate problems, see [7], [8], [9], [10]. Balancing domain decomposition methods form a class of fast scalable parallel solvers for systems of equations arising from different discretizations of many types of problems, cf. e.g. [11], [12], [13], [14] and references therein. However, to our knowledge there are no results for a BDD type of method for a discretization of a fourth order problem built on nonmatching grids.
This work was partially supported by Polish Scientific Grant: N N201 0069 33.
R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 70–79, 2010. c Springer-Verlag Berlin Heidelberg 2010
Balancing DDM for a Plate Problem
71
In our paper we first introduce a mortar discretization of a model clamped plate problem, cf. [15], which in subdomains utilize HCT macro finite element. Then to construct our method, which is of a substructuring type, the local variables interior to subdomains have to be eliminated and a new problem is formulated. We construct our balancing domain decomposition preconditioner using an abstract Additive Schwarz Method (ASM) scheme, cf. [16], [12]. The resulting preconditioner is almost fully parallel as usually in the balancing domain decomposition methods. Finally, we present almost optimal convergence estimates and discuss a few implementation issues. The remainder of the paper is organized as follows: in Section 2 we present a discrete problem, in Section 3 we construct our BDD method and give convergence estimates, and Section 4 is devoted to implementation.
2
Discrete Problem
Let Ω be a polygonal domain in the plane. Our model clamped plate problem is of the form: Find u∗ ∈ H02 (Ω) such that a(u∗ , v) = f v dx ∀v ∈ H02 (Ω),
(1)
Ω
where
a(u, v) = Ω
[uv + β(2ux1 x2 vx1 x2 − ux1 x1 vx1 x1 − ux2 x2 vx2 x2 )] dx,
u∗ is the displacement, f ∈ L2 (Ω) is the body force, and the constant β ∈ (1/2, 1). Here we use the notation: H02 (Ω) = {v ∈ H 2 (Ω) : v = ∂n v = 0 on ∂Ω}, 2
u ∂n is the normal unit derivative outer to ∂Ω, and uxi xj = ∂x∂i ∂x for i, j = 1, 2. j This problem has a unique solution what follows from the Lax-Millgram theorem. We assume that our domain Ω is decomposed into non-overlapping triangular subdomains, that form a shape regular coarse triangulation i.e.
Ω=
N
Ωk
k=1
and Ω k ∩ Ω l is a common vertex, an edge or the empty set, cf. e.g. [17]. Let introduce Hk = diam(Ωk ) and let H = maxk Hk be a parameter of this coarse triangulation. We introduce in an each substructure Ωk a triangulation made of triangles Th (Ωk ) which is quasi-uniform with a parameter hk = maxτ ∈Th (Ωk ) diam (τ ), cf. [17].
72
L. Marcinkowski
Let the local finite element space Xh (Ωk ) be defined as, cf. Figure 1, a subspace of C 1 (Ω k )∩HC2 (Ωk ) formed by all functions which are in P3 (τi ) for each of three triangles τi ⊂ τ, i = 1, 2, 3 formed by connecting the vertices of τ ∈ Th (Ωk ) to its centroid. Here HC2 (Ωk ) = {v ∈ H 2 (Ωk ) : v = ∂n v = 0 on ∂Ωk ∩ ∂Ω}.
Fig. 1. HCT macro element
The degrees of freedom of the HCT element are the following {v(p), vx1 (p), vx2 (p), ∂n v(m)} , where p is a vertex of an element and m the midpoint of an edge of an element, cf. Figure 1. We introduce an auxiliary global spaces Xh (Ω) as Xh (Ω) =
N
Xh (Ωk ).
k=1
An important role is played by the global interface Γ = ∂Ωk \ ∂Ω. k
Note that each interface which is a common edge to two adjacent subdomains: Γkl = ∂Ωk ∩ ∂Ωl ⊂ Γ inherits two 1D triangulations: one from Ωk - Ti,h (Γij ), and one from Ωl - Tj,h (Γij ), cf. Figure 2. We choose one side as a master (mortar) denoted by γkl ⊂ ∂Ωk while the second one as a slave (nonmortar) δlk ⊂ ∂Ωl . Here, the choice of the master side is arbitrary.
Balancing DDM for a Plate Problem
73
Let Mthl (δlk ) be an auxiliary space defined on δlk as the space of C 1 -smooth functions which are piecewise cubic over each element of the hl triangulation of δlk except of two elements which touch the ends of δlk , where the functions are piecewise linear. We also define a space Mnhl (δm,l ) as the space of continuous functions which are piecewise quadratic over each element of the hl triangulation of δlk and are piecewise linear on the two elements which touch the ends of δlk .
Γ ij
Ω
Ω
i
j
γ
δ
ij
ji
Fig. 2. Independent meshes on an interface
We can now define a discrete space V h as the subspace of Xh (Ω) of all function which are continuous at the cross-points (the common vertices of substructures), and satisfy the following two mortar condition on each interface Γij = δji = γij ⊂ Γ: (ui − uj )φ ds = 0 ∀φ ∈ Mth (δji ), (2) δji
δji
(∂n ui − ∂n uj )ψ ds = 0
∀ψ ∈ Mnh (δji ).
Note that a function u ∈ V h is continuous at a cross-point but in general ∇u is discontinuous there. Then we can introduce our discrete problem: Find u∗h ∈ V h such that ah (u∗h , v) = f v dx ∀v ∈ V h , (3) Ω
74
L. Marcinkowski
where ah (u, v) = ak (u, v) =
k
Ωk
ak (u, v) for
[uv + β(2ux1 x2 vx1 x2 − ux1 x1 vx1 x1 − ux2 x2 vx2 x2 )] dx.
and β, f are as in the differential problem (1). There is a unique solution to (3) and the error estimates are established, see [15].
3
Balancing Domain Decomposition
In this section we present a balancing domain decomposition method following the abstract scheme of [12], cf. also Chapters 4-6 in [16]. Our method is a substructuring one so first we decompose each function u ∈ Xh (Ωk ) into two parts: u = Hk u+Pk u, where Pk u is the ak orthogonal projection of u onto Xh,0 (Ωk ) = {u ∈ Xh (Ωk ) : u = ∂n u = 0 on ∂Ωk }, i.e. Pk u ∈ Xh,0 (Ωk ) and it satisfies ak (Pk u, v) = ak (u, v) ∀v ∈ Xh,0 (Ωk ), and Hk u = u − Pk u is the discrete biharmonic part of u uniquely defined by ∀v ∈ Xh,0 (Ωk ) ak (Hk u, v) = 0 (4) T r|∂Ωk Hk u = T r|∂Ωk u on ∂Ωk where T r|∂Ωk u = (u|∂Ωk , ∇u|∂Ωk ) are the traces of u onto ∂Ωk in the H 2 sense. We also introduce global discrete biharmonic part of any function u = (u1 , . . . , uN ) ∈ Xh (Ω) as follows Hu = (H1 u1 , . . . , HN uN ), and the local space of discrete biharmonic functions: Wk = Hk Xh (Ωk ), and the space of all discrete biharmonic function satisfying the mortar conditions: W = {u ∈ V h : u = Hu} ⊂ W1 × . . . × WN . Note that the discrete biharmonic function has a so called minimal energy property: ak (Hk u, Hk u) = min{ak (w, w) : T r|∂Ωk w = T r|∂Ωk Hk u = T r|∂Ωk u}. and that a discrete biharmonic function in Ωk is uniquely defined by its T r|∂Ωk , i.e. the values of degrees of freedom interior to this subdomain can be computed by solving (4). Thus a function in W is uniquely defined by the value of all degrees of freedom at nodes which are on masters (mortars) or which are vertices of subdomains. The degrees of freedom at nodes (vertices and midpoints) which are on slaves are determined by (2) and the ones at nodal points interior to subdomains by (4)
Balancing DDM for a Plate Problem
75
Next we can decompose the solution of the discrete problem (3) into two parts u∗h = P u∗h + u ˜∗h where P u∗h := (P1 u∗h , . . . , PN u∗h ) and u ˜∗h = Hu∗h is the discrete ∗ biharmonic part of uh . Note that P u∗h can be computed by solving N independent local problems, which can be done in parallel and u ˜∗h ∈ W is a unique solution of the following problem: u∗h , v) = ah (˜
f v dx Ω
∀v ∈ W.
(5)
We now introduce our balancing method using the abstract scheme of the additive Schwarz method (ASM), cf. [16]. We have to define a decomposition of W into subspaces and introduce local projections like operator onto these subspaces. We first introduce extensions operators. For each subdomain Ωi , an extension operator Ei : C 1 (Ω i ) → W is defined as follows: 1 u(cr ) for any cross-point cr which is a vertex of ∂Ωi \ ∂Ω, – Ei u(cr ) = N (c r) where N (cr ) is the number of domains which have cr as a vertex. – ∇Ei u(v) = ∇u(v) for any vertex v of substructure Ωi , which is not on ∂Ω, – Ei u(x) = u(x), ∇Ei u(x) = ∇u(x), and ∂n Ei u(m) = ∂n u(m) for any a nodal point (vertex) x and any midpoint m of an element of the hi triangulation of any mortar γij ⊂ ∂Ωi , – T r Ei u = 0 on remaining masters and at the cross-points which are not on ∂Ωi .
The values of the degrees of freedom of Ei u on a slave are defined by the mortar = i by (4) as Ei u ∈ W , i.e. is conditions (2) and interior to a subdomain Ωj , j discrete biharmonic. Note that for u = (u1 , . . . , uN ) ∈ W we have N
Ek uk = u,
k=1
we can say that the set of operators {Ek }N k=1 is a partition of unity. Next we introduce a coarse space: V0 = E1 P1 (Ω1 ) + E2 P1 (Ω2 ) + . . . + EN P1 (ΩN ) ⊂ W, where P1 (Ωk ) ⊂ Wk is a space of linear polynomials define over Ω k . This dimension of V0 is equal to 3 ∗ N minus the number of vertices of substructures which are on ∂Ω. Let P0 : W → V0 ⊂ W be the orthogonal projection onto V0 in terms of ah (u, v), i.e. P0 u ∈ V0 and ah (P0 u, v) = ah (u, v)
∀v ∈ V0 .
Next we introduce local spaces as follows Vk = (I − P0 )Ek Wk,0 ,
76
L. Marcinkowski
where Wk,0 is formed by all discrete biharmonic functions which has zero value at the three vertices of ∂Ωk . Next we introduce operators Tk : W → Vk defined as Tk u = (I − P0 )Ek Tˆk u for Tˆk u ∈ Wk,0 , which satisfies: ak (Tˆk u, v) = ah (u, (I − P0 )Ek v)
∀v ∈ Wk,0 .
(6)
Note that the operator Tk is properly defined as the problem (6) has a unique solution because the form ak (u, v) is Wk,0 elliptic. We also see that Tk is ah (·, ·) symmetric and nonnegative definite as ah ((I − P0 )Tk u, v) = ah ((I − P0 )Ek Tˆk u, v) = ak (Tˆk u, Tˆk v). Finally, we introduce the operator T : W → W as T = P0 +
N
Tk u
(7)
k=1
and we replace (5) by a new problem Tu ˜∗h = g,
(8)
where g = g0 + N ˜∗h and gk = Tk u ˜∗h . It follows from the next k=1 gk with g0 = P0 u theorem that the problems (8) and (5) are equivalent. It is important to note that g, g0 , gk can be computed without knowing u ˜∗h , see e.g. [16], cf. also Section 4. Theorem 1. For any u ∈ W , we have 2 H ah (u, u), c ah (u, u) ≤ ah (T u, u) ≤ C 1 + log h where H = maxk Hk with Hk = diam(Ωk ), h = mink hk , and c, C are positive constants independent of all mesh parameters hk , Hk . The proof of this theorem follows the lines of proofs in [18]. From this theorem it follows that if we apply a conjugate gradient iterative method for solving (8), then the number iterations is proportional to (1 + log(H/h)), i.e. grow logarithmically.
4
Implementation
Here we briefly describe an implementation of the method, cf. also [12], see Sections 3.2 and 3.3 there.
Balancing DDM for a Plate Problem
77
We first represent Tk and T in an operator form. Let ·, · and ·, · k are any inner products in W and Wk,0 , respectively, e.g. they may be respective L2 products in these spaces. We introduce the following operators: S : W → W , Sk,0 : Wk,0 → Wk,0 , EkT : W → Wk,0 by Su, v = ah (u, v) Sk,0 u, v k = ak (u, v)
T Ek u, v k = u, Ek v
∀v ∈ W, ∀v ∈ Wk,0 , ∀v ∈ Wk,0 .
Note that the operator Sk,0 is nonsingular. Then, from the definition of Tˆk , see (6), it follows that for any v ∈ Wk,0 we have
Sk,0 Tˆk u, v = ak (Tˆk u, v) = ah (u, (I − P0 )Ek v) k
Thus we get
= ah ((I − P0 )u, Ek v) = S(I − P0 )u, Ek v
= EkT S(I − P0 )u, v k .
−1 T Ek S(I − P0 ) Tˆk = Sk,0
k = 1, . . . , N,
what yields that −1 T Ek S(I − P0 ) Tk = (I − P0 )Ek Tˆk = (I − P0 )Ek Sk,0
k = 1, . . . , N.
Finally, the operator T can be represented as: N −1 T T = P0 + (I − P0 )( Ek Sk,0 Ek )S(I − P0 ).
(9)
k=1
Remark 1. If we introduce nodal basis in W, Wk and Wk,0 in a standard way, and take l2 inner product in this spaces then (9) presents a matrix representation of our balancing method. If we want to solve (8) by a CG or Richardson iteration method, then in each iteration we have to compute an action of T to a residual vector in each iteration. According to the above operator formula for T it requires to compute an action of P0 twice, but we now show how using a simple trick we can reduce this to one action of P0 . In real applications, to solve the problem (8), CG method is utilized, but for the simplicity of presentation, we restrict ourselves to the Richardson iterations. Let u0 be an arbitrary vector. Then in first step, we compute ˜∗h ) u1 = u0 − P0 (u0 − u that satisfies
P0 (u1 − u˜∗h ) = 0,
(10)
78
L. Marcinkowski
as P0 is an orthogonal projection, i.e. we get u1 − u ˜∗h orthogonal to V0 with respect to the bilinear form ah (u, v). We next show that all successive Richardson iterates un has this property, i.e. n ˜∗h is orthogonal to V0 . u −u The Richardson iteration is defined as follows: un+1 = un − τ T (un − u ˜∗h ) n = 1, 2, . . . where τ is a parameter chosen according to the spectral properties of T . Then by a simple induction argument, we get P0 (un+1 − u ˜∗h ) = P0 (un − u ˜∗h ) − τ P0 T (un − u ˜∗h ) = −τ P0 T (un − u˜∗h ) but we have P0 T = P0 P0 = P0 what results in
˜∗h ) = P0 (un − u ˜∗h ) = 0, P0 T (un − u
because P0 is an orthogonal projection operator. Thus we get that for Richardson iteration method that if u1 − u ˜∗h is orthogonal n ∗ to V0 then P0 (u − u ˜h ) = 0 as well. Hence our Richardson iterative method can be represented by u
n+1
n
= u − τ (I − P0 )
N k=1
−1 Ek Sk,0 (Ek )T S(un − u∗h )
n = 1, 2, . . .
(11)
Thus we have to apply P0 only once per iteration. Taking u1 such that P0 (u1 −u∗h ) = 0 yields that in each iteration of a Richardson we have to: – solve one global coarse problem to compute an action of P0 , – compute actions of Ek , EkT , – solve local independent problems with Sk,0 . Note that except computing the action of P0 , all the steps can be done in parallel. The same observation is true in the case of CG method, see e.g. Sections 3.2 and 3.3 in [12]. N −1 (Ei )T the Remark 2. Introducing s preconditioner B −1 = (I − P0 ) i=1 Ei Si,0 iteration method (11) can be interpreted a Richardson iteration with this preconditioner applied to solve the problem Su∗h = f˜
which is an operator form of (5). Here f˜ ∈ W satisfies: f˜, v = Ω f v dx for all v ∈ W.
Balancing DDM for a Plate Problem
79
References 1. Bernardi, C., Maday, Y., Patera, A.T.: A new nonconforming approach to domain decomposition: the mortar element method. In: Nonlinear partial differential equations and their applications. Coll`ege de France Seminar, vol. XI (Paris, 1989– 1991), Paris. Pitman Res. Notes Math. Ser., vol. 299, pp. 13–51. Longman Sci. Tech., Harlow (1994) 2. Bjørstad, P.E., Dryja, M., Rahman, T.: Additive Schwarz methods for elliptic mortar finite element problems. Numer. Math. 95(3), 427–457 (2003) 3. Braess, D., Dahmen, W., Wieners, C.: A multigrid algorithm for the mortar finite element method. SIAM J. Numer. Anal. 37(1), 48–69 (1999) 4. Dryja, M., Widlund, O.B.: A generalized FETI-DP method for a mortar discretization of elliptic problems. In: Domain decomposition methods in science and engineering, pp. 27–38. Natl. Auton. Univ. Mex., M´exico (2003) 5. Dryja, M.: A Neumann-Neumann algorithm for a mortar discetization of elliptic problems with discontinuous coefficients. Numer. Math. 99, 645–656 (2005) 6. Kim, H.H., Widlund, O.B.: Two-level Schwarz algorithms with overlapping subregions for mortar finite elements. SIAM J. Numer. Anal. 44(4), 1514–1534 (2006) 7. Xu, X., Li, L., Chen, W.: A multigrid method for the mortar-type Morley element approximation of a plate bending problem. SIAM J. Numer. Anal. 39(5), 1712–1731 (2001/2002) 8. Marcinkowski, L.: Domain decomposition methods for mortar finite element discretizations of plate problems. SIAM J. Numer. Anal. 39(4), 1097–1114 (2001) 9. Marcinkowski, L.: An Additive Schwarz Method for mortar Morley finite element discretizations of 4th order elliptic problem in 2d. Electron. Trans. Numer. Anal. 26, 34–54 (2007) 10. Marcinkowski, L., Dokeva, N.: A FETI-DP method for mortar finite element discretization of a fourth order problem. In: Langer, U., Discacciati, M., Keyes, D.E., Widlund, O.B., Zulehner, W. (eds.) Domain decomposition methods in science and engineering XVII. Lect. Notes Comput. Sci. Eng., vol. 60, pp. 583–590. Springer, Berlin (2008) 11. Mandel, J., Brezina, M.: Balancing domain decomposition for problems with large jumps in coefficients. Math. Comp. 65(216), 1387–1401 (1996) 12. Le Tallec, P.: Neumann-Neumann domain decomposition algorithms for solving 2D elliptic problems with nonmatching grids. East-West J. Numer. Math. 1(2), 129–146 (1993) 13. Pencheva, G., Yotov, I.: Balancing domain decomposition for mortar mixed finite element methods. Numer. Linear Algebra Appl. 10(1-2), 159–180 (2003) 14. Tu, X., Li, J.: A balancing domain decomposition method by constraints for advection-diffusion problems. Commun. Appl. Math. Comput. Sci. 3, 25–60 (2008) 15. Marcinkowski, L.: A mortar element method for some discretizations of a plate problem. Numer. Math. 93(2), 361–386 (2002) 16. Toselli, A., Widlund, O.: Domain decomposition methods—algorithms and theory. Springer Series in Computational Mathematics, vol. 34. Springer, Berlin (2005) 17. Brenner, S.C., Scott, L.R.: The mathematical theory of finite element methods, 3rd edn. Texts in Applied Mathematics, vol. 15. Springer, New York (2008) 18. Marcinkowski, L.: A balancing Neumann - Neumann method for a mortar finite element discretization of fourth order elliptic problems (in preparation)
Application Specific Processors for the Autoregressive Signal Analysis Anatolij Sergiyenko1, Oleg Maslennikow2 , Piotr Ratuszniak2 , Natalia Maslennikowa2, and Adam Tomas3 1
3
National Technical University of Ukraine, pr. Peremogy 37, 03056 Kiev, Ukraine
[email protected] 2 Koszalin University of Technology, ul. Partyzanow 17, 75-411 Koszalin, Poland
[email protected] Cz¸estochowa University of Technology, ul. Armii Krajowej 21, 42-200 Cz¸estochowa, Poland
Abstract. Two structures of the processors for the autoregressive analysis are considered. The first of them implements the Durbin algorithm using the rational fraction calculations. Such calculations provide higher precision than integer calculations do, and are simpler than the floating point calculations. The second of them implements the adaptive lattice filter. These processors are configured in FPGA, and give the possibility of the signal analysis with the sampling frequency up to 300 MHz. They provide new opportunities for the real time signal analysis and adaptive filtering in radio receivers, ultrasonic devices, wireless communications, etc.
1
Introduction
Autoregressive (AR) analysis is the effective and relatively simple method for the estimation of the physical object parameters. Additionally, it provides the modeling of these objects. Therefore, the AR analysis is widely used in economic process prediction, seismic exploring, medical diagnostics, mobile phones, music instrument modeling, etc. The use of the field programmable gate arrays (FPGAs) provides the digital signal processing in the frequency range up to hundreds of megahertz. The FPGA architecture is adapted to the fixed point arithmetic calculations. But the AR analysis algorithms afford the increased precision, and therefore, in most of cases they use the floating point calculations. In the paper the structures of the AR analysis processors are proposed, which are configured in FPGA. They provide AR analysis of the signals in the wide frequency range using the fixed point calculation units. AR analysis is based on the hypothesis that the physical object is represented by the AR filter model. This model implements the filtering of some excitation signal, and generates the samples of output signal x(n). The white noise is usually selected as the excitation signal. However it can have the special form, as one in the vocoders. AR filter coefficients, named as the prediction coefficients R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 80–86, 2010. c Springer-Verlag Berlin Heidelberg 2010
Application Specific Processors for the AR Signal Analysis
81
ai (i = 0, . . . , p), are derived in two steps. Firstly, the autocorrelation function samples rxx (i) of the signal x(n) are estimated. Then the normal Yule-Walker equation system is solved, like the following: Rxx (a1 , . . . , ap )T = −(rxx (1), . . . , rxx (p))T
(1)
whereRxx is the symmetrical Toeplitz matrix, which first row is equal to rxx (0), . . . , rxx (p − 1), a0 = 1 [1]. Equation system (1) is usually solved using the Durbin algorithm, which can be represented by the following loop nest for i = 1 to p a0 = 1; ki = −(a0 ri−1 + . . . + ai−1 r0 )/Ei−1 ; ai = ki ; for j = 1 to i − 1 aj = aj + ki ai−j ; end Ei = (1 − ki2 )Ei−1 ; end;
(2)
where the initial value of the error prediction is E0 = rxx (0) = r0 . Both prediction coefficients ai and reflection coefficients ki are used as the AR analysis results. The coefficients ai are often used in the methods of the high resolution spectrum analysis. The distance between coefficients ki , which values approach to ±1, is interpreted as the distance between medium layers. The excitation signal is reflected from these layers. The analysis of the algorithm (2) shows that the operation number is of the order of p2 , and this algorithm is complex to be parallelized. Therefore, more complex, but well parallelizable algorithms like QR-factorization algorithm are often recommended except the Durbin algorithm [2,3]. Therefore, the Givens algorithm and the Toeplitz matrix inversion algorithm are of success by their FPGA implementation [4]. In the algorithm (2) the division to the error prediction coefficient Ei can occur, when Ei approaches to zero, which forces the high result error. Therefore, this equation solving affords the calculations with the increased precision. Analogous properties have the similar algorithms of the Toeplitz equation solving [2]. Therefore, the AR analysis problem is usually solved using the floating point microprocessors. In the paper two AR processor structures are proposed, which find the effective implementations in the modern FPGAs.
2
AR Processor Based on the Durbin Algorithm
To derive the stable estimations of ai in (1), the values rxx (i) must be accumulated for at least qp = 10p samples of data [1]. When is limited by tenths, then
82
A. Sergiyenko et al.
the correlation function Rxx complexity is approximately in times higher then equation system (1) solving complexity. Usually the input data are sampled in sequence. Then the correlation accumulation is predominant in the time balance of the whole AR analysis. As a result, the equation system solving using the parallel processor array could not give the valuable throughput. In this situation the processor structure for the AR analysis is proposed, which contains correlation processor and Durbin algorithm processor. The correlation processor is the linear array of processor units, operating with the fixed point data. The Durbin algorithm processor implements the calculations with the increased precision. This structure is illustrated by the Fig. 1. Both processors can implement their tasks approximately for the equal period of time in the pipelined manner. The critical path in the algorithm (2) approaches through the cyclic path of the coefficient ki calculation. By the pipelined implementation of the arithmetical operations the iteration period is no less than the stage number of the adder and multiplier pipelines. If the floating point arithmetical unit is used then the iteration period is no less than 7 clock cycles, and this unit is underloaded [6].
Fig. 1. Structure of the AR analysis processor based on the Durbin processor
In [5] the rational fraction arithmetic operations were proposed for the linear algebra problem solving, which provide the advantages by the FPGA implementation. The rational fraction number x is represented by two integers, one of them is a numerator nx , and another one is a denominator dx , i.e. x = nx /dx . All the data are represented by these fractions. Then multiplication, division, and addition are calculated by the formulas: x ∗ y = nx ny /(dx dy );
x/y = nx dy /(dx ny );
x + y = (nx dy + ny dx )/(dx dy )
Such calculations are supported in the modern FPGA by a set of hundreds of multiplication and accumulation units like DSP48 unit in Xilinx Virtex FPGAs. The use of calculations with the rational fractions provides both small error level and expanded dynamic range, as well as the high speed and simple implementation in FPGA. Besides, division operation, which is included in the algorithm critical path, is very short, and adds the minimum error to the calculations.
Application Specific Processors for the AR Signal Analysis
83
If the natural representation of the results is needed then the computations (2) are finished by the division of numerators to denominators of the results ai or ki by the usual division operation. Such a division slightly increases the Durbin processor operation time. The structure of the rational fraction arithmetic unit is shown in [5]. The structure of the correlation processor can reflect the FIR filter structure.
3
AR Processor Based on the Lattice Filter
FIR filter with the coefficients ai is the reverse one to the AR filter. Therefore, the adaptive FIR filter is often used to derive the autoregressive coefficients [1,7]. But the slow adaptation of such a filter prevents its use for the frequent physical process analysis. The lattice form of the adaptive FIR filter is well known, which implements the calculations [1]: f b Ejf (n) = Ej−1 (n) + kj Ej−1 (n − 1);
f b Ejb (n) = Ej−1 (n − 1) + kj Ej−1 (n),
b where Ejf (n), Ej−1 (n) are forward and backward prediction errors in the n−th algorithm stage, respectively, kj is the reflection coefficient, E0f (n) = E0b (n) = x(n), j = 1, . . . , p. In the FIR filters with the sequential adaptation process the coefficients kj are calculated using some recursive gradient algorithm, which converges in several hundreds of steps. In the lattice filter which computes N = qp data samples these coefficients are calculated by the following formula for N steps [1]: f f b b (n))2 ) (n)Ej−1 (n)/( Σn (Ej−1 (n))2 Σn (Ej−1 kj = −Ej−1
Here the sums in numerator and denominator are the partial correlation coefficients. In this computational schema the coefficients kj are found step by step, beginning at k1 . By this process one input data array is inputted p times to the lattice filter. If the signal is stationary one in the time window of pN samples, then the data stream can be fed into the filter permanently without the buffering. The processor for the AR analysis consists of the lattice filter and the coefficient kj calculation unit (see Fig. 2). At the j−th step of adaptation the coefficient calculation unit is attached to the inputs of the j−th filter stage. It accumulates the partial correlation coefficients for N input data, and calculates the coefficient kj . This coefficient is loaded into the multiplicaton units of the j−th filter stage. As we see, the step of the correlation function estimation and the step of the reflection coefficient finding are combined in this computational schema. Comparing to the usual adaptive filters, here the robust convergence to the result is achieved only for pN clock cycles. Comparing to the considered above Yule-Walker equation solver, in the lattice filter the computation error accumulation is minimized, and always the stable results occur. Note that the stable
84
A. Sergiyenko et al.
Fig. 2. Structure of the AR analysis processor based on the lattice filter
results mean that |kj | < 1. Therefore, to implement the coefficient calculation unit the usual integer number calculations would do. Such an AR analysis processor can be configured in FPGA, but the integers have to be at least of doubled bit width. The convergence speed can be increased by the attachment of coefficient calculation units to all the filter stages. But then the hardware volume increases dramatically.
4
AR Processors Implemented in FPGA
Two mentioned above schemes for the AR analysis were carried out in the application specific processors, which are configured in the Xilinx FPGA. The Durbin algorithm processor is based on the arithmetic unit, which implements multiplication, division, accumulation of the rational fractions as well as transformation them in integers. The fraction contains 18-bit numerator and 18-bit denominator. Multiplication and division operations are implemented in the pipelined mode with the period of a single clock cycle. The latent delay is 7 clock cycles. Accumulation operation period lasts 4 clock cycles because this operation result is fed directly to its input. The results of the processor implementation for 16-bit data, N = 10 and different values of p are represented in the table 1. Their comparison shows that processors based on the Durbin algorithm have higher clock frequency, less hardware volume and shorter adaptation period. The processor analysis shows that input data sampling frequency can be increased more than in two times if the correlation processor clock signal is increased. The diagram on the Fig. 3 shows the prediction error of the lattice filter by the analysis of the white noise, which was filtered through the 6-th order IIR band pass filter. The square impulses show the periods of the coefficient kj calculation. One can see the high speed of the convergence of the adaptation. The processor based on the lattice filter has the advantage for small parameters p and for N < 10, as well as for processing the continuous data flows. Besides, it provides the robust estimation of kj for any input data.
Application Specific Processors for the AR Signal Analysis
85
Table 1. Processors of the order p implemented in Xilinx XC4VDX35-12 FPGA Processor based on
Hardware volume CLB slices+DSP48 p=10
p=30
p=90
Max. clock Period of deriving frequency, MHz clock cycles p=10 p=30 p=90 p=10 p=30
Durbin algorithm 1330+16 1850+36 4753+96 154 lattice filter
1446+24 2998+64 6822+184 151
p=90
159
150
610 1830
5490
147
145
1650 10950 86850
Fig. 3. Error prediction signal E f (n) at the lattice filter output
Comparing to the similar processors for the AR analysis based on FPGA [9,10] the proposed processors have substantially less hardware volume. The filter described in [10] has in 20 times more logic cells than the processor based on the Durbin algorithm having the equal multiplier unit number.
5
Conclusions
In the paper two structures of the processors for the AR analysis are proposed, which give the possibility of the signal analysis with the sampling frequency up to 300 MHz. The processors are configured in Xilinx FPGA. These processors give the possibility to expand substantially the area of the AR analysis use, for example in the ultrasonic devices, wireless communications. The proposed rational fraction number system provides higher precision than integers do, and is simpler in its implementation than the floating number system.
References 1. Candy, F.V.: Model-Based Signal Processing, p. 677. Wiley, Hoboken (2006) 2. Brent, R.P.: Old and new algorithms for Toeplitz systems. In: Proc. SPIE. Advanced Algorithms and Architectures for Signal Processing III, vol. 975. SPIE, Bellingham (1989) 3. Bojanchuk, A.W., Brent, R.P., de Hoog, F.R.: Linearly Connected Arrays for Toeplitz Least-Squares Problems. J. of Parallel and Distributed Computing 9, 261– 270 (1990)
86
A. Sergiyenko et al.
4. Sergyienko, A., Maslennikow, O.: Implementation of Givens QR Decomposition inFPGA. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2001. LNCS, vol. 2328, pp. 453–459. Springer, Heidelberg (2002) 5. Sergyienko, A., Maslennikow, O., Lepekha, V.: FPGA Implementation of the Conjugate Gradient Method. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 526–533. Springer, Heidelberg (2006) 6. Karlstr¨ om, P., Ehliar, A., Liu, D.: High performance, low latency FPGA based floating point adder and multiplier units in a Virtex4. In: 24th IEEE Norchip Conf., Link¨ oping, Sweden, November 20-21, pp. 31–34 (2006) 7. Widrow, B., Stearns, S.D.: Adaptive Signal Processing. Prentice-Hall, Englewod Clifs (1985) 8. Pohl, Z., Matouek, R., Kadlec, J., Tich´ y, M., L´ıcko, M.: Lattice adaptive filter implementation for FPGA. In: Proc. 2003 ACM/SIGDA 11th Int. Symp., Monterey, California, USA, pp. 246–250 (2003) 9. Hwang, Y.T., Han, J.C.: A novel FPGA design of a high throughput rate adaptive prediction error filter. In: 1st IEEE Asia Pacific Conf. on ASICs, AP-ASIC 1999, pp. 202–205 (1999) 10. Lin, A.Y., Gugel, K.S.: Feasibility of fixed-point transversal adaptive filters in FPGA devices with embedded DSP blocks. In: 3rd IEEE IWSOC 2003, pp. 157– 160 (2003)
A Parallel Non-square Tiled Algorithm for Solving a Kind of BVP for Second-Order ODEs Przemyslaw Stpiczy´ nski Department of Computer Science, Maria Curie–Sklodowska University Pl. M. Curie-Sklodowskiej 1, 20-031 Lublin, Poland
[email protected]
Abstract. The aim of this paper is to show that a kind of boundary value problem for second-order ordinary differential equations which reduces to the problem of solving tridiagonal system of linear equations with almost Toeplitz structure can be efficiently solved on modern multicore architectures using a parallel tiled algorithm based on the divide and conquer approach for solving linear recurrence systems with constant coefficients and novel data formats for dense matrices. Keywords: BVP for ODEs, parallel non-square tiled algorithm, multicore, novel data formats for dense matrices.
1
Introduction
Let us consider the following boundary value problem which arises in many practical applications [4]: −
d2 u = f (x) ∀x ∈ [0, 1], dx2
(1)
where u (0) = 0 and u(1) = 0. Numerical solution to the problem (1) reduces to the problem of solving tridiagonal systems of linear equations. Simple algorithms based on Gaussian elimination achieve poor performance, since they do not fully utilize the underlying hardware, i.e. memory hierarchies, vector extensions and multiple processors. The matrix of such systems have a special (almost Toeplitz) form and it clear that it should also be exploit. The aim of this paper is to present a new implementation of the algorithm presented in [6]. The new implementation is based on the idea of developing parallel tiled algorithms which are suitable for multicore architectures. The performance of the algorithm can be improved by using non-square tiles which can better fit into L1 cache [1].
2
Method
We want to find an approximation of the solution to the problem (1) in the grid points 0 = x1 < x2 < . . . < xn+1 = 1, R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 87–94, 2010. c Springer-Verlag Berlin Heidelberg 2010
88
P. Stpiczy´ nski
where xi = (i − 1)h, h = 1/n, i = 1, . . . , n + 1. Let fi = f (xi ) and ui = u(xi ). Using the approximation for the second derivative u (xi ) ≈
ui+1 − 2ui + ui−1 h2
and the boundary conditions u (0) ≈
u(0 − h) − 2u1 + u2 , h2
u (0) ≈
u2 − u(0 − h) 2h
we get the following equations 2(u1 − u2 ) = h2 f1 , −ui−1 + 2ui − ui+1 = h2 fi , i = 2, . . . , n − 1,
(2)
−un−1 + 2un = h fn . 2
Thus, we can rewrite (2) as the problem of solving the system of linear equations Au = d, where
(3)
⎞ 1 −1 ⎟ ⎜ −1 2 −1 ⎟ ⎜ ⎜ ⎟ . . . .. .. .. A=⎜ ⎟ ∈ IRn×n ⎜ ⎟ ⎝ −1 2 −1 ⎠ −1 2 ⎛
T
T
and the vectors satisfy u = (u1 , . . . , un ) , d = (d1 , . . . , dn ) and d1 = 12 h2 f1 , di = h2 fi for i = 2, . . . , n. The matrix A can be factorized as A = LR, where L and R are bidiagonal Toeplitz matrices ⎛ ⎞ ⎛ ⎞ 1 1 −1 ⎜ −1 1 ⎟ ⎜ ⎟ 1 −1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ .. .. .. .. L=⎜ (4) ⎟, R = ⎜ ⎟ . . . . ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ −1 1 1 −1 ⎠ −1 1 1 The solution to the system (3) can be found using Gaussian elimination without pivoting. A simple algorithm for solving (3) based on such a factorization can comprise two sequential stages, namely solving the systems Ly = d (forward reduction) and Ru = y (back substitution). The first stage can be done using y1 = d1 (5) yi = di + yi−1 for i = 2, . . . , n. Then (second stage) we find the final solution to the system (3) using u n = yn ui = yi + ui+1 for i = n − 1, n − 2, . . . , 1.
(6)
Parallel Non-square Tiled Algorithm for Solving BVP for ODEs
89
More sophisticated approach can be based on the divide-and-conquer algorithm for solving linear recurrence systems [5]. The main idea of the algorithm is to rewrite the considered systems as block-bidiagonal systems of linear equations [5,7]. Without loss of generality, let us assume that there exist two positive integers r and s such that rs ≤ n and s > 1. The method can be used for finding y1 , . . . , yrs . To find yrs+1 , . . . , yn , we use (5) directly. Then for j = 1, . . . , r, we define vectors
T
T dj = d(j−1)s+1 , . . . , djs , yj = y(j−1)s+1 , . . . , yjs ∈ IRs and find all yj using the following formula y1 = L−1 s d1 yj = L−1 s dj + y(j−1)s e for j = 2, . . . , r,
(7)
where e = (1, . . . , 1)T and the matrix Ls ∈ IRs×s is of the same form as L given by (4). Analogously, we can perform the second stage using ur = Rs−1 yr (8) uj = Rs−1 yj + ujs+1 e for j = r − 1, . . . , 1,
T where uj = u(j−1)s+1 , . . . , ujs ∈ IRs and Rs ∈ IRs×s is of the same form as R given by (4). However, it is better to find the matrix U = (u1 , . . . , ur ) ∈ IRs×r
(9)
instead of individual vectors uj . Also, there is no need to use temporary vectors yj . It is clear that we have to allocate only one s × r array U and initially store vectors dj in its columns, namely U(∗, j) = dj . Then (Step 1A) we perform the following loop do k=2,s U(k,1:r)=U(k,1:r)+U(k-1,1:r) end do In general, during this step we use a previously updated row to update the next row of the table. During Step 1B we compute the bottom row of the table using the following simple loop do k=2,r U(s,k)=U(s,k)+U(s,k-1) end do Finally (Step 1C) we update s − 1 first entries of each column (except for the first one) using the entries of the bottom row. do k=2,r U(1:s-1,k)=U(1:s-1,k)+U(s,k-1) end do
90
P. Stpiczy´ nski Stage 1
Step A
Step B
Step C
Step A
Step B
Step C
Stage 2
used
modified
direction of updates
Fig. 1. Divide and conquer algorithm using a 2-D array
Similarly we proceed during the second stage of the algorithm (Figure 1). During Step 2A we update rows of the table bottom-up. Then (Step 2B) we update the first row of the table from right to left. Finally (Step 2C) using elements from the first row, we update the remaining entries of each column (except for the last one). do k=s-1,1,-1 U(k,1:r)=U(k,1:r)+U(k+1,1:r) end do ! do k=r-1,1,-1 U(s,k)=U(s,k)+U(s,k+1) end do ! do k=1,r-1 U(2:s,k)=U(2:s-1,k)+U(1,k+1) end do Both stages can be easily vectorized and parallelized. The array can be divided into blocks of columns and each processor can be responsible for computing one block. Steps A and C can be performed in parallel but steps B (in both stages) are sequential. Unfortunately, the performance of this algorithm can be poor because of cache misses which may occur during its execution. Note that
Parallel Non-square Tiled Algorithm for Solving BVP for ODEs
91
Stage 1
Step B
Step A
Step C
Stage 2
Step C
Step B
Step A
used
modified
direction of updates
Fig. 2. Divide and conquer algorithm using non-square tiles
it cannot be improved using simple standard techniques like loop interchange, because entries of the table are accessed row-wise (the steps A in both stages) and column-wise (the steps C in both stages).
3
Non-square Tiled Algorithm
In [6] we showed that the performance of the algorithm can be significantly improved by using novel data formats for dense matrices with the square blocked full column major order [3]. The performance of the algorithm can still be improved by the use of non-square tiles which can better fit into L1 cache than square blocks used in the square blocked full column major order format. Let us observe that the matrix U defined by the equation (9) can be partitioned as follows ⎞ ⎛ U11 . . . U1ng ⎜ .. ⎟ . (10) U = ⎝ ... . ⎠ Umg 1 . . . Umg ng
92
P. Stpiczy´ nski
Each submatrix Uij is stored as a rectangular (generally non-square) mb × nb tile which occupies a contiguous block of memory. The matrix U can be stored in a four dimensional array declared in Fortran 95 as follows real (kind=8) :: U(mb,nb,mg,ng) where mb = mb , nb = nb , mg = s/mb and ng = r/nb . For simplicity we assume that mb mg = s and nb ng = r. The new tiled algorithm also comprises two stages, each with three steps A, B, C but it operates on tiles (Figure 2). During Step 1A, we update columns of tiles (i.e. panels) top-down using vector addition. It is clear that when we operate within a single tile which is in L1 cache, a cache miss will not occur. However, on the border between two tiles in a panel, a cache miss may occur. During Step 1B, we operate on a bottom row of tiles and we have such a border between each couple of adjacent tiles. Finally, during Step 1C, we use bottom row of each tile in the bottom row of tiles to update columns in each panel. Similarly we can implement the second stage of the algorithm. During Step 2A, we update tiles within panels bottom-up and during Step 2B we update tiles from right to left. Note that in Step 2C it is better to update panels from left to right and tiles within each panel should be updated top-down. Then we can reduce the number of cache misses. It is clear that in both stages the steps A and C can be done in parallel. Each processor can be responsible for computing a group of panels. The algorithm can be expressed as a number of small computational tasks operating on tiles. Such tasks can be scheduled for parallel execution using OpenMP [2] and the well-known parallel construct. !$omp parallel private(...) shared(...) ! Stage 1 !$omp do private(...) default(shared) <---- Step 1A do j=1,ng ... end do !$omp single <---- Step 1B ... !$omp end single !$omp do private(...) default(shared) <---- Step 1C do j=1,ng ... end do ! Stage 2 ..... !$omp end parallel Within the parallel region we have three constructs: a parallelized do-loop for Step 1A, a single construct for Step 1B, and again a parallelized do-loop for Step 1C. Similarly we can implement the second stage on the algorithm.
Parallel Non-square Tiled Algorithm for Solving BVP for ODEs
4
93
Experimental Results
The algorithms for solving the problem (1) have been implemented in Fortran 95 with OpenMP directives [2]. The experiments carried out on a dual processor Quad-Core Xeon (3.2 GHz, 12 MB L2 cache, 32 GB RAM running under Linux) workstation using Intel Fortran Compiler show that the new algorithm achieves much better performance than the simple scalar algorithm based on (5)–(6).
Table 1. Performance of the simple scalar algorithm n 2097152 1048576
time (s) 0.173E-1 0.701E-2
Mflops 362.92 448.69
Table 2. Exemplary results of the performance of the parallel tiled algorithm n 2097152 2097152 1048576 1048576
s 2048 2048 1024 1024
r 1024 1024 1024 1024
mb 64 32 64 32
nb 16 32 16 32
time (s) speedup 0.312E-2 5.46 0.416E-2 4.17 0.194E-2 3.69 0.217E-2 3.29
Mflops 1984.68 1511.96 1618.86 1444.53
Table 2 shows exemplary results of the performance of the algorithm for various problem sizes (values of n, s, r) and block sizes (values of mb and nb ). Three last columns show execution time in seconds, speedup measured against athesimple scalar algorithm and Mflops. One can observe that the performance of the simple algorithm is very poor (Table 1) – only a very small fraction of the peak performance is achieved (about 400 Mflops using double precision). The performance of the new parallel non-square tiled algorithm is much better. The performance is better for the bigger problem size (n = 2097152) and the algorithm achieve reasonable speedup. One can observe that the algorithm with non-square tiles achieves better performance than for square blocks.
5
Conclusions
We have shown that the performance of the simple algorithm for solving a kind of boundary value problem for second-order ordinary differential equations which reduces to the problem of solving tridiagonal systems of linear equations can be highly improved by the use of the divide and conquer vectorized algorithm using novel data formats for dense matrices. Further improvements can be obtained by the use of non-square tiles. It should be noticed that the method can be applied to solve such problems on many-core graphics processing units with CUDA.
94
P. Stpiczy´ nski
References 1. Buttari, A., Langou, J., Kurzak, J., Dongarra, J.: A class of parallel tiled linear algebra algorithms for multicore architectures. Parallel Computing 35, 38–53 (2009) 2. Chandra, R., Dagum, L., Kohr, D., Maydan, D., McDonald, J., Menon, R.: Parallel Programming in OpenMP. Morgan Kaufmann Publishers, San Francisco (2001) 3. Gustavson, F.G.: New generalized data structures for matrices lead to a variety of high performance algorithms. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2001. LNCS, vol. 2328, pp. 418–436. Springer, Heidelberg (2002) 4. Scott, L.R., Clark, T., Bagheri, B.: Scientific Parallel Computing. Princeton University Press, Princeton (2005) 5. Stpiczy´ nski, P.: Solving linear recurrence systems using level 2 and 3 BLAS routines. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 1059–1066. Springer, Heidelberg (2004) 6. Stpiczy´ nski, P.: Solving a kind of boundary value problem for ODEs using novel data formats for dense matrices. In: Ganzha, M., Paprzycki, M., Pelech-Pilichowski, T. (eds.) Proceedings of the International Multiconference on Computer Science and Information Technology, vol. 3, pp. 293–296. IEEE Computer Society Press, Los Alamitos (2008) 7. Stpiczy´ nski, P., Paprzycki, M.: Fully vectorized solver for linear recurrence systems with constant coefficients. In: Proceedings of VECPAR 2000 - 4th International Meeting on Vector and Parallel Processing, Porto, pp. 541–551. Facultade de Engerharia do Universidade do Porto (June 2000)
Graph Grammar Based Petri Nets Model of Concurrency for Self-adaptive hp-Finite Element Method with Rectangular Elements Arkadiusz Szymczak and Maciej Paszy´ nski Department of Computer Science AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Cracow, Poland
[email protected],
[email protected] http://home.agh.edu.pl/~ paszynsk
Abstract. The paper presents the Petri nets model describing the process of rectangular finite element mesh generation and h adaptation. This is the first step towards the formal analysis of all the parts of the existing graph grammar based parallel adaptive hp-Finite Element Method algorithms expressed by graph grammar productions. The mesh transformations are expressed as composite programmable graph grammar productions. The transitions on Petri nets correspond to the execution of graph grammar productions. The graph grammar based algorithms modeled as a Petri net can be analyzed for deadlocks, starvation or infinite execution. The parallel adaptive algorithms have many applications in the area of numerical simulations of different engineering problems, including material science, heat transfer, wave propagation and other. Keywords: Finite Element Method, hp adaptivity, Petri nets.
1
Introduction
The paper presents the Petri nets model dedicated to the process of two dimensional rectangular finite element mesh generation and adaptation. Besides the parallel solver, the mesh generation and adaptation are the most essential parts of adaptive hp-Finite Element Method (hp-FEM) algorithm [1,2,11]. The algorithm has been used to solve many engineering problems [3,4,1,2]. The adaptive parallel hp-FEM algorithm has been expressed by CP-graph grammar for triangular [5] as well as rectangular finite elements [9,10]. The models utilize the CP graph grammar introduced by [6,7,8]. The Petri nets model has been already created for triangular finite elements [12]. The extension of the model for rectangular finite elements is critical, since the graph grammar based model of parallelization of the self-adaptive hp-FEM algorithms has been fully developed for rectangular finite elements [10]. The Petri nets model will allow to analyze the algorithms for deadlocks, starvation or infinite execution. Actually, we have encountered several deadlocks with anisotropic R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 95–104, 2010. c Springer-Verlag Berlin Heidelberg 2010
96
A. Szymczak and M. Paszy´ nski
mesh refinements, related to the enforcement of the 1-irregularity mesh rule, for the case when the refinements have been executed over the computational mesh distributed into several sub-domains, and later the mesh has been repartitioned in order to maintain quasi-uniform load balance. The Petri nets model extended to anisotropic refinements will allow to make necessary corrections to the mesh refinements algorithm, to avoid the deadlock.
2
2D Mesh Generation
In this section we present a set of graph grammar productions for generation of two-dimensional mesh with rectangular finite elements. This extends the graph grammar introduced in [9] generating a row of rectangular elements. The new productions PI1 - PI5, presented in Figure 1, realize the topology generation of an arbitrary shape 2D rectangular mesh. Meaning of the node symbols is as follows: – Iel unstructured mesh element node – NIL - fake node denoting no adjacent node – ANY - fake node meaning that given node is irrelevant to the given production The new productions PI6 - PI9, presented in Figure 2, are responsible for identification of adjacent elements to prevent element overlapping. Since new element is always derived from one existing element and each element can have up to 4 adjacent elements, missing up to 3 bonds must be detected. Therefore, as soon as any of productions PI6 - PI9 matches, it must be executed before subsequent execution of productions PI1 - PI5. The generation of the topology of an initial mesh is followed by the generation of the structure of finite elements, which in turn is followed by the execution of mesh refinements. The exemplary generation of the topology of a simple square finite element mesh, is presented on the first row in Figure 3. This corresponds to execution of (PI1)-(PI5)-(PI4)2 (PI7) graph grammar productions. The generation of the structure of four finite elements followed by the h refinement of the left-bottom element is presented on the second row in Figure 3. This corresponds to execution of (PE)4 -(PCEH)2 (PCEV)2 -(PJI)4 -(PBI)-(PFE)2 -(PBE)2 graph grammar productions.
3
Graph Grammar Based Model of Concurrency
In this section we present the control diagram based model of concurrency for the process of an initial mesh generation as well as mesh h adaptation. The model is presented in Figure 4. The following list summarizes symbols used in the control diagrams. Some graph grammar productions referred in the list were defined in [9], while the productions PI1 - PI9 were described in the previous section.
Graph Grammar Based Petri Nets Self-adaptive hp-FEM
97
Fig. 1. Set of graph grammar productions PI1 - PI5 for the generations of rectangular finite element two dimensional mesh
Fig. 2. Graph grammar productions PI6 - PI9 connecting layers of generated elements
– PE - production for generation of element structure – PCEH - production for identification of common edge of horizontally adjacent elements
98
A. Szymczak and M. Paszy´ nski
Fig. 3. Exemplary generation of the topology of four finite element mesh, followed by the generation of the structure of finite elements, followed by a single h refinement
– PCEV - production for identification of common edge of vertically adjacent elements – PJI - production allowing for breaking element interior – PFE - production allowing for breaking edge between two broken interiors – PBI - production for breaking element interior – PBE production for breaking element edge – PWest - production for propagation of adjacency data for west edge of element – PEast - production for propagation of adjacency data for east edge of element
Graph Grammar Based Petri Nets Self-adaptive hp-FEM
99
Fig. 4. Left panel: Control diagram based model of concurrency for the process of the generations of rectangular finite elements two dimensional mesh. Right panel: Control diagram based model of concurrency for the process of h adaptation.
– PSouth - production for propagation of adjacency data for south edge of element – PNorth - production for propagation of adjacency data for north edge of element
100
4
A. Szymczak and M. Paszy´ nski
Petri Nets Model of Mesh Transformations
In this section we introduce the Petri net based model of concurrency for the initial mesh generation as well as for h adaptation. In addition to the control diagram based model it provides analysis tools typical for Petri nets like reachability graphs, place/transition invariants, etc. A system modeled as a Petri net can be analyzed for deadlocks, starvation or infinite execution. This model also allows to express some quantitative relationships between graph grammar productions. Figure 5 presents a priority Petri net with guards for initial mesh generation. Transitions are named after productions they execute. Priorities are assigned to transitions (in subscript) to enforce certain sequence of execution. Transitions PI6 - PI9 (priority 1) should be executed as soon as they get activated, before any other transition being active at the same time. The prioritization has been introduced to ensure correct mesh construction (with no overlapping elements). Transitions PI6 - PI9 have an additional activation guard - they can be activated only if there is an actual matching for these productions in the mesh being generated. They also have place p3 as input since whenever new adjacency is identified, the number of free bonds decreases and the potential for new elements generation is reduced. The condition expressed with the guard is not enforced by the net itself. Edge weights denote the number by which the marking of the destination place is increased and the marking of the origin place is decreased. The meaning of some of the places is as follows:
Fig. 5. Petri nets model of concurrency for the process of rectangular finite element two dimensional mesh generation
Graph Grammar Based Petri Nets Self-adaptive hp-FEM
101
– p0 number of elements left to be produced – p3 number of elements that can be produced from current mesh shape – p12 number of potential candidates for adjacency identification The initial marking is: p1 - 1, p0 - N, where N is a parameter to the system telling how many elements the initial mesh will consist of. The rest of the places are empty. The only transitions that increase the number of tokens in the net are PE and PI2 - PI5. Transition PE input to place p3 may be fired only once since the initial marking assigns only one token to place p1. Transitions PI2 - PI5 have place p0 as input which limits the number of their activations since the initial marking of that place is only decreasing during the system execution. Therefore the Petri net is bounded and the initial mesh generation will eventually stop. The reachability graph is finite and suitable for analysis. Place p0 is the determinant of the net’s liveness. When that place gets empty, the net quickly reaches a dead marking - as soon as places p4 - p11, p13 and p14 also get empty. Such a dead marking means the end of initial mesh generation; no deadlock is possible while place p0 contains tokens. The Petri net structure is fixed; different shapes of generated meshes will result in choosing different paths in the reachability graph. Figure 6 presents the Petri net for h adaptation. The initial marking of place p0 is the number of the mesh elements chosen for adaptation by an external driver. This includes elements that have to be broken as prerequisites to the adaptation of the chosen elements (according to the mesh regularity rule). The marking of place p1 determines the adaptation ”potential” of the mesh and initially it will be: number of elements with 4 neighbors multiplied by 4 + number of elements with 3 neighbors times 3 + number of elements with 2 neighbors times 2 + number of elements with 1 neighbor. The marking of places p1 and p6 should be preserved between subsequent iterations of h adaptation.
Fig. 6. Petri nets with guards model of concurrency for the process of h adaptation
102
A. Szymczak and M. Paszy´ nski
Fig. 7. Exemplary reachability graph for the two finite mesh generation
The branch with place p2 models interior breaking of elements with 1 neighbor; branch with place p3 - elements with 2 neighbors; branch with place p4 elements with 3 neighbors; branch with place p5 elements with 4 neighbors. Two transitions linked with a dashed line denote actually all activated transitions of a given type fired concurrently. Although their numbers depend on the actual problem being solved, the reachability graph is still finite and can be analyzed since the net is bounded thanks to the place p0. Mesh regularity rules are enforced with the transition guards. Therefore the net can reach a dead state even if places p1 and/or p6 still contain tokens.
5
Conclusions
This paper presented the Petri nets model for the process of generation of an initial two dimensional rectangular finite element mesh, as well as for the process of mesh h adaptation. The mesh transformations were expressed by graph grammar productions. The order of execution of graph grammar productions was defined by the control diagram. The Petri nets were developed based on the presented control diagrams. The transitions on the Petri nets correspond to the execution of graph grammar productions. The developed model allows for the formal analysis of the process of mesh generation and adaptation, modeled by the graph grammar. For example, the reachability graphs can be automatically generated from the Petri nets, for given initial marking, denoting the number of finite elements to be either generated or refined. The example presented in Figure 8, shows an automatically generated reachability graph for the case of the Petri net from Figure 5 with 2 initial markers. The graph nodes denote the Petri net states(numbers denote marking of places from P0 to P15):
Graph Grammar Based Petri Nets Self-adaptive hp-FEM
103
S0 = {2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} S1 = {1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} S2 = {1, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} S3 = {0, 0, 0, 6, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0} S4 = {0, 0, 0, 6, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0} S5 = {0, 0, 0, 6, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} S6 = {0, 0, 0, 6, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} S7 = {0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0} S8 = {0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0} S9 = {0, 0, 0, 6, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0} S10 = {0, 0, 0, 6, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0} S11 = {0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1} The graph is a DAG whereof all paths start in the initial state S0 and end in S11, denoting the successful finish of the mesh generation. All possible paths contain transition PI1 followed by PE and one of transitions PI2 - PI5 followed by PE, which means that exactly two mesh elements will be generated and structured. Also, in any case one of transitions PCEH or PCEV will be fired, so the common edge between the two elements will be identified. Such a reachability graph is a convenient tool for deadlock, infinite loops and stop condition analysis. However, it quite quickly becomes very complex with the number of mesh elements to generate, but then a coverity graph may be used instead. The future work will involve the development of the Petri nets model for the execution of the parallel direct solver over the generated hp refined mesh. The graph grammar model for these parts of the algorithm has been already developed [13]. The presented model involves only two dimensional rectangular finite elements. The future work will also involve the extension of the graph grammar model to the anisotropic mesh refinements and for the three dimensional finite element meshes, followed by development of the Petri nets model. We also intend to model all aspects of the self-adaptive hp-FEM algorithm by trace theory [14], with the comparison of the descriptive power of the developed models. Acknowledgments. The work reported in this paper was supported by the Foundation for Polish Science under Homming Programme.
References 1. Demkowicz, L.: Computing with hp-adaptive Finite Elements, vol. I. Chapman & Hall/Crc Applied Mathematics & Nonlinear Science, Taylor & Francis Group, Boca Raton (2006) 2. Demkowicz, L., Pardo, D., Paszy´ nski, M., Rachowicz, W., Zduneka, A.: Computing with hp-adaptive Finite Elements, vol. II. Chapman & Hall/Crc Applied Mathematics & Nonlinear Science, Taylor & Francis Group, Boca Raton (2007)
104
A. Szymczak and M. Paszy´ nski
3. Gawad, J., Paszy´ nski, M., Matuszyk, P., Madej, L.: Cellular automata coupled with hp-adaptive Finite Element Method applied to simulation of austenite-ferrite phase transformation with a moving interface. In: Steel Research International, pp. 579–586 (2008) 4. Pardo, D., Torres-Verdin, C., Paszy´ nski, M.: Simulations of 3D DC borehole resistivity measurements with a goal-oriented hp finite-element method. Pt. 2, Throughcasing resistivity instruments, Computational Geosciences 12, 83–89 (2008) 5. Paszy´ nska, A., Paszy´ nski, M., Grabska, E.: Graph transformations for modeling hp-adaptive Finite Element Method with triangular elements. In: Bubak, M., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2008, Part III. LNCS, vol. 5103, pp. 604–613. Springer, Heidelberg (2008) 6. Grabska, E.: Theoretical concepts of graphical modeling. Part One: Realization of CP-graphs, Machine Graphics and Vision 2(1), 3–38 (1993) 7. Grabska, E.: Theoretical concepts of graphical modeling. Part Two: CP-graph grammars and languages. Machine Graphics and Vision 2(2), 149–178 (1993) 8. Grabska, E., Hliniak, G.: Structural aspects of CP-graph languages. Schedae Informaticae 5, 81–100 (1993) 9. Paszy´ nski, M., Paszy´ nska, A.: Graph transformations for modeling parallel hpadaptive Finite Element Method. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 1313–1322. Springer, Heidelberg (2008) 10. Paszy´ nski, M.: On the parallelization of self-adaptive hp-Finite Element Method. Part I. Composite Programmable Graph Grammar Model, Fundamenta Informaticae 93, 411–434 (2009) 11. Pla˙zek, J., Bana´s, K., Kitowski, J., Boryczko, K.: Exploiting two-level parallelism in FEM applications. In: Hertzberger, B., Sloot, P.M.A. (eds.) HPCN-Europe 1997. LNCS, vol. 1225, pp. 878–880. Springer, Heidelberg (1997) 12. Szymczak, A., Paszy´ nski, M.: Graph grammar based Petri nets model of concurrency for self-adaptive hp-Finite Element Method with triangular elements. LNCS, vol. 5145, pp. 845–854. Springer, Heidelberg (2009) 13. Paszy´ nski, M.: Parallel direct solver for hp Finite Element Method controlled by Composite Programmable Graph Grammar. In: Proceedings of the 2008 International Conference on Artificial Intelligence and Pattern Recognition, Orlando, Florida, USA, July 7-10 (2008) 14. Diekert, V., Rozenberg, G.: The book of traces. World Scientific Publishing, Singapore (1995)
Numerical Solution of the Time and Rigidity Dependent Three Dimensional Second Order Partial Differential Equation Anna Wawrzynczak1 and Michael V. Alania2 1 2
Institute of Computer Science, University of Podlasie 3 Maja 54, Siedlce, Poland
[email protected] Institute of Math. and Physics, University of Podlasie, 3 Maja 54, Siedlce, Poland
[email protected]
Abstract. The method of the numerical solution of the time and rigidity dependent three dimensional (3-D) second order partial differential equation describing the anisotropic diffusion of the galactic cosmic ray particles in the heliosphere is proposed. A good agreement between the results of the numerical solution and experimental data is demonstrated. The proposed numerical method gives a broad possibility for implementation of the parallel programming techniques, and could be successfully used to solve second order partial differential equations, among them equations describing different classes of galactic cosmic ray intensity variations observed due to anisotropic diffusion of the galactic cosmic ray particles in the heliosphere. Keywords: partial differential equations, numerical method, modelling.
1
Introduction
Solar activity is characterized by number of sunspots on the surface of the Sun; increasing or dropping number of sunspots in time is called a solar cycle. More dominant and very well established one is the 11-year solar cycle. Each 11-year solar activity cycle is lasting from one minimum (when few sunspots appear or they are not visible at all) to another minimum epoch. The exceptional phenomenon of solar activity is an existence of the solar wind - an extension of the outer atmosphere of the Sun (the corona) into interplanetary space [1]. The solar wind is a stream of charged particles - a plasma, coming out from the Sun in all directions at very high speeds - an average of about 400 km/sec. It consists mostly of equal numbers of electrons and protons with energies of about 1 keV (1000 electron Volts (eV)). The electric currents in the Sun generate a complex magnetic field which extends out into interplanetary space to form the interplanetary magnetic field. The Sun’s magnetic field is carried out through the solar system by the solar wind. The rotation of the Sun winds up the magnetic field into a large rotating spiral, known as the Parker spiral. The magnetic field is primarily directed outward from the Sun in one of its hemispheres, and inward in R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 105–114, 2010. c Springer-Verlag Berlin Heidelberg 2010
106
A. Wawrzynczak and M.V. Alania
the other. This causes opposite magnetic field directions in the Parker spiral. The thin layer between the different field directions is described as the neutral current sheet. Since this dividing line between the outward and inward field directions is not exactly on the solar equator, the rotation of the Sun causes the current sheet to become ”wavy”, and this waviness is carried out into interplanetary space by the solar wind [2]. At the maximum epoch of each 11-year solar cycle (a peak in the sunspot numbers), the polarity of the Sun’s global magnetic field reverses, so that the North magnetic pole becomes the South and vice versa. Every 22 years the magnetic polarity of the Sun is returning to its earlier state. So, there exists the 22-year solar magnetic cycle. Each 11-year period of the 22-year solar magnetic cycle is lasting from one maximum to another maximum epoch. When the global magnetic field lines are directed outward from the north hemisphere of the Sun and are directed backward to south hemisphere, this 11-year period is conventionally called the positive (A > 0) magnetic polarity period, while in vice versa case, it is called the negative (A < 0) polarity period. The solar wind creates the heliosphere, a vast bubble in the interstellar medium surrounding the solar system. The heliosphere extends up to 100AU [3] (AU-astronomical unite is the Sun-Earth distance ∼ 150 million km). So, the whole heliosphere is saturated by the solar wind, a supersonic and super-Alfvenic plasma flow with the regular and turbulence magnetic field and is continuously expanding into the interplanetary space. Galactic cosmic ray (GCR) protons (we consider GCR protons with energies > than 109 eV) propagating through the interplanetary medium are undergoing to the influence (modulation) of the diverged solar wind (source of convection and adiabatic energy change of the GCR particles), and the combination of the regular (averaged) and turbulent (to be stochastic with the zero mean) interplanetary magnetic field being the sources of various kind of drifts and diffusion of the GCR particles. The modulation (changes) of the GCR intensity is generally typified with different long and short scale quasi periodic variations (22- years, 11-years and 27-days), and short time (a few days) changes related in main with the similar changes of the solar activity and solar wind. A short time decrease and recovery of the GCR intensity during 8-10 days is called the Forbush decrease (Fd) [4]. A time dependent modulation of the GCR intensity besides the above mentioned processes depends on the energy or on the rigidity of the GCR particles, so to describe a modulation (changes) of the GCR intensity the time and rigidity dependent model of the transport equation for three dimensional (3-D) heliosphere should be considered. However, in particular cases, e.g. for long period changes of the GCR intensity there is possible to consider stationary (time independent) modulation diminishing in this way a dimension of transport equation and simplify problem of solution [5]. In this paper our aim is to propose method of numerical solution of the transport equation in general case, i.e. to consider time and rigidity dependent 3-D model of the GCR particles propagation in the heliosphere (in the interplanetary space).
Numerical Solution of the Second Order Partial Differential Equation
2
107
GCR Particles Transport Equation
GCR particles are propagating through the causal and stochastic interplanetary magnetic field and their motion is described by the normal anisotropic diffusion equation in the heliosphere [6,7,8,9,10,11,12] ∂N 1 ∂ = ∇i (Kij ∇j N ) − ∇i (Ui N ) + (N R)(∇i Ui ), ∂t 3 ∂R
(1)
where N and R are density and rigidity of cosmic ray particles, respectively; Ui - solar wind velocity, Kij is the anisotropic diffusion tensor of GCR for the three dimensional IMF with components Br ,Bθ and Bϕ , and t-time. The term ∇i (Kij ∇j N ) describes the diffusion of the GCR particles, the term ∇i (Ui N ) ∂ convection and 13 ∂R (N R)(∇i Ui ) the change of the particles energy due to the interaction of the particles and the diverged solar wind with the IMF turbulence. In the heliocentric spherical coordinate system (r, θ, ϕ) for the anisotropic diffusion tensor Kij and for the radial solar wind velocity U ≡ U (Ur , 0, 0) the equation (1) has a form: 1 ∂ ∂N ∂N ∂N r ∂N = 2 (r2 Krr + rKrθ + Krϕ ) ∂t r ∂r ∂r ∂θ sin θ ∂ϕ 1 ∂ ∂N sin θ ∂N 1 ∂N (sin θKθr + Kθθ + Kθϕ ) + r sin θ ∂θ ∂r r ∂θ r ∂ϕ 1 ∂ ∂N 1 ∂N 1 ∂N + (Kϕr + Kϕθ + Kϕϕ ) r sin θ ∂ϕ ∂r r ∂θ r sin θ ∂ϕ 1 ∂ n 1 ∂ R ∂N − 2 (r2 N Ur ) + ( + )( 2 [r2 Ur ]). r ∂r 3 ∂R 3 r ∂r
(2)
We set the dimensionless density f = NN0 , time t∗ = tt0 , distance r = rr0 , and R , where N and N0 are density in the interplanetary space and rigidity R = 1GV in the local interstellar medium (LISM), respectively; N0 = 4πI0 , where the −2.8 intensity I0 in the LISM [13,14] has the form I0 = 1+5.85T21.1T −1.22 +1.18T −2.54 , T √ is kinetic energy in GeV (T = R2 + 0.9382 − 0.938); t0 is the characteristic time of GCR modulation corresponding to the change of the electro-magnetic conditions in the modulation region for the certain class of GCR variation; r is the radial distance and r0 is the size of the modulation region. For the case of Fd we accepted that t0 is equal to the Sun’s complete rotation period (27−days ∼ 2.3328 ∗ 106 s) and the size of modulation region r0 equals 30 AU; Kij is the anisotropic diffusion tensor of GCR for the three dimensional IMF taken as in [15,16]. The Eq. (1) for the dimensionless variables f , t∗ and r in the spherical coordinate system (r, θ, ϕ) can be written: A0
∂f ∂f = M f + A11 , ∗ ∂t ∂R
where M = A1
∂2 ∂2 ∂2 ∂2 ∂2 ∂2 + A + A + A + A + A 2 3 4 5 6 ∂r2 ∂θ2 ∂ϕ2 ∂r∂θ ∂θ∂ϕ ∂r∂ϕ
(3)
108
A. Wawrzynczak and M.V. Alania
+A7
∂ ∂ ∂ + A8 + A9 + A10 . ∂r ∂θ ∂ϕ r2
The coefficient A0 = t0 K0II and coefficients A1 , A2 ,...,A11 are functions of the spherical coordinates (r, θ, ϕ), rigidity R of GCR particles and time t∗ .
3
Method of the Numerical Solution of the Transport Equation
Solution of the full transport equation (3) is very complicated, even, after some simplification e.g.: 1. if
∂f ∂R
= 0, Eq. (3) is a parabolic time-dependent 3-D equation ∂f M = f, ∗ ∂t A0
(4)
∂f 2. if ∂t ∗ = 0, Eq. (3) is a stationary, but it remains as a 3-D parabolic problem with respect to the rigidity R
∂f M =− f. ∂R A11
(5)
The full model describing short period changes of the GCR intensity (e.g. Forbush decrease) requires the non stationary Parker’s transport equation (3) containing the simultaneous dependence of the GCR intensity on the particles rigidity and time. The equation (1) is not possible to solve analytically in general form without the extensive simplifications which makes the model to poor to describe the processes in the interplanetary space during the Fd. So, there is necessary to solve numerically the appropriate model based on the Eq. (3). Unfortunately, up to present there is no standard numerical method of solution of the partial differential equation of the form (3) which could take into consideration simultaneous dependence on time and on the GCR particles rigidity. We propose an approach to the numerical solution of this type of partial differential equations, as follows: First, we consider the equation (5) and transform it by the implicit finite difference scheme method [17] to the system of the algebraic equations: 2A1 + Δr1 A7 2A2 + Δθ1 A8 n+1 + fi,j+1,k Δr1 (Δr2 + Δr1 ) Δθ1 (Δθ2 + Δθ1 ) 2A3 + Δϕ1 A9 n+1 + fi,j,k+1 (6) Δϕ1 (Δϕ2 + Δϕ1 ) 2A1 2A2 2A3 A11 n+1 + fi,j,k ) (− − − + A10 − Δr1 Δr2 Δθ1 Δθ2 Δϕ1 Δϕ2 ΔR A4 A5 n+1 n+1 + fi+1,j+1,k + fi+1,j,k+1 (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) A A4 6 n+1 n+1 + fi,j+1,k+1 + fi+1,j−1,k (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) (Δr1 + Δr2 )(Δθ1 + Δθ2 ) n+1 fi+1,j,k
Numerical Solution of the Second Order Partial Differential Equation
109
−A5 −A6 n+1 + fi,j+1,k−1 (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) −A4 −A5 n+1 n+1 + fi−1,j+1,k + fi−1,j,k+1 (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) −A6 2A1 − Δϕ2 A7 n+1 n+1 + fi,j−1,k+1 + fi−1,j,k (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) Δϕ2 (Δϕ2 + Δϕ1 ) 2A − Δθ A 2A − Δϕ 2 2 8 3 2 A9 n+1 n+1 + fi,j,k−1 + fi,j−1,k Δθ2 (Δθ2 + Δθ1 ) Δϕ2 (Δϕ2 + Δϕ1 ) A4 A5 n+1 n+1 + fi−1,j−1,k + fi−1,j,k−1 (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) A6 A11 n+1 n + fi,j−1,k−1 + fi,j,k = 0. (Δϕ1 + Δϕ2 )(Δθ1 + Δθ2 ) ΔR n+1 + fi+1,j,k−1
n The bottom index (i, j, k) in fi,j,k denotes the point of the grid in the space (r, θ, ϕ) a (grid is 91x91x72x25x81; 91 steps in distance, 91 - in heliolatitudes, 72 - in heliolongitudes and 25 - in rigidities) and the upper index n denotes the rigidity layer. As mentioned above, the Eq. (5) is parabolic type problem with respect to the rigidity R. So, to solve the Eq. (5) the initial and boundary conditions [18,19,20,21] were used:
– initial condition f |R0 =150GV = 1 which means that R0 = 150GV is the higher limit of unmodulated rigidity of the GCR particles; – the boundary conditions with respect the distance r: f |r=30AU = 1, ∂f ∂r |r=0 = ∂f o = o = 0 and heliolongitude ϕ : f |ϕ=ϕ = 0, heliolatitude θ: ∂f | | θ=0 θ=180 1 ∂θ ∂θ f |ϕ=ϕL+1 , f |ϕ=ϕ−1 = f |ϕ=ϕL−1 . Then the system of equations (6) was solved by the Gauss-Seidel iteration method [17]. This is the first stage of solutions of full Eq. (3). Namely we find solutions for each layer of rigidity R (R1 = 130GV, R2 = 110GV,..., RL = 10GV) for the stationary case. After, these solutions for each rigidity layers are considered as an initial conditions for the non stationary case (Eq. (3)) for the given rigidity R and at time t∗ = 0 f (r, θ, ϕ, R, t∗ )|t∗ =0 = f (r, θ, ϕ, R).
(7)
Then the equation (3) (by the same method like Eq. (5)) was transformed to the system of the algebraic equations: 2A1 + Δr1 A7 2A2 + Δθ1 A8 t+1 + fi,j+1,k Δr1 (Δr2 + Δr1 ) Δθ1 (Δθ2 + Δθ1 ) 2A + Δϕ A 3 1 9 t+1 + fi,j,k+1 (8) Δϕ1 (Δϕ2 + Δϕ1 ) 2A1 2A2 2A3 A0 t+1 + fi,j,k (− − − + A10 − ) Δr1 Δr2 Δθ1 Δθ2 Δϕ1 Δϕ2 Δt∗ A4 A5 t+1 t+1 + fi+1,j,k+1 + fi+1,j+1,k (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) t+1 fi+1,j,k
110
A. Wawrzynczak and M.V. Alania
A6 A4 t+1 + fi+1,j−1,k (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) (Δr1 + Δr2 )(Δθ1 + Δθ2 ) −A5 −A6 t+1 t+1 + fi+1,j,k−1 + fi,j+1,k−1 (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) −A4 −A5 t+1 t+1 + fi−1,j+1,k + fi−1,j,k+1 (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) −A 2A1 − Δϕ2 A7 6 t+1 t+1 + fi,j−1,k+1 + fi−1,j,k (Δθ1 + Δθ2 )(Δϕ1 + Δϕ2 ) Δϕ2 (Δϕ2 + Δϕ1 ) 2A2 − Δθ2 A8 2A3 − Δϕ2 A9 t+1 t+1 + fi,j−1,k + fi,j,k−1 Δθ2 (Δθ2 + Δθ1 ) Δϕ2 (Δϕ2 + Δϕ1 ) A4 A5 t+1 t+1 + fi−1,j−1,k + fi−1,j,k−1 (Δr1 + Δr2 )(Δθ1 + Δθ2 ) (Δr1 + Δr2 )(Δϕ1 + Δϕ2 ) t+1 + fi,j+1,k+1
t+1 + fi,j−1,k−1
n
n+1
fi,j,k − fi,j,k A6 A0 t + fi,j,k + A11 = 0, (Δϕ1 + Δϕ2 )(Δθ1 + Δθ2 ) Δt∗ ΔR
n where bottom index (i, j, k) in fi,j,k denotes the point of the grid in the space (r, θ, ϕ) and the upper index n denotes the rigidity layer; simultaneously the t ) the time layer. The system of equations (8) corresponding upper index t (fi,j,k to non stationary case (Eq. (3)) was solved by the Gauss-Seidel iteration method for each rigidity layer (Fig. 1).
SCHEMA I R0=150GV t=0
ft+1(i,j,k)
fn(i,j,k)=1 R1=130 GV t=Δ Δt
R1=130GV t=1
Ra= R0-aΔ ΔR t=Δ Δt
Ra= R0-aΔ ΔR t=1
RL=10 GV t=Δ Δt
RL=10 GV t=1
R1=130GV t=0
fn+1(i,j,k) Ra=R0-aΔ ΔR t=0
fn+a(i,j,k)
RL=10GV t=0
fL(i,j,k)
Fig. 1. The first numerical scheme applied in the solution of the three dimensional non stationary transport equation (3)
The proposed model of the short time changes (Fd) of the GCR intensity is stipulated by the changes of the diffusion coefficient K taking place in the disturbed vicinity of the interplanetary space [18,19,20,21] corresponding to the experimental data observed 9-20 September 2005 by neutron monitors on
Numerical Solution of the Second Order Partial Differential Equation
111
GCR intensity [%]
0 -2 -4 -6 -8 Calgary Halekala Thule Nagoya N0VV
-11 11/09
09/09
13/09
15/09
17/09
19/09
Date, Year 2005
Fig. 2. Temporal changes of the GCR intensity (smoothed over 3 days) registered by Calgary, Halekala and Thule neutron monitors and Nagoya N0VV muon telescope in the period of 9-20 September 2005 corresponding to the effective rigidities: Calgary(Canada), Thule(Greenland) ∼ 10 GV, Halekala (USA)∼ 15 ÷ 20 GV, and Nagoya (Japan)∼ 25 ÷ 35GV
GCR intensity [%]
the Earth surface (Fig. 2). The diffusion coefficient including in the model has a form: K = K0 K(r)K(R, ν(t∗ )), where K0 = 4.5 × 1021 cm2 /s (value of normalization of diffusion coefficient for rigidity of 10GV), K(r) = 1 + 0.5r (dependence of the diffusion coefficient on the distance r), K(R, ν(t∗ )) = 5t∗ −0.5 1.5 ∗ R1.2−35( 3 ) Exp(−8(t −0.1)) . Solutions of the transport Eq. (3) for different GCR particles rigidities are presented in Fig. 3. We see from Fig. 3 that the expected temporal changes of the GCR intensity shows the similar changes as in experimental data (Fig. 2). 0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10
R=130GV R=60GV R=30GV R=18GV R=10GV 0
2
4
6
8 days
10
12
14
Fig. 3. Changes of the expected GCR intensity for different GCR particles rigidities obtained as the solutions of the transport Eq. (3) based on the I numerical scheme (Fig. 1)
Using the above presented transformations (6) and (8) the solution of the equation (3) can be found by the another way, too. Namely, we find solutions for each layer of rigidity R (R1 = 130GV, R2 = 110GV,..., RL = 10GV) for the stationary case (Fig. 4). After, we find solution for the non stationary case for layer R1 = 130 GV (as before), and then solution corresponding to the times Δt, 2Δt, 3Δt, ..., t using as the initial conditions for all other rigidity layers as it is shown by arrows in Fig. 4. Results of solution of the Eq. (3) based
112
A. Wawrzynczak and M.V. Alania SCHEMA II R0=150GV t=0
fn(i,j,k)=1
ft(i,j,k) R1=130 GV t=Δ Δt
R1=130 GV t=1
R1=130GV t=0
fn+1(i,j,k) ΔR Ra=R0-aΔ t=Δ Δt
Ra=R0-aΔ ΔR t=0
Ra=R0-aΔ ΔR t=1
fn+a(i,j,k) RL=10 GV t=1
RL=10 GV t=Δ Δt
RL=10GV t=0
fL(i,j,k)
GCR intensity [%]
Fig. 4. The second numerical scheme applied in the solution of the three dimensional non stationary transport equation (3)
0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10
R =60GV sch I R=60GVsch II R =30GV sch I R=30GVsch II R =10GV sch I R=10GVsch II 0
2
4
6
8 days
10
12
14
Fig. 5. Changes of the expected amplitudes of the Fd of the GCR intensity for different GCR particles rigidities obtained as the solutions of the transport Eq. (3) for the Fd based on the I and II numerical scheme (Fig. 1 and Fig. 4)
on the II numerical scheme (Fig. 4) are presented in Fig. 5. Fig. 5 shows that results of solutions by the II numerical scheme coincides with the results obtained by I numerical scheme in the scope of accuracies of numerical solutions (Fig. 5). This comparison, to some extant, gives a possibility to judge about the valuedness of our solutions. However, in spite of a good enough coincidence of the numerical solutions obtained by two different schemes, there exists a natural question about the valuedness of the solutions. For a such very much complicated Eq. (3) (time and rigidity dependent 3-D second order partial differential equation) we have not any theoretical method allowing to judge about the valuedness of the solutions found by the numerical methods. So, in this kind of cases only a comparison of results of solutions with the experimental data (Fig. 6) should play a decisive role. In Fig. 6 are presented results of solutions of the Eq. (3) corresponding to the rigidity of GCR particles of 10GV based on the I (Fig. 1)
Numerical Solution of the Second Order Partial Differential Equation
113
GCR intensity [%]
0 -2 -4 -6 -8 Calgary R=10 GV sch I R=10 GV sch II
-10 -12 0
5
10 days
15
20
Fig. 6. Changes of the expected GCR intensity for rigidity 10 GV obtained as the solutions of the the transport Eq. (3) and the GCR intensity registered by Calgary neutron monitor in 9-20 September 2005 (Fig. 2)
and II (Fig. 4) numerical scheme. There are also presented the experimental data of the changes of the Fd of the GCR intensity observed by Calgary neutron monitor (corresponding as well to the rigidity of 10 GV) in 9-20 September 2005 (Fig. 2). We show that there are good enough coincidences of the profiles and levels of the expected intensity between the numerical solutions obtained by the I and II scheme on the one side (correlation coefficient equals 0.99 ± 0.05), and the both solutions with experimental data (correlation coefficient equals 0.99 ± 0.05 for first and second scheme), on the other. Thus, we state that the numerical approaching presented in this paper could successfully be relevant for the solution of the time and rigidity dependent 3-D second order partial differential equation describing the anisotropic diffusion of the GCR particles in the heliosphere.
4
Conclusions
A numerical scheme based on the implicit finite difference method of solution the time and rigidity dependent 3-D second order partial differential equation describing the anisotropic diffusion of the GCR particles in the heliosphere is proposed. We show an existence of the satisfactory correspondence between the results of the numerical solution and experimental data. We conclude that the proposed numerical method successfully could be used for the solution of the time and rigidity dependent 3-D second order partial differential equation describing different classes of GCR intensity variations stipulated due to anisotropic diffusion of the GCR particles in the heliosphere.
References 1. Parker, E.N.: Dynamics of the Interplanetary Gas and Magnetic Fields. Astrophys. J. 128, 664 (1958) 2. Jokipii, J.R., Thomas, B.: Effects of drift on the transport of cosmic rays. IV Modulation by a wavy interplanetary current sheet. Astrophysical Journal, Part 1 243, 1115–1122 (1981)
114
A. Wawrzynczak and M.V. Alania
3. Stone, E.C., Cummings, A.C., McDonald, F.B., et al.: Voyager 1 Explores the Termination Shock Region and the Heliosheath Beyond. Science 309, 2017 (2005) 4. Forbush, S.: World-Wide Cosmic-Ray Variations 1937-1952. Journal of Geophysical Research 59, 525 (1954) 5. Alania, M.V., Wawrzynczak, A., Bishara, A.A., et al.: Three Dimensional Modeling of the Recurrent Forbush Effect of Galactic Cosmic Rays. In: Proceedings 29th International Cosmic Ray Conference, Pune, India SH 2.6, vol. 1, pp. 367–370 (2005) 6. Krymski, G.F.: Diffusion mechanism of daily variation of galactic cosmic ray. Geomagnetism and Aeronomy 4(6), 977–986 (1964) 7. Parker, E.N.: The passage of energetic charged particles through interplanetary space. Planetary and Space Science 13, 9 (1965) 8. Axford, W.I.: Anisotropic diffusion of solar cosmic rays. Planet. Space Sci. 13, 115 (1965) 9. Gleeson, L.J., Axford, W.I.: Solar Modulation of Galactic Cosmic Rays. Astrophys. J. 149, 115–118 (1968) 10. Dolginov, A.Z., Toptygin, I.N.: Cosmic rays in the interplanetary magnetic fields. Ikarus 8(1-3), 54–60 (1968) 11. Jokipii, J.R.: Cosmic-Ray Propagation. I. Charged Particles in a Random Magnetic Field. Astrophysical Journal 146, 480 (1966) 12. Zhang, M.: A Markov Stochastic Process Theory of Cosmic-Ray Modulation. Astrophys. J. 513(1), 409–420 (1999) 13. Webber, W.R., Lockwood, J.A.: Voyager and Pioneer spacecraft measurements of cosmic ray intensities in the outer heliosphere: Toward a new paradigm for understanding the global modulation process: 2. Maximum solar modulation (19901991). J. Geophys. Res. 106(9), 323 (2001) 14. Caballero-Lopez, R.A., Moraal, H.: Limitations of the force field equation to describe cosmic ray modulation. J. Geophys. Res. 109, A01101 (2004) 15. Alania, M.V., Dzhapiashvili, T.V.: The Expected Features of Cosmic Ray Anisotropy due to Hall-Type Diffusion and the Comparison with the Experiment. In: 16th International Cosmic Ray Conference, vol. 3(80), p. 19 (1979) 16. Alania, M.V.: Stochastic Variations of Galactic Cosmic Rays. Acta Physica Polonica B 33(4), 1149–1165 (2002) 17. Kincaid, D., Cheney, W.: Numerical Analysis. WNT, Warsaw (2006) 18. Alania, M.V., Wawrzynczak, A.: On Peculiarities of The Anisotropic Diffusion During Forbush Effects of Galactic Cosmic Rays. Acta Phys. Polonica B 35(4), 1551–1563 (2004) 19. Wawrzynczak, A., Alania, M.V.: Peculiarities of galactic cosmic ray Forbush effects during October-November 2003. Adv. in Space Res. 35, 682 (2005) 20. Wawrzynczak, A., Alania, M.V., Modzelewska, R.: Features of the Rigidity Spectrum of Galactic Cosmic Ray Intensity during the Recurrent Forbush Effect. Acta Phys. Polonica B 37(5), 1667–1676 (2006) 21. Wawrzynczak, A., Alania, M.V.: Modeling of the recurrent Forbush effect of the galactic cosmic ray intensity and comparison with the experimental data. Adv. in Space Res. 41, 325–334 (2008)
Hardware Implementation of the Exponent Based Computational Core for an Exchange-Correlation Potential Matrix Generation Maciej Wielgosz, Ernest Jamro, and Kazimierz Wiatr AGH University of Science and Technology, al. Mickiewicza 30, 30-059 Krakow ACK Cyfronet AGH ul. Nawojki 11, 30-950 Krakow {wielgosz,jamro,wiatr}@agh.edu.pl Abstract. This paper presents an FPGA implementation of a calculation module for a finite sum of the exponential products (orbital function). The module is composed of several specially designed floating-point modules which are, fully pipelined and optimized for high speed performance. The hardware implementation revealed significant speed-up for the finite sum of the exponential products calculation ranging from 2.5x to 20x in comparison to the CPU. The orbital function is a computationally critical part of the Hartree-Fock algorithm. The presented approach aims to increase the performance of the part of the quantum chemistry computational system by employing FPGA-based accelerator. Several issues are addressed such as an identification of proper code fragments, porting a part of the Hartree-Fock algorithm to FPGA, data precision adjustment and data transfer overheads. The authors’ intention was also to make hardware application of the orbital function universal and easily attachable to different systems. Keywords: HPRC (High Performance Reconfigurable Computing), FPGA, quantum chemistry, floating-point.
1
Introduction
High performance computing has been recognized for many years as an area dominated only by microprocessors. The rapid increase of the logic resources of modern FPGAs (Field Programmable Gate Arrays) gives a huge opportunity to adopt hardware accelerators for calculations which involve processing a large volume of data. Such a vital rise in the computational capabilities of FPGAs has stimulated investigations to construct a reconfigurable hardware accelerator [1]. The proposed hardware module aims to speed up HPC chemistry and physics calculations and provide compatibility with the standard floating- point data formats. The RASC [2] platform has been chosen as a hardware accelerator because of the computational capabilities it offers for SMP (Symmetric Multiprocessing) supercomputers. Since achievable computation speed-ups strictly depends on the R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 115–124, 2010. c Springer-Verlag Berlin Heidelberg 2010
116
M. Wielgosz, E. Jamro, and K. Wiatr
selected part of the algorithm, the choice of it becomes a critical issue. Therefore the finite sum of exp() functions was found a proper candidate for hardware acceleration. This operation is predominant in scientific computations, therefore the described solution may be employed in many kinds of applications, (e.g. Quantum Monte Carlo)[3,4]. However, the implementation described hereby should be considered as a first step in hardware implementation of the exchange-correlation potential calculationimplementation. The ultimate goal is to design a hardware unit capable of generating an exchange-correlation potential matrix, which is one of the most computationally intensive routines within the Hartree-Fock calculations.
2
Algorithm
The time-independent Schroedinger equation for a system of N particles interacting via the Coulomb interaction is: ˆ = Eψ Hψ ˆ = H
N
N
(1) N
2 2 1 Zi Zj (− ∇i ) + 2m 2 4π i 0 |ri − rj | i=1 i=1
(2)
j=i
where: ψ is an N-body wavefunction, r denotes spatial positions and Z represents the charges of the individual particles. E denotes the energy of either the ground or an excited state of the system. Errors of approximations made in obtaining the wavefunction will be manifested in any property derived from the wave function. Where high accuracy is required, considerable attention must be paid to the derivation of the wavefunction and any approximations made. Therefore there has been careful investigation into what level of precision should be adopted in calculations to ensure satisfactory approximation. The general procedure of solving the Hartree-Fock equations is to make the orbitals self-consistent with the potential field they generate. It is achieved through an iterative trial-and-error computational process, thus the entire procedure is called the self-consistent field (SCF) method. The SCF procedure of solving the Hartree-Fock equation leads to the following set of eigenvalue equations in matrix formulation: F C − SCE = 0
(3)
F is the Fock-operator, C the matrix of the unknown coefficients, S the overlap matrix and E is the energy eigenvalues. All matrixes are of the same size. The Hartree-Fock equation solving is an iterative process. The coefficients (corresponding to an electric field) are used to build the Fock-operator F, with which the system of linear equations is solved again to get a new solution (a new electric field). The procedure is repeated until the solution no longer changes.
Hardware Implementation of the Exponent Based Computational Core
117
The Fock operator depends on orbitals, which in turn are its eigenfunctions. Therefore orbitals have to be calculated for each iteration of the whole HartreeFock algorithm. That is why the finite sum of exponential products contributes scientifically to the overall performance of the application. Orbital function is expressed by equation 4 [5]: 2 χklm (r) = rxk ryl rzm Ci Ni e−αi r (4) i
The algorithm (equ. 4) was split into several sections [8] to allow adoption of a modular design approach. Consequently, a fully pipelined structure was obtained.
Fig. 1. EP module block diagram
Three different units have been employed within the EP module [Fig. 1]. All of them are specially designed, fully pipelined floating-point modules [6,7] optimized to high speed performance. Input widths as well as their internal data path scales from single to double data precision. The input and output data format complies with the IEEE-754 floating-point standard. Nevertheless intermediate data representations employ different non-standard floating-point format in order to reduce hardware resources.
118
M. Wielgosz, E. Jamro, and K. Wiatr
1. Multiplier For double precision data format inputs mantissa is 53-bit (54-bit including leading one) wide, therefore the product width is 108-bit wide. Such a bitwidth is far beyond the required precision, therefore the LSBs of the product are usually disregarded [10]. Consequently the LSB’s part of the multiplier performs operations which are not used in the next arithmetic operations. As a result, in the proposed architecture, some of the LSBs logic is not implemented at all. This approach allows for significant area reduction. A more detailed description of the multiplier will be available in a separate paper. 2. Exponential function This is mixed table-polynomial implementation of the exp function [6,7], based on commonly known mathematical identities ex = 2x∗log2 (e) = 2xi ∗ exf where xi is an integer part of x ∗ log2 (e) and xf = x − xi /log2 (e) and ex1 +x2 = ex1 ∗ ex2
(5)
(6)
where x1 , x2 are different bit-fields of xf . 3. Accumulator This is a fully pipelined mixed precision floating point unit. It can process one datum per clock cycle due to its advanced and optimized structure. A more detailed description of the accumulator will be available in a separate paper.
3
Platform
The Altix 4700 series [11] is a family of multiprocessor distributed shared memory (DSM) computer systems that currently ranges from 8 to 512 CPU sockets (up to 1,024 processor cores) and can accommodate up to 6TB of globally shared memory in a single system while delivering a teraflop of performance in a smallfootprint rack. The RASC module communicates with the Altix system in the same manner as any CPU does, i.e. employing NUMALink interconnection. Data transfer between the FPGA and NUMALink bus is realized by an applicationspecific integrated circuit (ASIC) denoted as TIO. SGI RASC RC100 Blade consists of two Virtex-4 LX 200 [9] FPGAs, with 40 MB of external SRAM logically organized as two 16MB blocks (as shown in Fig. 2) and an 8MB block. Each QDR (Quad Data Rate) SRAM block is capable of transferring 128-bit data every clock cycle (at 200MHz) both for reads and writes. The RC100 Blade is connected using the low latency NUMALink interconnect to the SGI Altix 4700 Host System, for a rated peak bandwidth of 3.2GB per second each direction. The RASC algorithm execution procedure, from the processor’s perspective, is composed of several functions which reserve resources, queue commands and perform other preparation steps. The resource reservation procedure,
Hardware Implementation of the Exponent Based Computational Core
119
Fig. 2. RASC architecture and the host-fpga data link
once conducted, allows many runnings of the algorithm - which amounts to a huge time savings, since the procedure takes approximately 7.5 ms. Therefore optimal structure of the hardware-accelerated application should contain as few initializations of the FPGA chips as possible so the FPGAs configuration time will be only a fraction of the algorithm execution time.
4
Implementation and Acceleration
The EP module implemention results on the RASC [2] platform are presented below. Core services, provided by SGI together with the RASC module, is an extra logic incorporated in an FPGA which provides communication with the on-board hub. Units which the EP module is built of are strongly parametrized, therefore resources occupation varies depending on the data width (different calculation precision) of the module. Table 1. Implementation results of the EP module - single precison Implementation results # 4-input LUT # flip-flops # 18-Kb BRAMs EP module alone 2229 (1%) 1975(1%) 2(0.006%) EP module with the core services 11560 (7%) 15922(9%) 25(7%)
Exp() function and the accumulator module consume the highest amount of resources, due to their complexity. The overall pipeline latency of the exp module is 33 clock cycles for the single precison data format. The latency strongly depends on the width of input data.
120
M. Wielgosz, E. Jamro, and K. Wiatr Table 2. Implementation results of the EP module - double precison
Implementation results # 4-input LUT # flip-flops # 18-Kb BRAMs EP module alone 8684 (4.8%) 7891(4%) 6(0.02%) EP module with the core services 18015 (10%) 21838(12%) 29(8%) Table 3. Implementation results of the elementary modules - single precision Implementation results # 4-input LUT Exp() 820 (0.5%) Multiplier 549 (0.3%) Accumulator 860 (0.5%)
# flip-flops # 18-Kb BRAMs 920(0.5%) 2(0.006%) 444(0.25%) 0 605(0.4%) 0
Table 4. Implementation results of the elementary modules - double precison Implementation results # 4-input LUT Exp() 5025 (3%) Multiplier 2316 (1%) Accumulator 1343 (0.5%)
# flip-flops # 18-Kb BRAMs 5223 (3%) 6 (1.8%) 1850(1%) 0 818(0.4%) 0
Table 5. Delays introduced by the EP module’s components - single precision Module Multiplier Exp() Accumulator EP module Pipeline delay [clk] 4 21 8 33 Table 6. Delays introduced by the EP module’s components - double precision Module Multiplier Exp() Accumulator EP module Pipeline delay [clk] 5 30 10 45
It is worth underlining that all the modules have adjustable widths of data paths within the range from single to double precision. The Hartree-Fock algorithm is executed differently, depending on the set of molecules it is employed for. This, in turn impacts the precision requirements at different stages of computations. It may happen that for a certain chemical substance some sections of the algorithm process much more data than others. Besides, in the Hartree-Fock algorithm, the result precision increases (error decreases) with every algorithm iteration, therefore the calculation precision may be somehow scaled with the iteration number. Utilizing FPGAs in Hartree-Fock computations allows for the adjustment of data buses widths within the system so the proper precision is maintained across all the stages of the algorithm execution. The FPGA configuration time (7.5 ms) is significantly shorter than the overhead introduced by the oversized data format. It may appear that some sections of calculations can be successfully computed at single precision (or any other precision, e.g. 48-bits),
Hardware Implementation of the Exponent Based Computational Core
121
saving a huge amount of resources that would otherwise be wasted if double precision data format was adopted. A hardware approach allows gradual switching from single to double data format together with the algorithm’s rising demand for precision.
Fig. 3. Example of the precision adjustment within the EP module
All the data connections within the EP module are adjustable so it is possible to change them in order to meet precision requirements. Different approach can be taken to parameterize the EP module but the most common is a topdown method, which involves gradual modification of the inter module interface’s width. Eventually number of guard bits of the basic components is established. For instance ( Fig. 3) optimal data width with regards to the precision of the interface between first multiplier and that exp() unit is 37 bits. The EP module is intended to be a part of a larger system performing exchange-correlation potential calculation, so some preliminary tests were carried out in order to determine top achievable performance of the module on the RASC platform. Each of the EP modules performs calculations in single precision so it is possible to aggregate two of them to increase overall throughput. Additionally due to the algorithm’s structure some of the data (atom base coefficients) is used multiple times. So only a single 32-bit data chunk is streamed in and out the FPGA. This in turn allows to aggregate up to four EP module (single precision) on the
122
M. Wielgosz, E. Jamro, and K. Wiatr
Fig. 4. Configuration of EP modules on the RASC platform
single FPGA. This holds as the parallization degree is limited by the external memory data transfer rate rater than the FPGA resources. Single exponential operation calculation together with two-directional data transfer takes roughly 8 ns on the RASC platform whereas the same operation executed on the Itanium takes roughly 20 ns for a highly optimized code. Both the values were obtain as the result of a conducted tests. Table 7. Acceleration results Number of EP modules [RASC/Itanium] speed-up 1 2.5 2 5 4 10
Employing both Xilinx Virtex 4 chips available on the RASC platform leads 20x speed-up comparing to Itanium 2 1.6 GHz processor. There is a huge difference between optimized and a simple code in execution time. Tables 8 and 9 present different degree of optimization applied to the processor source code. What shows how such an acceleration evaluation can somethimes be misleading, especially when hardware performance is not compared with the optimized software one.
Hardware Implementation of the Exponent Based Computational Core
123
Table 8. Acceleration results as the results of the processor source code optimization Itanium algorithm execution Not optimized C code Not optimized C code with compiler highest effort (-O3 compiler flag set) Fully optimized C code
Itanium a single operation execution time [us] 0.2 0.16
[RASC/Itanium] up 25 14.6
0.02
2.5
speed-
Table 9. C implementation of the orbital calculation routine Not optimized C code
Optimized C code
for(t=0;t<size;t++) for(t=0;t<size;t++) result+= a[t]*exp(_table(b[t]*c[t]); b_c_table[t] = b[t]*c[t]; for(t=0;t<size;t++) exp_table[t] = exp(b_c_table[t]); for(t=0;t<size;t++) result = result + a[t]*exp_table[t];
5
Summary
There were some other papers published regarding Hartree-Fock(HF) calculations in FPGAs [12,13]. But the solution presented in [12] aims to accelerate ERI (Compute electron repulsion integrals) computations and additionally the paper deals with various issues regarding coarse-grain parallelism aspects of the implementation. Therefore it does not directly correspond to the orbital function calculation for exchange-potential generation. It is worth noting that a finite sum of the exponential products (eq.4) is, as a computational routine, ubiquitous in scientific calculations because of its universal function. Many different processes in the real world can be described by exponential function. Combination of the exp() leads to an even more uniform formula. Performance tests revealed that the FPGA is much faster than a processor - even with data stored in the processor’s memory and the full optimization provided. The module can be scaled from single to double precision to adjust to calcualtion profile. It is worth taking into consideration that the Hartree-Fock algorithm, due to its iterative execution, allows the adoption of gradually adjustable data precision, which will give a speed boost to FPGAs in the final application. So there is still a huge potential in hardware implementation of quantum chemistry computational routines.
124
M. Wielgosz, E. Jamro, and K. Wiatr
References 1. Underwood, K.D., Hemmert, K.S., Ulmer, C.: Architectures and APIs: assessing requirements for delivering FPGA performance to applications. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC 2006, Tampa, Florida, November 11-17, p. 111. ACM, New York (2006) 2. Silicon Graphics, Inc. Reconfigurable Application-Specific Computing User’s Guide, Ver. 005, SGI (January 2007) 3. Gothandaraman, A., Peterson, G., Warren, G., Hinde, R., Harrison, R.: FPGA acceleration of a quantum Monte Carlo application. Parallel Computing 34(4-5), 278–291 (2008) 4. Gothandaraman, A., Warren, G.L., Peterson, G.D., Harrison, R.J.: Reconfigurable accelerator for quantum Monte Carlo simulations in N-body systems. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC 2006, Tampa, Florida, November 11-17, p. 177. ACM, New York (2006) 5. Mazur, G., Makowski, M.: Development and Optimization of Computational Chemistry Algorithms. In: KDM 2008, Zakopane, Poland (March 2008) 6. Wielgosz, M., Jamro, E., Wiatr, K.: Highly Efficient Structure of 64-Bit Exponential Function Implemented in FPGAs. In: Woods, R., Compton, K., Bouganis, C., Diniz, P.C. (eds.) ARC 2008. LNCS, vol. 4943, pp. 274–279. Springer, Heidelberg (2008) 7. Jamro, E., Wielgosz, M., Wiatr, K.: FPGA implementation of 64-bit exponential function for HPC. In: FPL Proceedings, FPL Netherlands, August 27-29 (2007) 8. Omondi, A.R.: Computer Arithmetic Systems. Prentice Hall, Cambridge (1994) 9. Xilinx Virtex-4 Family Overview, http://www.xilinx.com/support/documentation/data_sheets/ds112.pdf 10. Parhi, K.K., Chung, J.G., Lee, K.C., Cho, K.J.: Low-error fixed-width modified booth multiplier, US Patent: 957244 11. SGI Altix 4700, http://www.sgi.com/products/servers/altix/4000/ 12. Ramdas, T., Egan, G.K., Abramson, D., Baldridge, K.: Converting massive TLP to DLP: a special-purpose processor for molecular orbital computations. In: Proceedings of the 4th International Conference on Computing Frontiers, CF 2007, Ischia, Italy, May 7-9, pp. 267–276. ACM, New York (2007) 13. Kandaswamy, M., Kandemir, M., Choudhary, A., Bernholdt, D.: Optimization and Evaluation of Hartree-Fock Application’s I/O with PASSION. In: Proceedings of the ACM/IEEE Conference on Supercomputing (SC), November 1997, pp. 31–50 (1997)
Parallel Implementation of Conjugate Gradient Method on Graphics Processors Marcin Wozniak, Tomasz Olas, and Roman Wyrzykowski Institute of Computational and Information Sciences, Czestochowa University of Technology Dabrowskiego 74, 42-201 Czestochowa, Poland {marcell,olas,roman}@icis.pcz.pl
Abstract. Nowadays GPUs become extremely promising multi/manycore architectures for a wide range of demanding applications. Basic features of these architectures include utilization of a large number of relatively simple processing units which operate in the SIMD fashion, as well as hardware supported, advanced multithreading. However, the utilization of GPUs in an every-day practice is still limited, mainly because of necessity of deep adaptation of implemented algorithms to a target architecture. In this work, we propose how to perform such an adaptation to achieve an efficient parallel implementation of the conjugate gradient (CG) algorithm, which is widely used for solving large sparse linear systems of equations, arising e.g. in FEM problems. Aiming at efficient implementation of the main operation of the CG algorithm, which is sparse matrix-vector multiplication (SpMV ), different techniques of optimizing access to the hierarchical memory of GPUs are proposed and studied. The experimental investigation of a proposed CUDA-based implementation of the CG algorithm is carried out on two GPU architectures: GeForce 8800 and Tesla C1060. It has been shown that optimization of access to GPU memory allows us to reduce considerably the execution time of the SpMV operation, and consequently to achieve a significant speedup over CPUs when implementing the whole CG algorithm.
1
Introduction
Nowadays GPUs become extremely promising multi/many-core architectures for a wide range of general-purpose applications demanding high-intensive numerical computations [5,7]. Basic features of these architectures include utilization of a large number of relatively simple processing units which operate in the SIMD fashion, as well as hardware supported, advanced multithreading. For example, NVIDIA Tesla C1060 is equipped with 240 cores, delivering the peak performance of 0.93 TFLOPS. The tremendous step towards a wider acceptation of GPUs in general-purpose computations was development of software environments which made possible to program GPUs in high-level languages [15]. The new software developments, such as NVIDIA CUDA [7,8] and OpenCL [9], allow programmers to implement R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 125–135, 2010. c Springer-Verlag Berlin Heidelberg 2010
126
M. Wozniak, T. Olas, and R. Wyrzykowski
algorithms on existing and future GPUs much easier. For example, the CUDA environment makes it possible to develop applications for both the Windows and Linux operating systems, giving access to a well-designed programming interface in the C language. On a single GPU, it is possible to run several CUDA and graphics applications concurrently. The CUDA architecture includes two modules dedicated respectively to a general-purpose CPU and graphics processor. This allows for utilization of GPU as an application accelerator, when a part of the application is executed on a standard processor, while another part is assigned to GPU. However, the utilization of GPUs in an everyday practice is still limited. The main reason is the necessity of deep adaptation of implemented applications and algorithms to a target architecture [21], in order to match its internal characteristics. This paper deals with the problem how to perform such an adaptation efficiently for the conjugate gradient (CG) method [1,10], which is widely applied for solving iteratively large sparse linear systems of equations. Such systems arise e.g. when modeling numerically different problems described by partial differential equations, with the use of the Finite Element Method (FEM) [21]. The material of this paper is organized in the following way. In Section 2, the CG method is introduced, as well as key elements of the NVIDIA GPUs architecture which are important for efficiency of CUDA applications [6,8]. Among them are: organization of thread execution, and memory architecture. Section 3 is devoted to methods of implementing the sparse matrix-vector multiplication, which determines the efficiency of the whole CG algorithm. To speed up the multiplication, we propose a way how to optimize access of threads to memory. Some performance results of numerical experiments carried out on NVIDIA GeForce 8800 GTX and NVIDIA Tesla C1060 are presented in Section 4.
2 2.1
Conjugate Gradient Algorithm and GPU Architecture Conjugate Gradient Method
The CG method is one of the most widely used iterative methods dedicated to solution of linear systems of equations with sparse, symmetric and positive-definite matrix of coefficients [1,10,21]. A version of the CG algorithm implemented in this work is shown in Fig. 1 This version was derived based on the assumption that subsequent residue vectors ri+1 are mutually orthogonal, while subsequent search directions pi are mutually conjugate. The most time-consuming part of this algorithm is multiplication of the sparse matrix A by a dense vector pi . That is why, we focus on the problem how to optimize the execution of this operation. 2.2
GPU Architecture
Graphics processors operate in accordance with the SIMT (Single Instruction Mutiple Threads) model [5,15]. In this model, all the GPU threads execute the
Parallel Implementation of Conjugate Gradient Method on GPU
127
r0 = b − Ax0 p0 = r0 j=1 repeat until stop criterion is satisfied { αj = (rj , rj )/(Apj , pj ) xj+1 = xj + αj pj rj+1 = rj − αj Apj βj = (rj+1 , rj+1 )/(rj , rj ) pj+1 = rj+1 + βj pj j =j+1 } Fig. 1. CG algorithm implemented in this work
same code, called kernel, on different data. The SIMT model is especially suitable for algorithms operating on large data arrays, such as operations performed on vectors and matrices in the CG algorithm. Fig. 2 illustrates the architecture of the NVIDIA GT200 graphics processor. In this case, the smallest execution unit (or core) is a Thread Processor (TP). TP units are connected in groups of 8 cores. Each group creates a Streaming Multiprocessor (SM), while three SMs together with 16 KB of L1 cache create a Thread Processing Cluster (TPC). The resulting architecture of GT200 consists of 240 TP cores, partitioned into 10 TPCs.
Fig. 2. Architecture of GT200 (source NVIDIA)
128
2.3
M. Wozniak, T. Olas, and R. Wyrzykowski
Organization of Thread Execution in GPU
The important feature of GPU architectures is organization of thread execution. Because of hardware support, creating, deleting and switching threads is extremely fast. Threads are connected in 1D, 2D or 3D blocks (maximum 512 threads in a single block). Each block is assigned to a particular multiprocessor. Threads within a block are able to communicate and synchronize, but the synchronization across different blocks is not allowed. For a particular multiprocessor, the number of active threads is much greater than the number of TPs. That is why, blocks are divided into portions, called warps, each consisting of 32 threads. Each time, every multiprocessor executes a certain warp; the rest of warps are suspended. Such an organization of thread execution allows for tolerating latencies caused by access to the GPU global memory (300 cycles). When a certain warp waits for data, it is suspended, and the multiprocessor is switching to the execution of another warp. Since synchronization of threads is possible only within blocks, the only way for synchronizing all the threads is to allow for finishing the execution of a kernel code. 2.4
Memory Architecture in GPU
The memory organization in graphics processors is rather complex (Fig. 3). After being initialized, each thread obtains access to a pool of registers and Local Memory. Moreover, threads within a particular block share 16 KB of the buffered, fast Shared Memory, which is used for their communications. For each thread assigned to a certain multiprocessor, the amount of available Local Memory and registers depends on the number of threads in a block - the high number of threads share resource of the multiprocessor the lower amount of resources correspond to each thread. GPUs are also equipped with a Global Memory accessible by all the threads (read and write). However, access to this memory is rather expensive. Another types of GPU memory are Constant Memory and Texture Memory. Their access time is rather short, but threads are allowed only to read from these memories. For all these three types of memory, their content is preserved between subsequent invocations of kernels; this content is accessible for all the threads running on a graphics card. In order to optimize the memory access in GPUs, it is advisable to use the Constant Memory and Texture Memory whenever possible. Also, the access to the Global Memory may be optimized by a suitable allocation of arrays in a way which allows threads to read data from contiguous memory areas.
3
Sparse Matrix-Vector Multiplication on GPUs
Because of irregular memory access, the efficient implementation of the multiplication of sparse matrix A by dense vector x (SpMV operation shortly) is a difficult problem for both traditional processors, and modern multi-/manycore
Parallel Implementation of Conjugate Gradient Method on GPU
129
Fig. 3. Memory organization of GPU (source NVIDIA)
architectures [11,19,20]. In order to optimize the efficiency of SpMV, a lot of storage formats were proposed for representing sparse matrices [13,16,14,17,19,22]. Some of them are suitable only for a certain type of matrices (e.g., Jagged Diagonal format for band matrices), or are heavily based on specific features of processor architecture (e.g., block formats allowing for efficient utilization of caches). In this work, we focus on the CSR format for its vewrsatility and popularity; this storage is then combined with the ELLPACK format. 3.1
CSR Format
The CSR format (Fig. 4) uses three arrays: r, c, and v. The arrays v and c contain respectively all non-zero elements of A-matrix, and column indexes of these non-zero elements, while elements of the array r point to first elements of subsequent rows of A-matrix in the previous arrays. Note that the number of non-zero elements in a particular row is determined by the difference between two consecutive elements of r. Having a matrix in this format, the result of SpMV is computed using a rather simple algorithm shown in Fig. 5. 3.2
ELLPACK Format
The row version of the ELLPACK format (Fig. 6) uses two 2D arrays: As and IA, each containing n × cmax elements, where cmax denotes the maximum number of non-zero elements in rows of A-matrix. Each row of the array As stores non-zero elements of the corresponding row of A-matrix, while the array IA contains column indexes for these non-zero elements. Note that in the array IA, indexes with values of 0 will appear only in the first row. All other values
130
M. Wozniak, T. Olas, and R. Wyrzykowski
Fig. 4. An example 6 × 6 matrix stored in CSR format
Fig. 5. Computing result b of SpMV operation when matrix is stored in CSR format
of 0, which may appear in any other row of the array IA, are not treated as column indexes, but they deliver information that the list of non-zero elements in the corresponding row of A-matrix has been already exhausted.
Fig. 6. An example 6 × 6 matrix stored in ELLPACK format
Parallel Implementation of Conjugate Gradient Method on GPU
3.3
131
Implementation of SPMV on GPU
For the CSR format, the simplest method of implementing the SpMV operation on GPUs is parallelization of the main loop of the algorithm (Fig. 7), when each thread is responsible for multiplying non-zero elements of a particular row of A-matrix by corresponding elements of the input vector. However, due to the irregular access to elements of both the matrix and vector, this scheme, known as ”1 thread per row”, slows down the resulting implementation [6].
Fig. 7. GPU implementation of SPMV for CSR format (“1 thread per row” scheme)
Other methods of implementing the SpMV operation on GPUs for the CSR format rely on assigning each row of A to more than one thread. The final result for a particular row is then computed in parallel by summing partial results within the Shared Memory, using the parallel reduction operation. Two implementations based on this scheme, known as ”1 warp per row” and ”half warp per row”, were proposed in [3] and [2], where each row is distributed across a warp containing 32 and 16 threads, respectively. In these implementations, in order to optimize access to the GPU memory, rows are filled up with zeros, up to sizes equal to multiples of the warp size. In this way, it is possible to achieve the access to contiguous areas of memory, which speeds up fetching elements of the matrix. The implementation available in the CUDA Data Parallel Primitives (CUDPP) Library [4] makes each thread responsible for computing the product of a single element of A-matrix by a corresponding element of vector. For a certain row of A, the obtained results are then summed up in parallel, using the Segmented Scan operation [12]. In this way, access to elements of A-matrix is optimized, while access to elements of vector is still irregular. For the ELLPACK format (row wersion), the GPU implementation of SpMV was proposed in [18]. It was shown that by filling up rows of A with zeros it is possible to optimize the GPU memory access for the ”1 thread per row” scheme.
132
M. Wozniak, T. Olas, and R. Wyrzykowski
In this work, we focus on the problem how to optimize the GPU implementation of the ”1 thread per row” scheme, using the CSR format. As it was shown before, the main problem here is the irregular memory access. To deal with this problem and optimize access to elements of A-matrix, we propose to reorganize the structure of this matrix represented in the CSR format by combining it with the ELLPACK storage. The proposed implementation is then compared with the solution available in the CUDPP library [4], as well as with solutions using the Global Memory and Texture Memory to implement the ”1 thread per row” scheme. When multiplying the sparse matrix represented in the standard CSR format, a particular thread fetches consecutive non-zero elements of the array v and their column indexes from the array c (see Fig. 4 and Fig. 8a). In the proposed, modified version of the CSR format, the sequence of non-zero elements of the matrix and their column indexed is changed in a way which allows all the threads to read the required data from contiguous and suitable allocated areas of memory. As a result, at the beginning of the array v we put all the first non-zero elements of all the rows of A; then this pattern is used for the second elements of all the rows, etc. This is illustrated in Fig. 8b. The identical modification is applied for the array c, while the array r is left without changes. After such a reorganization, each thread within a certain warp accesses data in a optimized way, fetching data from contiguous memory areas. Moreover, there is no need in communication and synchronization among threads, which also promotes increasing the computation speed.
Fig. 8. Patterns of GPU memory access for standard (a) and modified (b) CSR format
Parallel Implementation of Conjugate Gradient Method on GPU
133
Fig. 9. Speedups of different NVIDIA GF8800 implementations against CPU implementation
4
Performance Results
The performance experiments were carried out for the whole CG algorithm implemented in single precision on the following GPU platforms: (i) CUDA on NVIDIA GeForce 8800 GTX, (ii) CUDA on NVIDIA Tesla C1060. The speedups shown in Fig. 9 and Fig. 10 were determined for FEM matrices with different size, against the CPU implementation running on Intel Core2 Duo 2.33GHz, 4MB Cache. The CPU implementation used the standard CSR format, while in case of GPUs the following variants were investigated: 1. using the CUDPP library (marked as CUDPP), 2. standard CSR format and ”1 thread per row” scheme (marked as CSR), 3. variant 2) modified by utilization of the Texture Memory (CSR TEX),
Fig. 10. Speedups of different NVIDIA TESLA C1060 implementation against CPU implementation
134
M. Wozniak, T. Olas, and R. Wyrzykowski
4. using the modification of the CSR format proposed in this paper (CSR OPT), 5. variant 4) modified by utilization of the Texture Memory (CSR OPT TEX). Based on these experiments, we conclude that the proposed method of optimizing access to the Global Memory when implementing the SpMV operation on NVIDIA GPUs allows us to speed up the CG iterative algorithm as a consequence of reducing the execution time for the sparse matrix-vector multiplication.
Acknowledgment This work was supported by the Polish Ministry of Science and High Education under grant no. N516 4280 33.
References 1. Basermann, A., Reichel, B., Schelthoff, C.: Preconditioned CG methods for sparse matrices on massively parallel machines. Parallel Computing Journal 23(3), 381– 393 (1997) 2. Baskaran, M.M., Bordawekar, R.: Optimizing Sparse Matrix-Vector Multiplication on GPUs. IBM Research Report No. RC24704, W0812-047 (2009) 3. Bell, N., Garland, M.: Efficient Sparse Matrix-Vector Multiplication on CUDA. NVIDIA Tech. Report No. NVR-2008-004 (2008) 4. CUDPP: CUDA Data Parallel Primitives Library, http://gpgpu.org/developer/cudpp 5. Dokken, T., Hagen, T.R., Hjelmervik, J.M.: An Introduction to General-Purpose Computing on Programmable Graphics Hardware. In: Geometric Modelling, Numerical Simulation, and Optimization: Applied Mathematics at SINTEF, pp. 123– 161. Springer, Heidelberg (2007) 6. Fujimoto, N.: Faster matrix-vector multiplication on GeForce 8800GTX. In: Proc. IEEE Int. Symp. on Parallel and Distributed Processing - IPDPS 2008, pp. 1–8 (2008) 7. Nickolls, J., Buck, I., Garland, M., Skadron, K.: Scalable Parallel Programming with CUDA. Queue 6(2), 40–53 (2008) 8. NVIDIA CUDA Programming Guide 2.2, http://developer.download.nvidia.com/compute/cuda/2 2/toolkit/docs/ NVIDIA CUDA Programming Guide 2.2.pdf 9. OpenCL - The open standard for parallel programming of heterogeneous systems, http://www.khronos.org/opencl 10. Olas, T., Karczewski, K., Tomas, A., Wyrzykowski, R.: FEM Computations on Clusters Using Different Models of Parallel Programming. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2001. LNCS, vol. 2328, pp. 170–184. Springer, Heidelberg (2002) 11. Saad, Y.: SPARSKIT: A basic toolkit for sparse matrix computations, http://www-users.cs.umn.edu/~ saad/software/SPARSKIT/paper.ps 12. Sengupta, S., Harris, M., Zhang, Y., Owens, J.D.: Scan Primitives for GPU Computing. In: Proc. 22nd ACM SIGGRAPH/EUROGRAPHICS Symp. on Graphics Hardware, pp. 97–106 (2007)
Parallel Implementation of Conjugate Gradient Method on GPU
135
13. Smailbegovic, F.S., Gaydadjiev, G.N., Vassiliadis, S.: Sparse Matrix Storage Format. In: Proc. 16th Ann. Workshop on Circuits, Systems and Signal Processing, ProRisc 2005, pp. 445–448 (2005) 14. Stathis, P.T.: Sparse Matrix Vector Processing Formats. PhD Thesis, Delft University of Technology (2004), http://ce.et.tudelft.nl/publicationfiles/955_1_Thesis_P_T_Stathis.pdf 15. Strzodka, R., Doggett, M., Kolb, A.: Scientific computation for simulations on programmable graphics hardware. Simulation Modelling Practice and Theory 13(8), 667–860 (2005) 16. Tvrdik, P., Simecek, I.: A New Diagonal Blocking Format and Model of Cache Behavior for Sparse Matrices. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 164–171. Springer, Heidelberg (2006) 17. Tvrdik, P., Simecek, I.: A New Approach for Accelerating the Sparse Matrix-Vector Multiplication. In: Proc. 8th Int. Symp. on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC 2006, pp. 156–163. IEEE Computer Society, Los Alamitos (2006) 18. V´ azquez, F., Garz´ on, E.M., Mart´ınez, J.A., Fern´ andez, J.J.: Accelerating sparse matrix vector product with GPUs. In: Proc. 9th Int. Conf. Computational and Mathematical Methods in Science and Engineering, CMMSE, pp. 1081–1092 (2009) 19. Vuduc, R.W.: Automatic Performance Tuning of Sparse Matrix Kernels. PhD Thesis, University of California, Berkeley (2003), http://bebop.cs.berkeley.edu/pubs/vuduc2003-dissertation.pdf 20. Williams, S., Oliker, l., Vuduc, R., Shalf, J., Yelick, K., Demmel, J.: Optimization of sparse matrix-vector multiplication on emerging multicore platforms. Parallel Computing 35(3), 178–194 (2009) 21. Wyrzykowski, R.: PC-clusters and multicore architectures. EXIT, Warsaw (2009) (in Polish) 22. Wyrzykowski, R., Kanevski, J.: A Technique for Mapping Sparse Matrix Computations into Regular Processor Arrays. In: Lengauer, C., Griebl, M., Gorlatch, S. (eds.) Euro-Par 1997. LNCS, vol. 1300, pp. 310–317. Springer, Heidelberg (1997)
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES Eugeniusz Zieniuk and Agnieszka Boltuc Institute of Computer Science, University of Bialystok Sosnowa 64, 15-887 Bialystok, Poland {ezieniuk,aboltuc}@ii.uwb.edu.pl
Abstract. The paper presents an effective approach to solving boundary problems modeled by a nonlinear partial differential equation. An applied algorithm bases on: an iterative method used in order to find successive approximations to the solution of a boundary problem and PIES method as a tool for solving a boundary problem in each iteration step. The paper also contains some suggestions about an effective way of domains modeling and calculation of integrals over those domains. Authors also have performed an initial verification of a method on the basis of a linear equation and nonlinear equations with a various intensity of nonlinearity.
1
Introduction
A variety of technical problems have been modeled by means of partial differential equations such as Lapalces, Navier-Lame or Helmholtz equations. Frequently such practical problems, despite of their complexity, are modeled by linear differential equations. It causes simplification of solved problem and gives a guarantee of obtaining results. Problems defined in that way can be solved by means of various numerical methods: FEM [10], BEM [1,2] or mesh-free methods [6]. Each of them has some advantages and drawbacks, which are emphasized taking into account problems from different fields and with a different degree of difficulty. It must be mentioned that gainful feature of two first methods is existence of a very developed and easy available computer software. It is known, however, that in most realistic problems occurs nonlinearity. There is a question: can we use above mentioned methods for solving such kind of problems? Obviously it is possible, but is connected with different problems which complicate a task. Thus, considering classic BEM, which was popularized as a metod without domain discretization, for solving nonlinear problems (and not only) one has to calculate domain integrals. It is connected with necessity of division of an integration domain into smaller sub-regions called cells. As a result we have, from technical point of view, almost the same process as in FEM discretization of a domain by means of finite elements. In the research [7,8,9] carried out by authors a different approach to boundary problems (until now only linear) has been proposed. A basis of that approach is a method called parametric integral equation system (PIES) obtained on the R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 136–145, 2010. c Springer-Verlag Berlin Heidelberg 2010
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES
137
basis of classic boundary integral equation by its analytical modification [7]. That modification is concerned with elimination of discretization both a domain (as in BEM) and a boundary by including a boundary geometry (defined in a parametric way) directly in PIES kernels. Described approach was successfully applied to solving boundary problems modeled by various differential equations [7,8,9] and has given satisfactory and promising results. An effectiveness relates especially to: a number of data used for a boundary geometry modeling, a number of solved algebraic equations and an accuracy of obtained results. The purpose of this paper is to propose PIES method for solving 2D nonlinear problems, to analyze its advantages and drawbacks and to verify its reliability and efficiency considering also linear problems.
2
PIES in Linear and Nonlinear Boundary Problems
The following partial differential equation [6] is solved in the present paper ∇2 u(x) + εu(x)n = p(x), x ∈ Ω, ∂2 ∂x21
(1)
∂2 ∂x22
where ∇2 = + is the Laplaces operator, p(x) is a given source function, ε is a constant, and the domain Ω is enclosed by Γ = Γu ∪ Γp with boundary ∂u conditions u = u on Γu and ∂n ≡ p = p on Γp . Taking into account various values of n the partial differential equation (1) solved in the paper is linear (n = 1) or nonlinear (n = 1). Considering a linear equation there are two different possibilities. Both of them require an existence and knowledge of a fundamental solution of differential equation (1). In both situation we can use known solution of linear Laplaces equation [1]. However, that solution has to be modified taking into account an element εu(x), which is additional in relation to Laplaces equation. Other approach bases on an iterative process with the help of which we can approximate solution u(x) in finite number of steps. Taking into account universality of a proposed approach (taking a linear and a nonlinear problems into consideration) it is reasonable to apply the second approach. In connection with mentioned approach a new form of PIES for considered in the paper equation is following (it is a modification of known from [9] form of PIES for linear Poisson equation) 0.5ul (s1 ) =
n j=1
Sj Sj−1
∗
∗
{U lj (s1 , s)pj (s) − P lj (s1 , s)uj (s)}Jj (s)ds + + Ω
∗
U l (s1 , x) [εu(x)n − p(x)] dΩ(x),
where sl−1 ≤ s ≤ sl , sj−1 ≤ s ≤ sj ,Jj (s) =
(1)
∂Γj (s) ∂s
2
+
(2)
∂Γj (s) ∂s
(2)
2 0.5 .
138
E. Zieniuk and A. Boltuc
The first and second integrands from (2) are respectively fundamental and singular solutions for Laplaces equation (they are presented in an explicit form in [7]). ∗
Function U l from a domain integral from (2) takes the following form ∗
U l (s1 , x) = (1)
1 1 ln 0.5 , 2 2π η 1 + η 22
(3)
(2)
where η 1 = Γl (s1 ) − x1 , η 2 = Γl (s1 ) − x2 , whilst Γ (s) are parametric curves which describe a boundary (in the paper Bezier curves of the first and third degree are used [3]). In order to obtain wanted values of u(x) in a domain Ω one should know a form of an integral identity, which can be expressed as follows u(y) =
n j=1
Sj
Sj−1
∗
∗
{U lj (y, s)pj (s) − P lj (y, s)uj (s)}Jj (s)ds + + Ω
ˆ Ul∗ (y, x) [εu(x)n − p(x)] dΩ(x),
(4)
ˆ where function Ul∗ for domain integral takes the following form ˆ 1 Ul∗ (y, x) = 2π ln ˆ 1ˆ 0.5 , ηˆ1 = y1 − x1 , ηˆ1 = y2 − x2 , x1 , x2 ∈ Ω. η12 +η22
Analyzing above expressions (2,4) it can be seen, that there is a necessity of a domain integrals calculation. In connection with that fact one of the most important stages of performed in the paper researches is testing of a presented approach (on linear problems - as an initial verification of an algorithm, on nonlinear problems as a destination) and elaborating and verification of an effective way of calculation of mentioned integrals.
3
Numerical Solution of PIES Using an Iterative Method
Finally, a solution of PIES (2) leads to finding unknown functions uj (s) or pj (s) on each boundary segment of considered problem. Unknown functions are approximated by following expressions in each step of an iterative process pj (s) =
N
(k) (k)
pj fj (s), uj (s) =
k=0 (k)
(k)
where uj , pj
N
(k) (k)
uj fj (s),
(5)
k=0
are unknown coefficients on the segment j, N is a number of (k)
coefficients on the segment and fj (s) are the global base functions on individual segments (e.g. used in the paper Chebyshev polynomials of the first kind).
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES
139
After the substitution of (5) into (2) and application of the collocation method [4] we obtain an expression in the following general form 0.5ul (s1 ) =
n N j=1 k=0
(k)
{pj
Sj
Sj−1
∗
(k)
U lj (s1 , s) − uj + Ω
Sj
Sj−1
∗
(k)
P lj (s1 , s)}fj Jj (s)ds +
∗
U l (s1 , x) [εu(x)n − p(x)] dΩ(x).(6)
However, final solution of above equation requires special treatment, because a domain integral contains a nonlinear and unknown at the same time function u(x)n . An equation (6) written down in the form of an algebraic equations system can be presented as follows Hu = Gp + W,
(7)
where H = [H]M×M , G = [G]M×M (M = n × N ) are square matrices of elements expressed by integrals from equation (6), u and p are column matrices which contain coefficients of approximation series (5), whilst W is a column matrix of elements expressed by means of a domain integral from equation (6). After application boundary conditions and some transformations (moving known values to the one side and unknown to the other), an equation (7) takes the form of an algebraic equations system AX = F + W,
(8)
where vector X contains unknown coefficients of searched boundary approximating functions (5), whilst vector F is known and depends on given boundary conditions. The only problem is that in the case of considered problems (linear and nonlinear) the right-hand side (W) of an equation (8) is unknown. It depends on current value of solution ui (x) at chosen points of a domain Ω. For that reason it is necessary to apply an iterative method and assume an initial guess for a searched solution u0 (x). Taking into account convergence of a method, most effective is to assume for an iteration i = 0 real value of unknown function. It is also acceptable (in some cases) to choose constant or zero values. After calculation u1 (x) on the basis of (8), a solution becomes approximated in following iteration steps until fulfilling given stop criterion. An iterative process should be recognized as finished if a difference between two lastly obtained values at all considered points of a domain (or a boundary) is very small (|ui+1 (x) − ui (x)| ≤ ε).
4
Technique of Domain Integrals Computation
A very important stage of solving nonlinear (and not only) problems is calculation of domain integrals. Such inconvenient task (especially considering complex
140
E. Zieniuk and A. Boltuc
domains) in the case of classic BEM [1,2] was led to addition of integrals over smaller domains with simple shapes (a triangle, a rectangle). However, it caused getting rid of a main advantage of BEM viz no domain discretization. Exemplary division of a domain into integral cells is presented in Fig.1a.
Fig. 1. A definition of a domain in a) BEM, b) PIES
Realizing an idea of PIES author was guided by intensions of complete disposing of discretization, what was achieved and successfully tested on many examples [7,8]. For that reason, our concept was to obtain such algorithm of domain integrals calculation which completely eliminate necessity of division of a domain into cells. The fist problem was an effective definition of a domain for integration. Authors made a decision to use surfaces known from computer graphics [3]. In the paper rectangular Bezier surfaces of the first (Fig.2a) and third degree (Fig.2b) were used. Taking into account the first case (used for polygon definition) we need only corner points to domain definition, whilst in the second case (used for modeling of domains with curvilinear boundaries) together 16 control points. It is worth to mention, that in considered problems domains are flat, therefore one should pose only 12 control points (which define a boundary).
Fig. 2. A definition of a Bezier surface of a) the first degree, b) the third degree
Applied surfaces made an effective tool for modeling taking into account easiness of their definition and modification [3]. Moreover, very important is the fact that a geometry described by a surface (that is a mathematical formula) can be directly included into an integrand of a domain integral. The last step is
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES
141
to apply a high-order quadrature for numerical integration (in the paper rectangular Gaussian quadrature was used) [5]. In such a way authors have obtained a procedure [9] for domain integrals calculation, which does not require domain discretization. Finally, there is only one question: if applied quadrature will be proper for integrals calculated over strongly modified domains (with relation to base, rectangular shape of a surface). In response to a mentioned question further part of the paper consists of numerical analysis of two different geometries.
5
An Application to Practical Examples
In order to verify an idea described in the paper the proposed iterative algorithm with PIES was applied to solving a linear equation obtained after substitution n = 1 into (1). The analytical solution of that equation is known u(x) = x21 + x22 ,
(9)
for p(x) = ε{x21 + x22 }n + 4. The Dirichlet boundary condition is imposed on all sides according to (9). Such formulated problem, as was mentioned, can be solved without necessity of application iterative methods. However, in the present paper an iterative approach is used in order to authenticate correctness of proposed method, which is destined to solving nonlinear problems. Mentioned above linear equation was defined on two different domains presented in Fig.3. In both cases Bezier rectangular surfaces were used for domain modeling, but the domain from Fig.3a was defined by means of a surface of the first degree, whilst from Fig.3b by a surface of the third degree. Taking into account such selected examples (one of them is a modification of the second) we can verify generality of proposed modeling approach.
Fig. 3. Considered domains
The rest of parameters assumed for researches are: ε = 1 and three different initial values (the same at all considered domain points) of a solution u0 = 0, 18, 20, −30. Solutions obtained in vertical cross-sections of both domains are presented in Table 1 and 2. Authors have been concentrated not only on analysis of accuracy of obtained results, but also on effectiveness of an iterative process (Fig.4).
142
E. Zieniuk and A. Boltuc Table 1. Solutions from a chosen cross-section of the square domain (Fig.3a) x
y
analytical 0.5 0.05 0.2525 0.5 0.15 0.2725 0.5 0.25 0.3125 0.5 0.35 0.3725 0.5 0.45 0.4525 0.5 0.55 0.5525 0.5 0.65 0.6725 0.5 0.75 0.8125 0.5 0.85 0.9725 0.5 0.95 1.1525 a number of iterations
U u0 = 0 0.252329 0.27306 0.313309 0.37345 0.453532 0.55355 0.673504 0.813403 0.973202 1.15024 7
u0 = 18 0.252329 0.27306 0.313309 0.37345 0.453532 0.55355 0.673504 0.813403 0.973202 1.15024 9
u0 = −30 0.252329 0.27306 0.313309 0.37345 0.453532 0.55355 0.673504 0.813403 0.973202 1.15024 9
Table 2. Solutions from a chosen cross-section of the modified domain (Fig.3b) x
y
analytical 0.6 0.15 0.3825 0.6 0.25 0.4225 0.6 0.35 0.4825 0.6 0.45 0.5625 0.6 0.55 0.6625 0.6 0.65 0.7825 0.6 0.75 0.9225 0.6 0.85 1.0825 0.6 0.95 1.2625 a number of iterations
U u0 = 0 0.384326 0.422461 0.482743 0.562522 0.662908 0.782717 0.922777 1.08264 1.26263 6
u0 = 20 0.384326 0.422461 0.482743 0.562522 0.662908 0.782717 0.922777 1.08264 1.26263 7
u0 = −30 0.384326 0.422461 0.482743 0.562522 0.662908 0.782717 0.922777 1.08264 1.26263 7
As can be seen in Table 1 and 2, the proposed algorithm gives very satisfactory results for both tested linear problems. Starting from various initial values the same accurate results were obtained. An average relative error of solutions in a considered cross-section is equal to: 0.175% for the square domain and 0.073% for the quarter of a ring. An iterative process can be described as effective, because converge to proper solution very fast. As a proof two charts are presented in Fig.4. As can be noticed already in third iteration solutions converge to final (proper) value. Taking into account presented above results and some made conclusions it can be claimed that the proposed algorithm is promising, because gives accurate results in short time. As was mentioned, finally it should be applied to solving nonlinear problems. Therefore, a next step of researches is to apply the proposed technique to solving the equation (1) with different values of n = 1 which have influence on a scale of nonlinearity.
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES
143
Fig. 4. Convergence of an iterative process for a) the square domain, b) the quarter of a circular ring
The first nonlinear problem is connected with parameter n = 2. Authors have taken into account two domains (the same as in the first example) over which solving equation was defined. The following values of parameters were considered: ε = 1 and u0 =< −30..20 >.
Fig. 5. Solutions obtained by PIES for: a) the square domain, b) the quarter of a circular ring
Obtained by means of the proposed algorithm solutions for chosen initial values u0 (Fig.5) are characterized by high accuracy in comparison with analytical results. However, it should be mentioned that: a) a number of iterations is higher than in the case of a linear problem and b) taking into account some initial values (e.g. u0 = −20, −30, 20 for the square domain) an iterative process is divergent. The best results are obtained for u0 = 0, because a final solution is obtained most quickly. Starting from different initial values (e.g. u0 = 15, −10), after first iteration we have obtained results with high relative error (even 4100%), but finally a process converges to the same results as in the case with u0 = 0 . Described above results are presented in Fig.6. Analyzing above results it can be stated that taking into account solved problems the apprach based on an iterative process and PIES gives satisactory results
144
E. Zieniuk and A. Boltuc
Fig. 6. Convergence of the iteration process for: a) the square domain, b) the quarter of a circular ring
Fig. 7. Numerical and exact solutions for different value of for: a) the square domain, b) the quarter of a circular ring
also in the case fo nonlinear problems. There is a question about quality of results in the case of stronger nonlinearity. In order to answer on that question authors have solved a problem modeled by the equation (1) with n = 3 defined over two considered before domains (Fig.3). In the computation, the constant ε is taken to be 1 and 0.1, whilst an initial guess is a value from u0 =< −30..20 >. After solving a described problem with different values of the initial solution and the constant ε authors made some conclusions: a) a number of iterations gets larger with the increase of ε , b) there are some values of u0 for which the iterative process is divergent, e.g. for the square domain a range of variability for which solutions were obtained is < −3..3 > (but the solution for u0 = 0 was obtained in shortest time) and c) obtained final results are less accurate than in previous cases (an average relative error for the square domain is equal to 0.81%, whilst for n = 2 it was 0.26% and for n = 1 only 0.17%). Situation is getting better, when we weaken an influence of nonlinear element εu(x)n by imposing smaller value of ε = 0.1. Taking into account mentioned above value of ε obtained solutions are more accurate, an iterative process is faster, and possible range of variability of u0 is wider. Solutions obtained for both value of ε and both domains in comparison with analytical results are presented in Fig.7.
Iterative Solution of Linear and Nonlinear Boundary Problems Using PIES
6
145
Conclusions
The proposed in the paper algorithm for solving of boundary problems modeled by nonlinear differential equation has given satisfactory results. Both an initial verification of the approach on linear problems and the final application of it to solving problems with nonlinearity have finished successfully (obtaining an accurate numerical solutions) and seem to be effective (an iterative process converges to a final solution in small number of steps). There are also some cases (some values of initial guesses) for which an iterative process is divergent or an accuracy of obtained results is not acceptable. However, presented initial results are promising and encourage to further researches which should eliminate all faults observed on that stage.
References 1. Banerjee, P.K.: The Boundary Element Methods in Engineering. McGraw-Hill, London (1994) 2. Becker, A.A.: The Boundary Element Method in Engineering: A Complete Course. McGraw-Hill, London (1992) 3. Foley, J.D.: Introduction to Computer Graphics. WNT, Warszawa (2001) (in Polish) 4. Gottlieb, D., Orszag, S.A.: Numerical analysis of spectral methods. SIAM, Philadelphia (1977) 5. Stroud, A.H., Secrest, D.: Gaussian Quadrature formulas. Prentice-Hall, London (1966) 6. Zhu, T., Zhang, J., Atluri, N.: A meshless local boundary integral equation (LBIE) method for solving nonlinear problems. Computational Mechanics 22, 174–186 (1998) 7. Zieniuk, E.: Bezier curves in the modification of boundary integral equations (BIE) for potential boundary-values problems. International Journal of Solids and Structures 40(9), 2301–2320 (2003) 8. Zieniuk, E., Boltuc, A.: Non-element method of solving 2D boundary problems defined on polygonal domains modeled by Navier equation. International Journal of Solids and Structures 43, 7939–7958 (2006) 9. Zieniuk, E., Szerszen, K., Boltuc, A.: Global calculation of domain integrals in PIES for two-dimensional problems modeled by Navier-Lame and Poisson equations. Modelowanie inzynierskie 2(33), 181–186 (2007) (in Polish) 10. Zienkiewicz, O.C., Taylor, R.L.: The Finite Element Method. McGraw-Hill, London (1977)
Implementing a Parallel Simulated Annealing Algorithm Zbigniew J. Czech1,2 , Wojciech Mikanik1 , and Rafal Skinderowicz2 1
Silesia University of Technology, Gliwice, Poland {Zbigniew.Czech,Wojciech.Mikanik}@polsl.pl 2 University of Silesia, Sosnowiec, Poland
[email protected]
Abstract. The MPI and OpenMP implementations of the parallel simulated annealing algorithm solving the vehicle routing problem (VRPTW) are presented. The algorithm consists of a number of components which co-operate periodically by exchanging their best solutions found to date. The objective of the work is to explore speedups and scalability of the two implementations. For comparisons the selected VRPTW benchmarking tests are used. Keywords: parallel simulated annealing, MPI library, OpenMP interface, vehicle routing problem with time windows.
1
Introduction
This work presents two implementations of a parallel simulated annealing algorithm. The algorithm, consisting of a number of components which co-operate periodically by exchanging their best solutions found to date, is used to solve the vehicle routing problem with time windows (VRPTW). A solution to the VRPTW is the set of routes beginning and ending at the depot which serves a set of customers. For the purpose of goods delivery there is a fleet of vehicles. The set of routes should visit each customer exactly once, ensure that the service at any customer begins within the time window and preserve the vehicle capacity constraints. The VRPTW belong to a large family of the vehicle routing problems (VRP) which have numerous practical applications. The MPI and OpenMP implementations of the parallel simulated annealing algorithm solving the VRPTW are presented. The objective of the work is to explore speedups and scalability of these two implementations. The results of the work extend our previous efforts concerning the application of parallel simulated annealing to solve the VRPTW. However in the earlier projects [1,2,3,4,5,6] the parallel execution of processes was simulated in a single processor, and the main goal of investigations was to achieve a high accuracy of solutions through cooperation of processes. In section 2 the parallel simulated annealing algorithm is presented. Sections 3 and 4 describe the MPI and OpenMP implementations of the algorithm, R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 146–155, 2010. c Springer-Verlag Berlin Heidelberg 2010
Implementing a Parallel Simulated Annealing Algorithm
147
respectively. Section 5 contains the discussion of the experimental results. Section 6 concludes the work.
2
Parallel Simulated Annealing Algorithm
The parallel simulated annealing algorithm comprises p components which are executed as processes P0 , P1 , . . . , Pp−1 (Fig. 1). A process generates its own annealing chain divided into two phases (lines 6–18). A phase consists of a number of cooling stages, and a cooling stage consists of a number of annealing steps. The goal of phase 1 is to minimize the number of routes of the VRPTW solution, whereas phase 2 minimizes the total length of the routes. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
parfor Pj , j = 0, 1, . . . , p − 1 do Set co-operation mode to regular or rare depending on a test set; L := (5E)/p; {establish the length of a cooling stage} Create the initial solution using some heuristics; current solution j := initial solution; best solution j := initial solution; for f := 1 to 2 do {execute phase 1 and 2} {beginning of phase f } T := T0,f ; {initial temperature of annealing} repeat {a cooling stage} for i := 1 to L do annealing step f (current solution j , best solution j ); end for; if (f = 1) or (co-operation mode is regular ) then {ω = L} co operation; else {rare co-operation: ω = E} Call co operation procedure every E annealing step counting from the beginning of the phase; end if; T := βf T ; {temperature reduction} until af cooling stages are executed; {end of phase f } end for; end parfor; Produce best solution p−1 as the solution to the VRPTW;
Fig. 1. Parallel simulated annealing algorithm; constant E = 105 annealing steps
At every annealing step a neighbor solution is computed by migrating customers among the routes (line 2 in Fig. 2). In simulated annealing the neighbor solutions of lower costs are always accepted, where by accepting we mean that the neighbor solution becomes a current one in the annealing chain. The higher cost solutions are accepted with probability e−δ/Tk , where Tk , k = 0, 1, . . . , af , is a parameter called a temperature of annealing, and af is a number of cooling stages in phase f , f = 1 or 2. The temperature drops from an initial value
148 1 2 3 4 5 6 7 8 9 10 11
Z.J. Czech, W. Mikanik, and R. Skinderowicz procedure annealing step f (current solution, best solution); Create new solution as a neighbor to current solution (the way this step is executed depends on f ); δ := cost f (new solution)−cost f (current solution); Generate random x uniformly in the range (0, 1); if (δ < 0) or (x < e−δ/T ) then current solution := new solution; if cost f (new solution) < cost f (best solution) then best solution := new solution; end if; end if; end annealing stepf ; Fig. 2. Annealing step procedure
1 2 3 4 5 6 7 8 9 10 11
procedure co operation; if j = 0 then Send best solution 0 to process P1 ; else {j > 0} receive best solution j−1 from process Pj−1 ; if cost f (best solution j−1 ) < cost f (best solution j ) then best solution j := best solution j−1 ; current solution j := best solution j−1 ; end if; if j < p − 1 then Send best solution j to process Pj+1 ; end if; end if; end co operation; Fig. 3. Procedure of co-operation of processes
T0,f according to equation Tk+1 = βf Tk , where βf (βf < 1) and δ denote some constants and an increase of the solution cost, respectively (line 17 in Fig. 1). A sequence of steps for which the temperature of annealing stays constant constitutes a cooling stage. The cost of solution s to the VRPTW is defined in phase 1 of the parallel algorithm as: cost 1 (s) = c1 N + c2 D + c3 (r1 − r¯), and in phase 2 as: cost 2 (s) = c1 N + c2 D, where N is the number of routes in solution s (equal to the number of vehicles carrying out the service), D is the total travel distance of the routes, r1 is the number of customers in a route which is tried to be get rid of the current solution, r¯ is an average number of customers in all routes, and c1 , c2 , c3 are constants. The basic criterion of optimization is the number of routes, therefore c1 c2 is fixed. The processes of the parallel algorithm co-operate with each other every ω annealing step passing their best solutions found to date (lines 12–16 in Fig. 1 and Fig. 3). The chain of annealing steps of process P0 is entirely independent (Fig. 4). The chain of process P1 is updated at steps uω, u = 1, 2, . . . , um , to the better solution between the best solutions found by processes P0 and P1 to date. Similarly, process P2 chooses as the next point in its chain the better solution between its own best and the one obtained from process P1 . Thus the
Implementing a Parallel Simulated Annealing Algorithm
149
best solution found by process Pl is piped down for further enhancement to processes Pl+1 . . . Pp−1 . Clearly, after step um ω process Pp−1 holds the best solution, Xb , found by all the processes. ⎧ (0) X0 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ (0) ⎪ ⎪ X1 ⎪ ⎪ ⎪ ⎪ ⎨ • X0 → ⎪ ⎪ • ⎪ ⎪ ⎪ (0) ⎪ ⎪ Xp−2 ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ (0) Xp−1
(ω)
→ X0 ↓ (ω) → X1 ↓ • • (ω) → Xp−2 ↓ (ω) → Xp−1
(2ω)
→ X0 ↓ (2ω) → X1 ↓ • • (2ω) → Xp−2 ↓ (2ω) → Xp−1
(um ω)
→ • • → X0
↓
(um ω)
→ • • → X1
↓ •• • •• • (um ω) → • • → Xp−2 ↓ (um ω) → • • → Xp−1 → Xb
Fig. 4. Scheme of co-operation of processes (X0 – initial solution; Xb – best solution among the processes)
As mentioned before, the temperature of annealing decreases according to the equation Tk+1 = βf Tk for k = 0, 1, 2, . . . , af , where af is the number of cooling stages. In this work we investigate two cases in establishing the points of process co-operation with respect to temperature drops. In the first case, of regular co-operation, processes interact at the end of each cooling stage (ω = L) (lines 15–16 in Fig. 1). The number of annealing steps executed within a cooling stage is set to L = (5E)/p, where E = 105 is a constant and p, p = 5, 10, 15 and 20, is the number of processes (line 3 in Fig. 1). Such an arrangement keeps the parallel cost of the algorithms constant when different numbers of processes are used, provided the co-operation costs are neglected. Therefore in this case, as the number of processes becomes larger the length of cooling stages goes down, what means that the frequency of co-operation increases. In the second case, of rare co-operation, the frequency is constant and the processes exchange their solutions every ω = E annealing step (lines 17–18 in Fig. 1). For the number of processes p = 10, 15 and 20, the co-operation takes place after 2, 3 and 4 temperature drops, respectively.
3
The MPI Implementation
All processes Pj , j = 0, 1, . . . , p − 1, in the MPI implementation of the parallel simulated annealing algorithm, execute the code shown in Fig. 5. Like in the general description of the algorithm (Fig. 1) the computation of processes is divided into phases 1 and 2. Depending on the co-operation mode, regular or rare, the processes interact every L or E annealing step, respectively. The MPI implementation of the co-operation between processes is presented in Fig. 6.
150
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Z.J. Czech, W. Mikanik, and R. Skinderowicz Processes Pj , j = 0, 1, . . . , p − 1; Set co-operation mode to regular or rare depending on a test set; L := (5E)/p; {establish the length of a cooling stage} Create the initial solution using some heuristics; current solution := initial solution; best solution := initial solution; for f := 1 to 2 do {execute phase 1 and 2} {beginning of phase f } T := T0,f ; {initial temperature of annealing} repeat {a cooling stage} for i := 1 to L do annealing step f (current solution, best solution); end for; if (f = 1) or (co-operation mode is regular ) then {ω = L} MPI co operation; else {rare co-operation: ω = E} Call MPI co operation procedure every E annealing step counting from the beginning of the phase; end if; T := βf T ; {temperature reduction} until af cooling stages are executed; {end of phase f } end for; Produce best solution of process Pp−1 as the solution to the VRPTW; Fig. 5. MPI implementation
1 2 3 4 5 6 7 8 9 10 11 12
procedure MPI co operation; if j = 0 then Send best solution to process P1 using MPI Isend function; else {j > 0} Receive best solution r from process Pj−1 using MPI Recv function; if cost f (best solution r ) < cost f (best solution) then best solution := best solution r ; current solution := best solution r ; end if; if j < p − 1 then Send best solution to process Pj+1 using MPI Isend function; end if; end if; end MPI co operation; Fig. 6. Procedure of co-operation of MPI processes
4
The OpenMP Implementation
In the OpenMP implementation of the parallel simulated annealing algorithm, the current and best solutions found to date are kept in the shared memory in the arrays defined as follows current solution, best solution: array[0 .. p − 1] of solution;
Implementing a Parallel Simulated Annealing Algorithm
151
where p is the number of threads, and solution is a data structure representing a VRPTW solution. The arrays are initialized in lines 5–9 in Fig. 7. The most time consuming part of the parallel simulated annealing algorithm in Fig. 1 is the loop in lines 9–11 where each process computes its own annealing chain. In the OpenMP implementation these chains are built by the threads within the parallel regions (lines 13–19 in Fig. 7). The co-operation of the threads is shown in Fig. 8. Summing up, only lines 5–9 and 13–19 of the OpenMP implementation shown in Fig. 7 are executed in parallel by the team of threads. The remaining parts of the implementation is executed sequentially by a single thread.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
omp set num threads(p); {set number of threads to p} Set co-operation mode to regular or rare depending on a test set; L := (5E)/p; {establish the length of a cooling stage} Create the initial solution using some heuristics; #pragma omp parallel for for j := 0 to p − 1 do current solution[j] := initial solution; best solution[j] := initial solution; end for; for f := 1 to 2 do {execute phase 1 and 2} {beginning of phase f } T := T0,f ; {initial temperature of annealing} repeat {a cooling stage} #pragma omp parallel { j := omp get thread num(); for i := 1 to L do annealing step f (current solution[j], best solution[j]); end for; } if (f = 1) or (co-operation mode is regular ) then {ω = L} OpenMP co operation; else {rare co-operation: ω = E} Call OpenMP co operation procedure every E annealing step counting from the beginning of the phase; end if; T := βf T ; {temperature reduction} until af cooling stages are executed; {end of phase f } end for; Produce best solution[p − 1] as the solution to the VRPTW; Fig. 7. OpenMP implementation
152 1 2 3 4 5 6 7 8
Z.J. Czech, W. Mikanik, and R. Skinderowicz procedure OpenMP co operation; for j : = 1 to p − 1 do if cost f (best solution[j − 1]) < cost f (best solution[j]) then best solution[j] := best solution[j − 1]; current solution[j] := best solution[j − 1]; end if; end for; end OpenMP co operation; Fig. 8. Procedure of co-operation of OpenMP threads
Table 1. Results for tests R107 and R207; regular co-operation (p – number of processes or threads, τ¯ – average execution time, s – standard deviation of execution times, S – speedup, exp. – number of experiments)
Test p 1 5 R107 10 15 20 1 5 R207 10 15 20
5
MPI implement. τ¯ s S exp. 730.8 9.9 1.00 100 149.4 1.7 4.89 100 76.3 0.9 9.58 100 50.8 0.6 14.38 100 38.2 0.5 19.16 100 1113.7 36.9 1.00 100 231.5 8.6 4.81 100 115.2 4.6 9.67 100 77.5 2.4 14.38 100 58.6 2.3 19.01 100
OpenMP implement. τ¯ s S exp. 1047.0 10.1 1.00 100 263.5 3.9 3.97 100 135.8 2.5 7.71 100 91.0 0.9 11.51 100 69.0 0.7 15.18 100 1794.1 64.6 1.00 100 434.0 21.6 4.13 100 223.5 16.2 8.03 100 144.5 3.6 12.41 100 110.0 2.7 16.32 100
Experimental Results
The MPI implementation was run on a cluster of 288 nodes which consisted of 1 or 2 Intel Xeon 2.33 GHz processors, each with 4 MB level 3 cache. Nodes were connected with the Infiniband DDR fat-tree full-cbb network (throughput 20 Gbps, delay 5 μs). The computer was executing a Scientific Linux. The source code was compiled using Intel 10.1 compiler and MVAPICH1 v. 0.9.9 MPI library. The OpenMP computations were performed on a ccNUMA parallel computer SGI Altix 3700 equipped with 128 Intel Itanium 2 processors clocked at 1.5 GHz, each with 6 MB level 3 cache. The computer was running Suse Linux Enterprise Server 10 operating system. The source code was compiled using Intel C compiler 11.0 with the following compilation flags: -openmp -openmp-report2 -wd1572. The MPI and OpenMP implementations were compared on the selected VRPTW benchmarking tests proposed by Solomon [7]. The investigations reported in [6] indicated that Solomon’s tests can be divided into 3 groups: I – tests which can be solved quickly (e.g. using p = 20 processes) to good accuracy
Implementing a Parallel Simulated Annealing Algorithm
153
Table 2. Results for tests R104 and RC106; rare co-operation (meanings of columns as before) Test p 1 5 R104 10 15 20 1 5 RC106 10 15 20
MPI implement. τ¯ s S exp. 755.6 7.8 1.00 1000 154.7 2.4 4.88 1000 79.9 1.1 9.46 1000 52.1 1.0 14.51 1000 39.4 0.5 19.18 1000 705.5 6.9 1.00 200 143.6 1.5 4.91 200 73.1 0.7 9.65 200 49.2 0.6 14.35 200 35.5 0.6 19.33 200
20
◦ ◦ - MPI - OpenMP
15
◦
OpenMP implement. τ¯ s S exp. 1028.2 10.3 1.00 234 262.6 8.0 3.92 200 134.8 5.3 7.63 200 90.5 3.2 11.36 200 68.7 1.7 14.97 200 1072.0 20.5 1.00 100 278.1 4.9 3.85 100 139.9 29.8 7.66 100 95.4 1.6 11.24 100 73.8 1.0 14.54 100
20
◦ ◦ - MPI - OpenMP
15
◦
S 10 ◦
5
◦
5
1 ◦ 1
◦
S 10 test R107
◦
test R207
1 ◦ 5
10 p
15
20
1
5
10 p
15
20
Fig. 9. Speedup S vs. number of processes (threads) p for tests R107 and R207 (broken line shows the ideal speedup)
20 ◦ - MPI - OpenMP
15 S 10
◦ ◦
◦
5
◦
20
15
◦ - MPI - OpenMP
◦
5
test R104
◦ ◦
S 10
1 ◦ 1
◦
test RC106
1 ◦ 5
10 p
15
20
1
5
10 p
15
20
Fig. 10. Speedup S vs. number of processes (threads) p for tests R104 and RC106
with rare co-operation (ω = E); II – tests which can be solved quickly (p = 20) but the frequency of co-operation should be relatively high (in this work we call it regular) (e.g. ω = E/4 for p = 20) to achieve good accuracy of solutions; and III – tests whose solving cannot be accelerated as much as for the tests from
154
Z.J. Czech, W. Mikanik, and R. Skinderowicz 20
◦ ◦ - MPI - OpenMP
15 S 10
◦ ◦
◦
5
◦ ◦ - MPI - OpenMP
15
◦
5
test RC104
◦
◦
S 10
1 ◦ 1
20
test RC202
1 ◦ 5
10 p
15
20
1
5
10 p
15
20
Fig. 11. Speedup S vs. number of processes (threads) p for tests RC104 and RC202
groups I and II, i.e. the number of processes should be less than 20 to obtain good accuracy of solutions. For comparisons of the implementations, two tests from each of these groups were chosen: tests R104, RC106 from group I, tests R107, R207 from group II, and tests RC104, RC202 from group III. The results of experiments are presented1 in Tab. 1-2 and illustrated in Fig. 9-11. It can be seen that the OpenMP implementation achieves slightly worse speedups than the MPI implementation. The experiments are still in progress and they concentrate now on the accuracy of solutions to the VRPTW. According to results presented in [6], the RC104 and RC202 tests are difficult to solve, i.e. for a larger number of processes the loss of accuracy of solutions to these tests is observed. The goal of the ongoing experiments is to clarify this issue using the MPI and OpenMP implementations.
6
Conclusions
The MPI and OpenMP implementations of the parallel simulated annealing algorithm to solve the VRPTW were presented. The experimental results obtained for the selected tests by Solomon indicated that the the MPI implementation scales better than the OpenMP implementation. The issue of the accuracy of the VRPTW solutions is to be cleared up.
Acknowledgments We thank the following computing centers where the computations of our project were carried out: Academic Computer Centre in Gda´ nsk TASK, Academic Computer Centre CYFRONET AGH, Krak´ ow (computing grants 027/ 2004 and 069/2004), Pozna´ n Supercomputing and Networking Center, Interdisciplinary Centre for Mathematical and Computational Modelling, Warsaw University (computing grant G27-9), Wroclaw Centre for Networking and Supercomputing (computing grant 04/97). The research of this project was supported by the Minister of Science and Higher Education grant No 3177/B/T02/2008/35. 1
Because lack of space we omit the results for tests RC104 and RC202.
Implementing a Parallel Simulated Annealing Algorithm
155
References 1. Czarnas, P., Czech, Z.J., Gocyla, P.: Parallel simulated annealing for bicriterion optimization problems. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Wa´sniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 233–240. Springer, Heidelberg (2004) 2. Czech, Z.J., Czarnas, P.: A parallel simulated annealing for the vehicle routing problem with time windows. In: Proc. 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing, Canary Islands, Spain, January 2002, pp. 376–383 (2002) 3. Czech, Z.J., Wieczorek, B.: Solving bicriterion optimization problems by parallel simulated annealing. In: Proc. of the 14th Euromicro Conference on Parallel, Distributed and Network-based Processing, Montb´eliard-Sochaux, France, February 1517, pp. 7–14 (2006) (IEEE Conference Publishing Services) 4. Czech, Z.J., Wieczorek, B.: Frequency of co-operation of parallel simulated annealing processes. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Wa´sniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 43–50. Springer, Heidelberg (2006) 5. Czech, Z.J.: Speeding up sequential annealing by parallelization. In: Proc. of the International Conference on Parallel Computing in Electrical Engineering, PARELEC 2006, Bialystok, Poland, September 13-17, pp. 349–354 (2006) (IEEE Conference Publishing Services) 6. Czech, Z.J.: Co-operation of processes in parallel simulated annealing. In: Proc. of the 18th IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS 2007), Dallas, Texas, USA (November 13-15, 2006), pp. 401–406 (2007) 7. Solomon, M.M.: Algorithms for the vehicle routing and scheduling problems with time window constraints. Operations Research 35, 254–265 (1987), http://w.cba.neu.edu/~ msolomon/problems.htm
Parallel Computing Scheme for Graph Grammar-Based Syntactic Pattern Recognition Mariusz Flasi´ nski, Janusz Jurek, and Szymon My´sli´ nski IT Systems Department, Jagiellonian University Straszewskiego 27, 31-110 Cracow, Poland http://www.wzks.uj.edu.pl/ksi
Abstract. Real-time syntactic pattern recogniton imposes strict computing time constraints on new techniques developed. Recently, a method for an analysis of hand postures of the Polish Sign Language based on the ETPL(k) graph grammars (Flasi´ nski: Patt. Recogn. 26 (1993), 1-16; Theor. Comp. Sci. 201 (1998), 189-231) has been constructed. In order to make a system implemented more feasible for the users, a research into parallelization of a pattern recognition process has been led. Possible techniques of tasks distribution have been tested. It has allowed us to define an optimum strategy of parallelization. The results are presented in the paper. Keywords: parallel computing, syntactic pattern recognition, graph grammar.
1
Introduction
In the field of pattern recognition and image processing, both parallel architectures [11,18,22,26] and parallel processing techniques [2,14,15,19,23,25] have been widely used since the early eighties. An optical pattern (image) recognition is one of the main areas of using these techniques, since it usually imposes real-time constraints on computing process. Pattern recognition methods can be divided into two main categories: the statistical one and structural/syntactic one. Whereas a lot of research has been made into parallelization techniques of the first approach (see e.g. citations listed above), only a few results concerning the second area have been published till now. The main idea of syntactic pattern recognition consists in representing a pattern as a structure of the form of string, tree or graph. Then, a set of structures is treated as a formal language (string, tree or graph language) that can be analyzed with a (string, tree or graph) automaton (syntax analyzer, parser). A syntactic pattern recognition computing process consists of three main phases: - an image processing (filtering, enhancement, etc.), - a generation of a structural representation, and - a structural representation parsing (syntactic analysis). In the area of syntactic pattern recognition, a research into parallel computing techniques has concerned mainly the first two phases (see citations listed above). R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 156–165, 2010. c Springer-Verlag Berlin Heidelberg 2010
Parallel Computing Scheme for Syntactic Pattern Recognition
157
For the third phase, that usually demands a very large computational effort [12], a few results are known. A parallel version of the Earley’s parsing algorithm for syntactic patern recognition was defined by Chiang and Fu in 1984 [1]. Then, results of a research into a parallelization of parsing for attributed grammars were published in the nineties: [13,21] by Klaiber, Gokhale and independently by Tokuda and Watanabe (tasks: of parsing, and an attribute evaluation in sequence), and by Sideri, Efremidis and Papakonstantinou [20] (parallel both: parsing and an attribute evaluation). A parallelization technique of a structural pattern recognition was proposed by Wu and Rosenberg in 1988 [24]. A method of pipelined processing of attributed grammar parsing for syntactic pattern recognition was presented by Pavlados, Panagopoulos and Papakonstantinou in 2004 [17]. A research into parsing for graph grammars has resulted in defining a very efficient parser, O(n2 ), for ETPL(k) graph grammars [3,5,7,9]. It has been used as a syntactic pattern recognition model for a variety of applications such, as: a pattern recognition in the industrial robot control system [5], a software allocation system for a parallel/distributed environment [4], a solid modelling and analysis in the CAD/CAM integration system [6], and a real-time expert control system [8]. Recently, it has been also applied for a recognition of hand postures of the Polish Sign Language [10,16]. Hard real-time requirements of this last application have motivated us to search for such a parallel scheme of a recognition process that could allow us to make the system more feasible for the users. The results of this research are presented in the paper. A syntactic pattern recognition model of a hand posture analysis is shortly presented in the next chapter. An architecture of the hand posture recognition system is described in the third chapter. Chapter 4 includes considerations concerning two possible models of tasks distribution for parallel processing. It also contains results of experiments made to compare these models. Concluding remarks are presented in the last chapter.
2
Syntactic Pattern Recognition Model of Hand Posture Analysis
A syntactic pattern recognition of hand postures of the Polish Sign Language consists of three main phases shown in Fig. 1.
IMAGE PROCESSING AND RECOGNITION SYSTEM
HAND IMAGE
INITIAL PROCESSING
GENERATION OF GRAPH-BASED HAND IMAGE REPRESENTATION
SYNTACTIC ANALYSIS
HAND POSTURE
Fig. 1. Main phases of a syntactic pattern recognition of hand postures
158
M. Flasi´ nski, J. Jurek, and S. My´sli´ nski
The first phase of (initial) image processing includes: a skin region detection, a noise removal, a contour extraction, and a polygonal approximation of a contour (see Figures: 2(a) and 2(b)). A set of characteristic points of an image (that will be used as a set of graph nodes) consists of: the region centroid and polygon vertices as it is depicted in Fig. 2(b) [16]. The second phase is critical, because - after spanning a graph structure on a set of characteristic points (see Fig. 2(c)) - we have to define a graph representation (see Fig. 2(d)) fulfilling a formal definition of IE-graphs [5,7]. Only such graphs belong to a parsable class of ETPL(k) graph languages [3,7], and they can be parsed with an efficient, O(n2 ), parser [5].
Y
0
(a)
X
(b)
Y
Y
4
h
-h 6
b
3
i
1
(c)
X
0
-i
i g 5 -i 2
0
h
e
(d)
a
b
d
7
s X
Fig. 2. Results of phases: of image processing and a graph representation generation
A formal notion of IE-graph is defined in the following way [5,7]. Definition. An indexed edge-unambiguous graph, IE graph, over Σ and Γ is a quintuple g = (V, E, Σ, Γ, φ), where V is a finite, non-empty set of nodes that indices have been ascribed to in an unambiguous way,
Parallel Computing Scheme for Syntactic Pattern Recognition
159
Σ is a finite, non-empty set of node labels, Γ is a finite, non-empty set of edge labels, E is a set of edges of the form (v, λ, w), where v, w ∈ V, λ ∈ Γ , such that index of v is less than index of w, φ : V −→ Σ is a node-labeling function, and g contains a BFS spanning tree, which nodes are indexed in the same way as nodes of g and its edges are directed in the same way as edges of g. The final, third stage of syntactic analysis consists in parsing of an IE-graph representation of a hand posture image. This way, the parser identifies, with ca 95% accuracy, a hand posture of the Polish Sign Language.
3
System Architecture
A computer-aided recognition of any sign language imposes hard real-time constraints on a system. In fact, a fast identification of consequtive hand postures is a necessary condition for the system to be useful in practice. Therefore, to increase an efficiency of a hand posture identification process, the HandPostPolSign 2.0 system, constructed with the use of a model presented in a previous chapter, has been designed and implemented on the basis of a parallel computing paradigm. In a syntactic pattern recognition-based model, a result of each of the previous phases is an input for the next one. Treating each phase as an independent task enables their parallel execution on a multiprocessor (multicore) system in the form of different threads. A software architecture of the HandPostPolSign 2.0 system is shown in Fig. 3. Each of the processing modules (corresponding to the phase of a syntactic pattern recognition model) is designed as an independent sub-system. The system has been implemented in C++. For image processing, Intel Integrated Performance Primitives library has been used. The system consists of
1..* IMAGE SOURCE
1
1 Dispatcher
Colour image 1
1..*
1
1..*
Hand posture
1 1..*
1..*
«subsystem»
PreProcessor
Image characteristic points set
FURTHER PROCESSING
«subsystem»
GraphGenerator
Graph-based image representation
«subsystem» Parser
Fig. 3. A software architecture of the HandPostPolSign 2.0 system
160
M. Flasi´ nski, J. Jurek, and S. My´sli´ nski
seven main classes. Three of them, namely CPreProcessor, CGraphGenerator and CParser correspond to main syntactic pattern recognition phases. The remaining classes: CDispatcher, CCSystem, CImageSrc and CProcessingMutex are responsible for: a task allocation and a cooperation with multiple image sources, preserving a correct sequence of recognition results, a cooperation with input image devices, ensuring data consistency during multi-threaded processing (respectively). In HandPostPolSign 2.0 a number of threads corresponding to syntactic pattern recognition tasks can be changed dynamically (many hand posture images can be recognized simultaneously). However, a proper model of a task distribution should have been defined in order to optimize an utilization of a multiprocessor system. Results of a research into such a model are presented in the following chapter.
4
Task Distribution Model
In this chapter two possible models of a distribution of syntactic pattern recognition tasks are introduced. The results of time efficiency experiments for these models are presented and corresponding conclusions are discussed. 4.1
Vertical Model of Tasks Distribution (VMTD)
In a vertical model of recognition tasks distribution (VMTD model), each of the sub-systems (i.e. initial processing, a graph generation, a syntactic analysis) is initialized in a separate thread that is executed on a separate CPU (or core) of a multiprocessor (multicore) system. When a recognition process starts, the first image is inputed to the initial processing sub-system. Then, a set of characteristic points identified in a hand image is handed over to the graph generation subsystem. At the same time, the second image is inputed to the initial processing sub-system that has already finished processing of the first image. In the meantime, a graph representation of the first image is passed to the syntactic analysis sub-system. Then, a set of characteristic points identified for the second image Time
Thread
1
1
2
3
2
3
IP
4
IP GG
5
IP GG
SA
6
IP GG
SA
...
...
...
SA
...
...
Fig. 4. Vertical model of recognition tasks distribution. IP – Initial Processing, GG – Graph Generation, SA – Syntactic Analysis (Parsing). Various colours represent consecutive images.
Parallel Computing Scheme for Syntactic Pattern Recognition 6
161 1 thread
5
3 threads
]s 4 [ e3 m i T2
1 0 1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70
Vertices
Fig. 5. A VMTD model processing times of sequences of 10 images in relation to a number of vertices of their graph-based image representation
is handed over to the graph generation sub-system, while the third image is fed into the initial processing sub-system, etc. Successive images are put into the system as it is shown in Fig. 4. Such a task distribution model applied in a pattern recognition field seems to be natural, because recognition results are obtained in the same order as the order of images inputed to a system. However, experiments concerning a computing time have revealed its low efficiency. In Fig. 5, processing times of a hand image recognition in relation to a number of vertices of their graph representations are shown. For each test, sequences of 10 images have been used to acquire measurement results. One can easily see that increasing a number of threads nearly does not influence an overall performance of the system. A VMTD model can be efficient for pattern recognition applications, however, only if computing times of processing phases are comparable. As we have discussed it in chapter 2, algorithms used in the phases of: a graph representation generation and a syntactic analysis are based on the efficient model of ETPL(k)
Initial Processing
0,45 0,4 0,35 0,3 ]s [ 0,25 e m i 0,2 T 0,15 0,1 0,05 0
Graph Generation Syntactic Analysis
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70
Vertices
Fig. 6. Processing times of a hand image for three main phases
162
M. Flasi´ nski, J. Jurek, and S. My´sli´ nski
graph grammars [5,7]. In Fig. 6, processing times of a single hand image for three main phases of our method are shown. It can be easily noticed that the initial processing is the most time-consuming phase of the whole process. As a result, the graph generation sub-system and the parsing sub-system are idle for most of the time. The disproportion of processing time is so big that even performing the initial processing phase in one thread sequentially, and both a graph generation and parsing in the other thread, does not solve the problem. 4.2
Horizontal Model of Tasks Distribution (HMTD)
In a horizontal model of recognition tasks distribution (HMTD model), all three processing tasks related to an inputed image are initialized in one thread as it is shown in Fig. 7. Each of such threads is executed on a separate CPU (or core) of a multiprocessor (multicore) system. Time
Thread
1
2
3
4
5
IP
GG
1
IP
GG
SA
2
IP
GG
SA
...
3
IP
GG
SA
...
6
SA
...
...
...
Fig. 7. Horizontal model of recognition tasks distribution. IP – Initial Processing, GG – Graph Generation, SA – Syntactic Analysis (Parsing). Various colours represent consecutive images.
Processing times of a hand image recognition process (in relation to a number of vertices of their graph representations) performed in 1, 2 and 4 processing units simultaneously for this model are presented in Fig. 8. As one can easily notice, a remarkable performance boost of the system has been achieved for the HMTD model. However, in this case some problems had to be solved to assure a correctness of the recognition process. Since IE-graph representations of hand posture images differ in a number of vertices, their recognition times vary, as well. Of course, in a case of our application (an analysis of the Polish Sign Language), the same sequence of recognition results at the output of the system, as the sequence of images at the input has to be preserved. Therefore, a dispatching method for preserving an image sequence had to be constructed for a synchronization of processing units. It has been implemented as a dispatcher sub-system, mentioned in chapter 3. A sequence of images is directed to the dispatcher sub-system from an image source (i.e. a video camera). The dispatcher sends subsequent images to available processing units running in separate threads. The recognition results from the
Parallel Computing Scheme for Syntactic Pattern Recognition
163
6
1 thread
5
2 threads
]s 4 [ e3 m iT 2
4 threads
1 0 1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70
Vertices
Fig. 8. A HMTD model processing times of sequences of 10 images in relation to a number of vertices of their graph-based image representation
processing units are stored in a buffer. The mechanisms preserving the right order of the recognition results are implemented for the buffer. It allows the system to preserve a correct sequence of recognition results generated.
5
Concluding Remarks
The research into a system recognizing hand postures of the Polish Sign Language has revealed a need of making the method developed more efficient computionally, in order to be more feasible for the users. In the field of pattern recognition, such an increase of time efficiency is usually achieved by applying parallel computing techniques. Therefore, the hand posture recognition system HandPostPolSign 2.0 has been designed and implemented on the basis of a parallel computing paradigm. Two models of a distribution of recognition tasks (phases) of the system have been considered to increase the system throughput. In a vertical model VMTD, subsequent recognition tasks are initialized in separate threads. Although such a model seems to be more natural for image analysis applications, since recognition results are obtained in the same order as the order of images inputed to the system, it has turned out to be inefficient for our method. It results from the fact that (contrary to standard syntactic pattern recognition methods, in which a syntactic analysis phase may yield a combinatorial explosion [12]) we have used a very efficient parsing scheme, O(n2 ), based on ETPL(k) graph grammars [3,5,7,9]. The experiments made for each phase of a syntactic pattern recognition scheme separately, allow us to draw a general methodological conclusion in a field of the syntactic pattern recognition theory. The VMTD model can be efficient, only if computing times of phases of: image preprocessing, a structural representation generation, and parsing, are comparable. In a horizontal model HMTD, all three recognition tasks (phases) related to an inputed image are performed in one thread. In this case, a remarkable performance boost in a recognition process has been obtained. An increase of a
164
M. Flasi´ nski, J. Jurek, and S. My´sli´ nski
performance is ca. 41% and ca. 59% for 2 and 4 processing units, respectively (Intel Core 2 Quad, 2.33 GHz, 4 GB RAM), comparing to the system performance with no parallel processing. However, for the HMTD model, the mechanisms allowing the system to preserve the same sequence of recognition results at the output, as the sequence of input images, have to be implemented.
References 1. Chiang, Y., Fu, K.S.: Parallel parsing algorithms and VLSI implementation for syntactic pattern recognition. IEEE Trans. Pattern Anal. Machine Intell. PAMI-6, 302–313 (1984) 2. Davis, L.S., Rosenfeld, A., Inoue, K., Nivat, M., Wang, P.S.P. (eds.): Parallel Image Analysis: Theory and Applications. World Scientific, Singapore (1995) 3. Flasi´ nski, M.: Parsing of edNLC-graph grammars for scene analysis. Pattern Recognition 21, 623–629 (1988) 4. Flasi´ nski, M., Kotulski, L.: On the use of graph grammars for the control of a distributed software allocation. The Computer Journal 35, A165–A175 (1992) 5. Flasi´ nski, M.: On the parsing of deterministic graph languages for syntactic pattern recognition. Pattern Recognition 26, 1–16 (1993) 6. Flasi´ nski, M.: Use of graph grammars for the description of mechanical parts. Computer Aided Design 27, 403–433 (1995) 7. Flasi´ nski, M.: Power properties of NLC graph grammars with a polynomial membership problem. Theoretical Computer Science 201, 189–231 (1998) 8. Flasi´ nski, M.: Automata-based multi-agent model as a tool for constructing real-time intelligent control systems. In: Dunin-Keplicz, B., Nawarecki, E. (eds.) CEEMAS 2001. LNCS (LNAI), vol. 2296, pp. 103–110. Springer, Heidelberg (2002) 9. Flasi´ nski, M.: Inference of parsable graph grammars for syntactic pattern recognition. Fundamenta Informaticae 80, 379–413 (2007) 10. Flasi´ nski, M., Jurek, J., My´sli´ nski, S.: Multi-agent system for recognition of hand postures. In: Allen, G., et al. (eds.) ICCS 2009, Part II. LNCS, vol. 5545, pp. 815–824. Springer, Heidelberg (2009) 11. Guerra, C.: 2D object recognition on a reconfigurable mesh. Pattern Recognition 31, 83–88 (1998) 12. Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: a review. IEEE Trans. Pattern Anal. Machine Intell. PAMI-22, 4–37 (2000) 13. Klaiber, A., Gokhale, M.: Parallel evaluation of attributed grammars. IEEE Trans. Parall. Distr. Syst. 3, 206–220 (1992) 14. Lee, S.S., Tanaka, H.T.: Parallel image segmentation with adaptive mesh. In: Proc. 15th Int. Conf. Patt. Recogn., Barcelona, Spain, September 2000, vol. 1, pp. 635– 639 (2000) 15. Miguet, S., Montanvert, A., Wang, P.S.P. (eds.): Parallel Image Analysis. World Scientific, Singapore (1998) 16. My´sli´ nski, S.: On the generation of graph representation of hand postures for syntactic pattern recognition. Journal of Applied Computer Science 17 (2009) 17. Pavlatos, C., Panagopoulos, I., Papakonstantinou, G.: A programmable pipelined coprocessor for parsing applications. In: Proc. Workshop on Application Specific Processors, Stockholm, September 2004, pp. 84–91 (2004) 18. Rabah, H., Mathias, H., Weber, S., Mozef, E., Tanougast, C.: Linear array processors with multiple access modes memory for real-time image processing. Real Time Imaging 9, 205–213 (2003)
Parallel Computing Scheme for Syntactic Pattern Recognition
165
19. Ranganathan, N. (ed.): VLSI and Parallel Computing for Pattern Recognition and Artificial Intelligence. World Scientific, Singapore (1995) 20. Sideri, M., Efremidis, S., Papakonstantinou, G.: Semantically driven parsing of CFG. The Computer Journal 32, 91–98 (1994) 21. Tokuda, T., Watanabe, Y.: An attribute evaluation of context-free languages. Information Processing Letter 57, 91–98 (1994) 22. Weschler, H.: An overview of parallel hardware architectures for computer vision. Int. J. Patt. Recogn. Artif. Intell. 6, 629–649 (1992) 23. Wu, A.Y., Bhaskar, S.K., Rosenfeld, A.: Parallel processing of region boundaries. Pattern Recognition 22, 165–172 (1989) 24. Wu, A.Y., Rosenfeld, A.: Parallel processing of encoded bit strings. Pattern Recognition 21, 559–565 (1988) 25. Yalamanchili, S., Aggarwal, J.K.: Analysis of a model for parallel image processing. Pattern Recognition 18, 1–16 (1985) 26. Zhang, D., Pal, S.K. (eds.): Neural Networks and Systolic Array Design. Series in Machine Perception and Artificial Intelligence, vol. 49. World Scientific, Singapore (2002)
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse Marcin Gorawski Institute of Computer Science, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland
[email protected]
Abstract. In this paper several new aspects of spatial data warehouse modeling are presented. The extended cascaded star schema in spatial distributed data warehouse DSDW was defined. Research proven that there is a strong need for building many SDW’s extended cascaded star schemas as an outcome of separate spatio-temporal telemetric conceptual models. For one of these new data schemas, the definitions of cascaded ECOLAP operations were presented. These operations base on a relation algebra, and make possible ad-hoc queries executing.
1
Introduction
The designing of every data warehouse begins with creation of logical model that in an effective way presents described phenomenon. The rapid development in the field of the data warehouse is the cause for more sophisticated concepts to be modeled. Particularly, the Spatial Data Warehouse (SDW) store multidimensional data (data hyper-dimensionality) and describe processes that consists of multi-theme [1,11]. Hence exists a necessity to defining new logic schemas of SDW and models of on-line analytical processing OLAP, both for spatial and flat data [10]. Significant research in the area of SDW and Distributed Spatial Data Warehouses (DSDW) [2,5,6,7,8,9]. Previous works in the field of data modeling have contributed to creation of few, commonly used schemas. In section 2, the cascaded star schema is discussed and an example is presented, that shows disadvantages of this schema. Section 3 contains the formal definition of a new data model: the extended cascaded star, and in section 4 the definition of EOLAP operations for the new schema is presented. Section 5 summarizes the paper. Regulation of energy sector in EU created new market - media (electricity, gas, heat, and water) recipient market. The main problem is automated reading of hundreds of thousands recipients meters and critically fast analysis of terabyte sets of meters’ data. Comprehensive spatial telemetric data analysis can be performed by use of the technology called Automatic (Integrated) Meter Reading (A(I)MR) on the one hand, and data warehouse based Decision Support Systems (DSS) on the other [4,5]. Our solution consists of two layers (Fig. 1): the first is a Telemetric System of Integrated Meter Readings and the second is Distributed Spatial Telemetric Data Warehouse. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 166–175, 2010. Springer-Verlag Berlin Heidelberg 2010
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse Local radio communication
167
GSM Network
DSDW(t)
GSM
AIUT GSM Telemetric Server
TCP/IP
Data distribution in ETL process Data loading
DSTDW Servers
Client – server RMI communication DSTDW Client
33 10
12 34
31
1
21 15
14 22 30
Fig. 1. Two layers of a telemetric infrastructure
Telemetric system of integrated meter readings (TSIMR) is based on A(I)MR and GSM/GPRS technology. The system sends the data from meters located in a huge geographical region to the telemetric server using the GSM mobile phone net utilizing GPRS technology. AIUT GSM system possesses all those features. Distributed Spatial Telemetric Data Warehouse System (DSDW(t)) is a decision making support system. The decisions concerning given medium production are made basing on the short-term prognosis of a medium consumption. The prognosis are calculated using the analysis of the data gathered in DSDW(t) what, in turn, is supplied with data by the telemetric server. In order to meet the need for improved performance created by growing data sizes in Spatial Telemetric Data Warehouse (SDW(t)), parallel processing and efficient indexing become increasingly important [5]. DSDW(t) has a distributed architecture of SN type (shared nothing). The communication between nodes is made only through network connections (message passing) in a RMI technology (Remote Method Invocation). Relational analytical on-line processing (ROLAP) for spatial data, is performed in the DSDW(t) system based on cascaded star model [3] indexed by an aR-tree [4].
2
Data Models
The star model (schema) is a logical structure that consists of one, containing a huge amount of data, fact table and few smaller dimension tables. Between all dimension tables and the fact table the referential integrity is defined. Dimension tables are denormalized, however, the fact table is normalized. The advantages of this schema are: simple structure and high efficiency in query execution, that results from a low number of defined connections. One of the disadvantages of this schema is long data loading time, that results from tables denormalization. Another problem is that the star schema can be used only for data related to a single theme. In the snowflake schema the dimension tables are not denormalized. As a result of separation of dimension tables, the hierarchy is explicitly highlighted. Thanks to table normalization, in this model there is no data redundancy, and
168
M. Gorawski a
c
T
b
e
d f g
Fig. 2. Cascaded star schema S C with central fact table T
the schema is easy to modify. An additional virtue of normalization is faster data loading, but it has an influence on the answer time, which is longer than in a star schema. In some cases there is a need that few fact tables share the same dimension. That is where the fact constellation is used. This model is used very rarely, since there are not many concepts which can be modeled with that schema. Logical structures of OLAP models described above are not able to optimally model spatial data with regard for theirs hyper-dimensional and multitheme. The mentioned schemas are suitable for representing data which are single-process oriented and the associated OLAP operations are processed along the dimension hierarchy. According to this, using a star or a snowflake schema, presenting a single process, causes that a complex query in DSDW relating to many parallel processes described with different dimension levels, must be defined as several separate queries, which results are combined by the user. In the paper [11] the new formalized logical ROLAP model called Cascaded Star has been presented, which allows modeling of a hyper-dimensional data and multi-thematic decision support processes. The Cascaded Star S C is characterized by a fact, that some of its dimensions form separate nested single star schemas or cascaded stars. The structure of cascaded star S C is presented on Fig. 2. Definition 1. Cascaded Star [11]. The cascaded star is defined as S C = (SS, DC , T C ), where: • SS is a set of stars, such that Si ∈ SS is a single star schema S 1 or a cascaded star schema S C . • DC is a set of dimension tables S C , where every dimension table Di ∈ DC contains [Ai , P Ki ]. Ai is a set of attributes Di , and P Ki is the primary key of dimension Di . • T C = [V, SKD , P KD ] is the fact table of the cascaded star. V is a set of central measurements. SKD = SKi is a key of Si |Si ∈ SS, and P KD = P Ki is the primary key of Di |Di ∈ DC . Even though the cascaded star schema has extended the capabilities of logic DSDW schemas modeling, there are still situations in which this model is not sufficient for describing spatial, multithematic phenomenon. An example of such
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse Node_model node_model scope max_meters service type
Meter_model meter_model scope service type
Node node node_mod el location man f date
Date Location location x y z
Meter
Address Day
day_id day month_id
Month
node client model location manuf_date install_date medium
month_id month quarter_id
Status
Client
Payment
status income employment
status payment address family_members
payment description currency
169
Street street_id name city_id
City city_id name country_id
Quarter
Country
quarter_id quater year_id
country_id name zip_code
Year year_id year
Fig. 3. A part of the DSDW(t) schema - cascaded star and hierarchical schemas
system DSDW(t), which the APAS Group is currently working on [3,6]. Below (Fig. 4) an example of a part of the DSDW(t) system is presented. It contains cascaded star schema with three fact tables (NODE, METER and CLIENT) and two hierarchical dimensions ADDRESS and DATE. According to the definition of the cascaded star presented above, the dimensions of the cascaded star can be either single star or another cascaded star schema. Therefore, the hierarchical schemas DATE and ADDRESS are separated elements, and no connections between them and the cascaded star are defined. Query: Find addresses of these clients, whose meters have been installed in year 2005. The query is related to the cascaded star as well as to the hierarchical dimension ADDRESS and DATE. This query has to be divided into several separate queries, which results have to be joined by the user. In the beginning all dates (DAY.day id) in dimension DATE, for which Y EAR.year =’2005’ have to be found. Then the results of this query are used in the second query to find meters and address identifiers for clients, whose meters have been installed in the given year (DAY.day id = M ET ER.install date AND M ET ER.client = CLIEN T.id). Having address identifiers, full information about client address in hierarchical dimension ADDRESS can be found in a separate query (CLIEN T.address = ST REET.street id). As can be seen, to execute this query three separate queries to obtain expect result have to be defined. Only one example of the DSDW(t) schema, for which the cascaded star schema is insufficient, has been shown. In paper [6] more complex examples of the logic schemas of the DSDW(t) systems have been collected, with detailed description, that proof the need to work out a new schema for this system. Interested readers are asked to look up in the mentioned paper. Even though the cascaded star schema has extended the capabilities of modeling logic schemas for DSDW, there are still situations in which this model is not sufficient for describing spatial, multi-thematic phenomenon.
170
3
M. Gorawski
Formalization of the Extended Cascaded Star Schema
Definition 1 is based on the concept of a single star cube S 1 . In the paper [11] this concept was defined incorrectly because of: (a) ambiguity of the terminology (cube) and used definitions (dimension tables, fact tables), (b) lack of conceptual completeness of the schema - the influence of data granulation is not taken under consideration. In the new definition 2 of the single star schema these formal deficiencies were eliminated. The star schema projects the objects of the conceptual model into a logical model in the following way: (a) objects that keep information about facts are stored in the form of fact tables (base tables), (b) dimensions and their attributes are stored in the form of denormalized dimensions tables, (c) in the schema occurs the minimal number of joins between the tables. In the new Definition 2 these formal deficiencies were eliminated. Definition 2. Single star. The single star schema is defined as S S = (D, F ), where: 1. D is a set of flat dimensions tables Di . Every dimension table Di ∈ D contains [Ai , IKi , Gi ] where: • Ai is a set of attributes of Di , • IKi is a set of surrogate, numerical key identifying dimension Di , • Gi is a set of data granulations level of dimension Di . 2. F = [VS , P KS , GS ] is the fact table, where: • P KS is the primary key of S S , P KS = IKi , where IKi is a key that identifies Di |Di ∈ D, • VS is a set of facts, VS = f (GS ), where f is a function, that for a given granulation level GS |GS ∈ Gi, generates the values of numerical facts VS . The disadvantages of the star schema mentioned above can be eliminated by using the snowflake schema. The main change, which was introduced in the snowflake schema is: a) the departure from the full dimension tables denormalization towards their normalization - explicit presentation of the real, hierarchical structure of the dimensions, b) the facts tables store only one level of the aggregation, c) the maintenance of the relations between attributes of the same dimension with multiplicity of 1:1, 1:N, N:1 and N:M. Depending on dimensions table’s degree of normalization, four types of the snowflake schema can be distinguished (Def. 3). Definition 3. Snowflake The snowflake schema is defined as S F = (H, R, F ), where: 1. H is a set of hierarchical dimensions, where for the relation R of type: • {1 : 1, 1 : N, N : 1} every hierarchical dimension Hi ∈ H has a determined degree Li , defined as a number of levels of the hierarchy. Hi is a set of tables creating a normalized dimension with a minimal cardinality Li . Table hj ∈ Hi contains [Ahj , P Khj , IKhj ], where Ahj is a set of
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse
171
attributes hj , P Khj is the primary key of hj and IKhj is a set of foreign keys migrating from tables hj+1 , ..., hLj . Set IKhj is empty on the highest level of the hierarchy and for the one-level hierarchy (flat dimension), • {N : M } set R is a set of tables of relations rik Hi/ik+r between attributes of one dimension Hi |Hi ∈ H, such that rik Hi/ik+r , contains combinations of identifiers of the attributes of tables hj,k , being in relation in the dimension Hi with multiplicity N : M . 2. F = [VSF , P KH ] is a fact table, where VSF is a set of measurements, and P KH is a set of primary keys P Ki of the dimensions Hi . The dimensions hierarchy in the snowflake schema is explicitly represented by the normalized dimension tables (tables H1 , H3 on Fig. 4), in which data and relations were divided on additional tables (tables h10 , h11 , h12 ∈ H1 , {h30 , h31 , h32 , relations tables r30 H3/31 } ∈ H3 ). Between the attributes, included in the same dimension Hi of the snowflake schema, are some relations, which determinate the logical connections of attributes within the confines of one dimension. These relations are transitive within the confines of one hierarchy and have their own multiplicity. As mentioned before, the current analysis of the research and the results of experiments on the cascaded star schema for the DSDW(t) justify C the creation of a new DSDW schema called Extended Cascaded Star ES{P −t} , where P is schema’s degree described by the type and kind of the substars and by their multi-version, and t is a number of themes simultaneously described. The most recent experiential logic model of SDW(t) system represents the exC SDW(t) oriented on measurement tended star of II-nd degree schema ES{II−1} C (Fig. 5). The presented schema ES{II−1} SDW(t) is the S C (SDW(t)) schema extended of two components. The first expansion of S C (SDW(t)) schema concerns the using of the hierarchical dimensions. According to definition in [11], the S C schema allows only the existence of flat dimensions and cascaded dimensions. Hierarchical dimensions, described in Def. 3, are used in the snowflake C SDW(t) schema the hierarchical dimensions schema. In the case of the ES{II−1} are: time dimension (e.g. measure time, meter installation time, map expiry date, etc) and the dimension defining the clients address. The second expansion of S C (SDW(t)) schema concerns the way of linking fact tables with cascaded dimensions. In the cascaded star the key of the star and the function g, connected with this key, is used according to definition presented by Yu et al. [11]. Yet,
H4 h12
H1
H3 h30
h10
F
h11
h31 r30-H3/31 h32
H2 Fig. 4. Logical model of the snowflake schema
172
M. Gorawski WTH_MEASURE_CONF H_WTH_METER_INST_DATE
H_WTH_MEASURE_DATE WTH_MEASURE WTH_METER
WTH_METER_LOC
MEAS_CONFIDENCE MEAS_ZONE
WTH_METER_MODEL
H_ METER_INST_DATE
METER_MODEL MEASURE
METER_LOC METER
CLIENT
DAY MONTH QUARTER YEAR
NODE H_NODE_INSTALL_DATE
CL_STATUS
NODE_MODEL NODE_LOC
CL_PAYMENT MAP MAP_TYPE
MAP_LT_LOC
H_TIME
PROVINCE COUNTRY
H_MAP_EXP_DATE MAP_RB_LOC
STREET QUARTER CITY
H_ADDRESS
C Fig. 5. ES{II−1} SDW(t) schema oriented on measurement
in order to avoid complications connected with the definition of the g function, surrogate keys in fact tables are generated. For every record in the fact table a consecutive number is generated. The disadvantage of such approach is the necessity of defining an additional column for storing the key in the fact table (this disadvantage concerns also the approach that uses the star’s key). However, using the surrogate primary keys allows to avoid problems connected with defining the g function. Most of the recent relational database systems allow to define sequences. In Definition 4, the schema of the extended cascaded star of II-nd degree, single-theme (t=1), is described. Definition 4. Extended Cascaded Star of II-nd degree for t = 1. C The schema of the extended cascaded star of II-nd degree for t = 1 ES{II−1} C C C C is defined as ES{II−1} = (SS, H , T , L ), where: 1. SS is a set of cascaded dimensions - data schemas, such that Si ∈ SS is: C a) a single star schema S S , or b) a snowflake schema S F , or c) ES{II−1} schema. 2. H C is a set of hierachical dimensions of ES C . Every hierarchical dimension Hi ∈ H C has a given degree Li defined as a number of hierarchy levels in the S F schema. 3. T C = [V C , P KSR , P KH ] is the cascaded fact table, where V C is a set of central fact measures, V C = VSF ∪ VS . P KSR = SRi is a set of surrogate keys for cascaded dimensions Si |Si ∈ SS, and P KH = P Ki is a set of primary keys of hierarchical dimensions Hi |Hi ∈ H C . The primary key P Ki of dimension Hi is the primary key of table h0 . C The ES{II−1} schema can contain nested star schemas as well as nested snowflake schemas. This expansion allows to carry out the normalization of dimension tables.
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse
4
173
ECOLAP Operations for Extended Cascaded Star Schema
C Using the new structure ES{P −t} in OLAP analysis requires defining appropriate operations to aid this analysis. Below the Extended Cascaded OLAP (ECOLAP) C operations are defined for the extended cascaded star with ES{II−1} schema. UsC ing the new structure ES{P −t} , in OLAP analysis requires defining appropriate operations to aid this analysis. Hierarchical join is an operation that performs joins for the given hierarchy. The join is performed from the lowest level in the hierarchy to the level predetermined by the second parameter of the operation. This operation is defined in the following way: Hierachical Join(Hi , hi ) = πRH (σ(h = C)(h1 ...hi )) where: the parameter Hi defines the hierarchy from the set of hierarchies of the given snowflake schema, the parameter hi is a table, till which the join should be performed. Schema Hierarchical Join (SHJ) is an operation which receives, as parameters, a set of hierarchical dimensions and a set of tables, till which the join should be performed. SHJ : H × I → N ; SHJ(Hi , hi ) = Hierarchical Join(Hi , hi ), where: H = Hi - set of hierarchical dimensions, I = hi - set of tables, which for a given hierarchy Hi define the border level of the join hi within the confines of this hierarchy, N - set of hierarchical dimensions after the joins are performed. eTraverse (extended Traverse) allows to perform the basic OLAP operations within the confines of a single dimension S S or S F for a certain level of star M = k, k > 0. It consists of Relation Operators (SPJ) on simple attributes and SHJ operations for hierarchical dimensions. Using the expression of relational algebra this operation can be written down as: eT raverse(S, I) = πRS (σ(d = C)(SDSHJ(H, I))), where: S is a S S or S F schema, D is a set of flat dimensions, H is a set of hierarchical dimensions, I is a set of tables, which for a given hierarchy Hi ∈ H define the border level of the join hi within the confines of this hierarchy, d is a set of attributes for D and H, C is a set of selections conditions, RS is a set of the projections attributes S. eDecompose (extended Decompose) operation can be expressed, using the relational algebra, as: eDecompose(Q, P, I) = πRQ (σ(d = C)(Qtraverse(P, I))), where: P is a S S or S F schema, containing a set of hierarchical dimension H and it is a single dimension for Q, which is a extended cascaded star schema C ES{II−1} , I is a set of tables, which for a given hierarchy Hi ∈ H define the border level of join hi within the confines of this hierarchy, d is a set of attributes for P , C is a set of selections conditions, Rq is a set of the projections attributes within Q.
eJump (extended Jump). The operation of a two stars join, between which there is no direct relation, results in obtaining a set , which is a cartesian product
174
M. Gorawski
of both stars. Such set will contain a huge number of records. For this reason the definition of this operation should be understood as a join of two stars with using additional fact tables that contain appropriate relations. eJump operation can be expressed: eJump(Si , Sj , Ii , Ij ) = πRi,j (eT raverse(Si , Ii )eT raverse(Sj , Ij ))), where: Si and Sj indicate the source and destination of a S S or S F schema, containing the sets of hierarchical dimensions HSi and HSj , Ri,j is a set of projection operations attributes from Si and Sj , Ii and Ij are sets of tables, which for a given hierarchy Hi ∈ HSj and Hj ∈ HSj define the border level of the join hi and hj . eCascaded-roll-up. The Roll-up operation decreases the number of the presented data details, as a result some dimensions will be omitted. The starting point is a single star S S or a snowflake S F . Performing eTraverse operation, a set of attributes from a set of single schema is received. Then there is a move to C schema. Using expression a lower level to the parent star, which is a ES{II−1} of algebra relation this process can be written down as: eCascaded roll up(Q, P, Iq , Ip ) = eDecompose(Q, eT raverse(P, Ip ), Iq ), C where: Q is the ES{II−1} schema, P is a single star schema S S or a snowflake schema S F containing a set of hierarchical dimension H, Ip and Iq are sets of tables, which for a given hierarchy Hp ∈ H and Hq ∈ H define the border level of join hp and hq within the confines of these hierarchy. eCascaded-drill-down. This operation is the opposite to eCascaded-roll-up. It increases the number of details of the presented data, and in some particular cases the eCascaded-drill-down operation causes the appearance of new dimensions. Starting from the central star and performing consecutive eDecompose operations, a single star level is reached, where the operation eTraverse is performed. Notation in relational algebra is as following: eCascaded drill down(Q, P, Ipq , Iq ) = eT raverse(eDecompose(Q, P, Ipq ), Iq ), C where: Q is the ES{II−1} schema, P is a single star schema S S or a snowflake F schema S containing a set of hierarchical dimension H, Ipq and Iq are sets of tables, which for given hierarchy Hpq ∈ H and Hq ∈ H define the border level of join hpq and hq within the confines of these hierarchy. eCascaded-slice and eCascaded-dice. These operations reduce the number of presented elements for a multidimensional cube by setting additional selection conditions for given attributes of the substars (for eCascaded-slice operation they are attributes of a single dimension - star or snowflake) [5]. Multiple cubes operation (MCUBE) performs recursive calculations of type CUBE on relation such as base tables or materialized views [6].
5
Conclusions
In this paper a complex data modeling schema was presented - the extended cascaded star for distributed spatial data warehouses, with reference to the single star schema and the snowflake schema. An example of such system is a spatial
Extended Cascaded Star Schema for Distributed Spatial Data Warehouse
175
telemetric data warehouse DSDW(t). Also the basic cascaded ECOLAP operations were defined for the extended cascaded star schema: drill-down, roll-up, slice & dice and MCUBE, allow to analyze data, with usage of the new data C . Future works will be focused on defining ECOLAP operations model ES{II−1} for other schemas of the extended cascaded star and on the expansion of the set of operations on next elements taking under consideration data mining [6].
References 1. Atluri, V., Adam, N., Yu, S., Yesha, Y.: Efficient storage and management of environmental information. In: 19th IEEE Symposium on Mass Storage Systems, Maryland, USA (2002) 2. Bdard, Y., Merret, T., Han, J.: Fundamentals of Spatial Data Warehousing for Geographic Knowledge Discovery In Geographic Data Mining and Knowledge Discovery. In: Research Monographs in GIS, ch. 3, pp. 53–73. Taylor & Francis, London (2001) 3. Gorawski, M., Gorawski, M.J.: Modified R-MVB Tree and BTV Algorithm Used in a Distributed Spatio-temporal Data Warehouse. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 199– 208. Springer, Heidelberg (2008) 4. Gorawski, M., Malczok, R.: Materialized aR-tree in Distributed Spatial Data Warehouse. International Journal Intelligent Data Analysis 10(4), 361–377 (2006) 5. Gorawski, M.: Extended Cascaded Star Schema and ECOLAP Operations for Spatial Data Warehouse. In: Corchado, E., Yin, H. (eds.) IDEAL 2009. LNCS, vol. 5788, pp. 251–259. Springer, Heidelberg (2009) 6. Gorawski M.: Advanced Data Warehouse. Dissertation, Studia Informatica, vol. 30(3B), Pub. House of Silesian Univ. of Technology, p. 338 (2009) 7. Han, J., Stefanovic, N., Kopersky, K.: Selective Materialization: An Efficient Method for Spatial Data Cube Construction. In: Wu, X., Kotagiri, R., Korb, K.B. (eds.) PAKDD 1998. LNCS, vol. 1394. Springer, Heidelberg (1998) 8. Jensen, C., Kligys, A., Pedersen, T., Timko, I.: Multidimensional Data Modeling for Location-based Services. VLDB Journal 13, 1–21 (2004) 9. Stefanovic, N., Han, J., Koperski, K.: Object-based selective materialization for efficient implementation of spatial data cubes. In: IEEE TKDE, pp. 938–958 (2000) 10. Timoko, I., Pedersen, T.: Capturing Complex Multidimensional Data in Locationbased Warehouses. In: Proc. of CAN GIS, LNCS. Springer, Heidelberg (2004) 11. Yu, S., Atluri, V., Adam, N.: Cascaded star: A hyper-dimensional model for a data warehouse. In: Bressan, S., K¨ ung, J., Wagner, R. (eds.) DEXA 2006. LNCS, vol. 4080, pp. 439–448. Springer, Heidelberg (2006)
Parallel Longest Increasing Subsequences in Scalable Time and Memory Peter Krusche and Alexander Tiskin DIMAP and Department of Computer Science, The University of Warwick, Coventry, CV4 7AL, UK
Abstract. The longest increasing subsequence (LIS) problem is a classical problem in theoretical computer science and mathematics. Most existing parallel algorithms for this problem have very restrictive slackness conditions which prevent scalability to large numbers of processors. Other algorithms are scalable, but not work-optimal w.r.t. the fastest sequential algorithm for the LIS problem, which runs in time O(n log n) for n numbers in the comparison-based model. In this paper, we propose a new parallel algorithm for the LIS problem. Our algorithm solves the more general problem of semi-local comparison of permutation strings of length n in time O(n1.5 /p) on p processors, has scalable communi√ cation cost of O(n/ p) and is synchronisation-efficient. Furthermore, √ we achieve scalable memory cost, requiring O(n/ p) of storage on each processor. When applied to LIS computation, this algorithm is superior to previous approaches since computation, communication, and memory costs are all scalable.
1
Introduction
The longest increasing subsequence (LIS) problem is a classical problem in theoretical computer science and mathematics. We are given a sequence of n numbers, and we are asked to find the LIS. The problem is closely related to patience sorting, and has been studied extensively, beginning as early as [1] (see also e.g. [2] and references therein). In the patience sorting problem, we are looking for a cover of a given sequence of n integers which consists of a minimal number of decreasing subsequences. The number of decreasing subsequences of such a minimal cover is equal to the length of the LIS. For sequences of length n, the fastest sequential algorithm in the comparison-based model (which only allows less-than or equal comparison on the sequence elements) for patience sorting/LIS computation runs in time O(n log n)1 . A few parallel algorithms have been proposed for the LIS problem, however, these algorithms either do not achieve work-optimality w.r.t. the fastest sequential running time, or have rather restrictive slackness conditions. We would first 1
Research supported by the Centre for Discrete Mathematics and Its Applications (DIMAP), University of Warwick, EPSRC award EP/D063191/1. It is possible to achieve O(n log log n) in the integer arithmetic model. In this paper, we work in the comparison-based model.
R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 176–185, 2010. c Springer-Verlag Berlin Heidelberg 2010
Parallel Longest Increasing Subsequences in Scalable Time and Memory
177
like to point out that the problem is trivially in the complexity class NC, as it is an instance of the shortest path problem in a grid dag, which is in NC by reduction to (min, +) matrix multiplication [3]. However, the parallel algorithm resulting from this reduction is far from being work-optimal. A better generic approach is to reduce the problem to computing the longest common subsequence (LCS) of two strings of length n (see e.g. [4]). However, this is clearly not workoptimal as it gives cost O(n2 /p) on p processors. On the EREW PRAM model with p processors, Nakashima and Fujiwara [5] showed that the problem can be solved using computation O((n log n)/p), but only if p < n/k2 where k is the length of the LIS. This algorithm becomes asymptotically sequential once √ k > n. Any sequence of n numbers must have either a monotonically √ increasing or a monotonically decreasing subsequence of minimum length n [1]. Therefore, for any sequence of numbers, the condition p < n/k2 definitely inhibits parallelism either for running the algorithm on the sequence itself, or on its reversal. Sem´e [6] gave a CGM algorithm runs in O(n log(n/p)). Asymptotically, this is not faster than the sequential algorithm because the number of processors only appears in a lower order polynomial term. We consider the LIS problem as a special case of computing the LCS of two permutation strings, which can be solved sequentially in time O(n log n) in the comparison-based model [7]. While multiple work-optimal and scalable algorithms for the general LCS problem exist2 , it remains open to obtain a similar result for the LIS problem. It is remarkable that no such result is known, considering how well-studied the LIS problem is in most other aspects. The contribution of this paper is a parallel algorithm which extends the algorithm from [10] to be applicable to semi-local comparison of permutation strings, which as a special case also solves the LIS problem. For permutation strings of length n, the fastest sequential algorithm for semi-local comparison runs in O(n1.5 ) [11]. For comparing two arbitrary permutations of size n semi-locally on p processors, we obtain computation cost O(n1.5 /p), scalable communica√ √ tion cost O(n/ p), and scalable memory cost O(n/ p). Although this is not work-optimal for the LIS problem since the fastest sequential algorithm for this problem runs in O(n log n) = o(n1.5 ), using our algorithm can be advantageous since it is the asymptotically best parallel algorithm which actually achieves some scalability on this problem.
2
Definitions and Problem Analysis
Let x = x1 x2 . . . xn and y = y1 y2 . . . yn be two strings of length n over an alphabet Σ. In permutation string comparison, we have |x| = |y| = |Σ| = n, and x and y each contain exactly one instance of every character. A substring of any string x can be obtained by removing zero or more characters from the beginning and/or the end of x. A subsequence of string x is any string that can be obtained by deleting zero or more characters, i.e. a string with characters xjk , 2
The problem can be solved by combining the dynamic programming approach from [8] with the BSP algorithm for grid dag computation from [9].
178
P. Krusche and A. Tiskin
where jk < jk+1 and 1 jk m for all k. The longest common subsequence (LCS) of two strings is the longest string that is a subsequence of both input strings. The LIS problem for a string x can be solved by observing that any LCS of x and the sequence of all characters from Σ in ascending order is a LIS of x. We note that we are only interested in the length of the LCS/LIS here, but that the techniques used also allow retrieval of the actual sequence. Throughout this paper we will denote the set of integers {i, i + 1, . . . , j} by [i : j], and the set {i + 12 , i + 32 , . . . , j − 12 } of odd half-integers by i : j. Odd halfinteger variables are marked using a ˆ symbol. When indexing a matrix M by odd-half integer values ˆı and ˆj, we define that M(ˆı,ˆj) = M(i, j) with i = ˆı − 1/2 and j = ˆj − 1/2. Therefore, if a matrix has integer indices [1 : m] × [1 : n], it has odd-half integer indices 0 : m × 0 : n. Further, we define the distribution D(ˆı,ˆj) with (ˆı,ˆj) ∈ i : m × 0 : j. matrix DΣ of matrix D as DΣ (i, j) = The standard dynamic programming LCS algorithm compares all prefixes of one string to all prefixes of the other string [8] by dynamic programming. Semilocal string comparison is an alternative to this standard method. Solutions to the semi-local LCS problem are given by a highest-score matrix A,3 which we define as A(i, j) = |LCS(x, yi . . . yj )|. Since the values A(i, j) for different i and j are strongly correlated, it is possible to derive an implicit, space-efficient representation of matrix A(i, j): Lemma 1 ([11]). Let A be the highest-score matrix for comparing two permutation strings x and y of length n. The corresponding implicit highest-score matrix DA is a permutation matrix of which only a core of 2n nonzeros needs to be stored, and which can be computed in O(n1.5 ) time. We have: A(i, j) = j − i − DΣ A (i, j). Tiskin [13] further described a procedure which, given two implicit highest-score matrices which correspond to comparing two strings x and y to the same string z, computes the implicit highest-score matrix for comparing their concatenation xy and z. This procedure can be reduced to the multiplication of two integer matrices in the (min, +) semiring. We will describe this method and its parallelisation briefly in the next section. For analysing parallel algorithms, we use the BSP model introduced by Valiant[14]. A BSP computation runs on p identical processors, and its execution is divided into S supersteps that each consist of local computation and a communication phase. At the end of each superstep, the processes perform a barrier-style synchronization. A BSP computation’s efficiency on p processors is characterized by its computation cost W, which gives the total time spent on computation, and its communication cost H, which gives the maximum total number of data elements that need to be transferred between any two processors throughout the computation. We will further consider memory cost M, which is the maximum amount of storage necessary on a processor across all supersteps. Consider a problem of size n which has input/output size I(n). We assume that a given sequential algorithm for this problem runs in time W(n) and memory 3
These highest-score matrices are related to the DIST matrices used e.g. in [12].
Parallel Longest Increasing Subsequences in Scalable Time and Memory
179
M(n). A BSP algorithm solving the problem is work-optimal relative to the given sequential algorithm if W(n, p) = O(W(n)/p). Let c > 0 be a constant. The algorithm achieves scalable communication if H(n, p) = O(I(n)/pc ). The algorithm achieves scalable memory if M(n, p) = O(M(n)/pc ).
3
Highest-Score Matrix Multiplication
We will now outline the matrix multiplication algorithm from [13] and the parallel algorithm from [10] as far as is necessary to understand the new method in the next section. As an input, we have the nonzeros of two N × N permutation matrices DA and DB . The procedure computes an N × N permutation matrix DC , such that the permutation distribution matrix DΣ C is the (min, +) product of perΣ Σ Σ mutation distribution matrices DΣ and D : D (i, j) = minj (DΣ A B C A (i, j)+DB (j, k)) with i, j, k ∈ [1 : N]. Matrices DA , DB and DC have half-integer indices rangΣ ing over 0 : N. The values DΣ A (i, j) + DB (j, k) are called elementary (min, +) products. The matrix multiplication method uses a divide-and-conquer approach that recursively partitions the output matrix DC into smaller blocks in order to locate its nonzeros. We compute the number of nonzeros contained in each block. The base of the recursion has two cases: either if the block does not contain any nonzeros, or if it is of size 1 × 1 and thus specifies the location of a nonzero. A rectangular submatrix of a permutation matrix D described by an interval I of indices in D will be called a D-block, and we denote the number of nonzeros contained in such a block as D(I) = (ˆı,ˆj)∈I D(ˆı,ˆj). Consider the square DC block with index interval I = i0 − h : i0 × k0 : k0 + h. To compute the number of nonzeros contained within this block, we need the values DΣ C (i0 , k0 ), Σ Σ (i − h, k ), D (i , k + h), and D (i − h, k + h) of the distribution matrix DΣ 0 0 C 0 C 0 0 C 0 at the four corners of the block, since D(I) = DΣ (i0 − h, k0 + w) − DΣ (i0 , k0 + w) − DΣ (i0 − h, k0 ) + DΣ (i0 , k0 ). Furthermore, the differences between the DΣ C values within our DC -block from Σ DC (i0 , k0 ) are determined only by sets of relevant nonzeros in DA and DB . The relevant nonzeros for a DC -block corresponding to indices in i0 − h : i0 × k0 : k0 + h are located in the DA -block with indices RA = i0 − h : i0 × 0 : N, and in the DB -block with indices RB = 0 : N × k0 : k0 + h. For carrying out our matrix product using only the density matrices DA and DB , we split these blocks at position j to obtain only the nonzeros relevant for the (min, +) products at the lower left corner of the block: RA (j) = i0 − h : i0 × 0 : j and RB (j) = j : N × k0 : k0 + h. As the block size decreases, fewer nonzeros are relevant for a block. We can therefore represent all data necessary to independently locate the nonzeros in our DC -block in O(h) space. For each DC -block, we need to know the relevant Σ nonzeros and the sequence of minima MI (d) = minj∈J(d) (DΣ A (i0 , j)+DB (j, k0 )), where J(d) = {j | D(RA (j)) − D(RB (j)) = d}. Sequence M stores the locally minimal elementary (min, +) products in the intervals of consecutive j-values where the number of relevant nonzeros does not change. For our DC -block, MI
180
P. Krusche and A. Tiskin
can be stored in O(h) space. Further, for a DC -subblock with index interval I ⊂ I, we can compute MI and the relevant nonzeros in time O(h). We use this method to set up the input data for the recursive calls when partitioning of the DC -blocks. This recursive method can be parallelised as follows (see [10] for more details). The sequential recursive divide-and-conquer computation has at most p independent problems at level 12 log2 p, since it uses a quadtree partitioning. In the parallel version of the algorithm, we start the recursion directly at this √ √ level. We assign a different DC -block of size N/ p × N/ p with index interval √ √ Iq = iq − N/ p : iq × kq : kq + N/ p to each processor q, and compute the relevant nonzeros and sequences MIq from scratch. To compute the values of sequence MIq for every processor q, we must compute the elementary (min, +) √ Σ products DΣ p is A (iq , j) + DB (j, kq ) with j ∈ [0 : N]. We assume w.l.o.g. that an integer, and that every processor has an unique identifier q with 0 q < p. Furthermore, we assume that all parameters match, such that e.g. N/p is integer. The initial distribution of the nonzeros of the input matrices is assumed to be even among all processors, so that every processor holds N/p nonzeros of DA and DB . Lemma 2 ([10]). Given the distributed implicit highest-score matrices DA and DB , all the p · N elementary (min, +) products can be computed on a BSP com√ puter in W = O(N/ p), H = O(p + N/p)[= O(N/p) if N > p2 ], and S = O(1). Lemma 3 ([10]). If every processor holds N/p elementary (min, +) products, as well as at most N/p nonzeros in DA and the same number of nonzeros in DB , we can redistribute data such that each processor q holds the sequence MIq and the relevant nonzeros for block Iq in one superstep using communication √ H = O(N/ p). These two theorems can be combined to obtain a first result for the BSP complexity of parallel highest-score matrix multiplication [10]. We can compute the elementary (min, +) products using Lemma 2, and redistribute them efficiently using Lemma 3. The resulting parallel highest-score matrix multiplication pro√ √ cedure has BSP cost W = O((N/ p)1.5 ) = O(N1.5 /p0.75 ), H = O(N/ p), and S = O(1).
4
Parallel Semi-local Comparison of Permutation Strings
The highest-score matrix multiplication procedure shown in the previous section is not work-optimal: in the denominator of the work cost we have p0.75 instead of p, which is required for work-optimality. Because not every block at level 12 log2 p may contain the same number of nonzeros, a load-imbalance can occur. In the √ √ worst case, only p blocks contain nonzeros, and therefore only p blocks of √ size N/ p are processed in parallel. We will now show a load-balancing strategy which allows to achieve workoptimality retaining the property of scalable communication, and therefore allows to use the procedure from the previous section for permutation string comparison. We first prove a small extension to the theorems by Tiskin [13].
Parallel Longest Increasing Subsequences in Scalable Time and Memory
181
Lemma 4. Consider a DC -block of size h × h in which we would like to locate k nonzeros. Assume we are given sequence MI for this block and the relevant nonzeros in DA and √ DB . On a single processor, the nonzeros in the DC -block can be located in O(h k) time. Proof. When partitioning the DC -block recursively, the number of subblocks can increase geometrically only up to level 1/2 log k. At this level, only k blocks can contain nonzeros from the subset we are looking for. Therefore, k indepen√ dent subproblems exist, each of which requires time O(h/ k) in order to be partitioned further. Since DC is a permutation matrix, the size of the k subproblems must decrease geometrically from this level on. Therefore, the running time is dominated by level √ 1/2 log k, and the cost in the worst case is bounded √ by O(h/ k · k) = O(h k). We apply Lemma 4 to have each processor locate nonzeros contained within a different set of leaves of the partitioning quadtree. One way to implement this is to pick subsets of nonzeros based on the order in which the sequential algorithm discovers the nonzeros. When partitioning, we then know for each subblock how many nonzeros the sequential algorithm would have discovered in this block, and we can therefore only partition those blocks which contain nonzeros in the subset we are looking for. After the pre-processing phase in which we obtain the relevant nonzeros and minima MI in each of the p DC -blocks, we introduce another step to balance the work of locating nonzeros. Lemma 5. Highest-score matrix multiplication of two N × N implicit highest√ score matrices on p processors has BSP cost W = O(N1.5 /p), H = O(N/ p), and S = O(1). √ Proof. Each processor holds data of size O(N/ p) to process a DC -block of √ √ size N/ p × N/ p. If this block contains more than N/p nonzeros, it cannot be processed work-optimally by a single processor. Therefore, we partition all blocks into zero or more complete groups with exactly N/p nonzeros, and possibly one incomplete group which contains less than N/p nonzeros. If the block contains at most N/p nonzeros, then we have a single group in this block. We broadcast the data for the block to enough processors such that each processor has to locate exactly one complete group of nonzeros. This is possible with BSP cost √ H = O(N/ p), and S = O(1) using a two-phase broadcast as shown in [15] (pp. 66–67). Each processor then only locates the N/p nonzeros in its respective group, and possibly the nonzeros in one incomplete group if one existed for its √ block. The work for locating N/p nonzeros in a block of size N/ p is O(N1.5 /p) due to Lemma 4. Moreover, the maximum number of complete groups is p, since there are only N nonzeros in total. Furthermore, maximally one incomplete group per processor can exist, which limits the number of such groups to p. The maximum total number of separate groups of nonzeros is therefore 2p. Each of these groups can be processed in time O(N1.5 /p) on a single processor, which gives the claimed work-optimal bound.
182
P. Krusche and A. Tiskin
Tiskin [11] showed that we can solve the semi-local permutation string comparison problem sequentially in O(n1.5 ) time by iterated application of the semi-local matrix multiplication procedure shown in the previous sections (see [16] for the details). We will now show a parallel algorithm for semi-local comparison of permutation strings. Theorem 1. The semi-local LCS problem for permutation strings of length n can be solved on a BSP computer using local computation W = O(n1.5 /p), com√ munication H = O(n/ p) and S = O(log p) supersteps. Proof. We partition one of the input strings into substrings of length n/p, and compute the highest-score matrix for each of these substrings compared to the other input string in time O((n/p)1.5 ) = O(n1.5 /p1.5 ). This is possible because the inputs are permutation strings, and therefore only n/p character matches exist for each substring. Furthermore, if we only store non-trivial nonzeros, i.e. these outside the main diagonal, the space necessary for representing the size of the resulting implicit highest score matrix reduces to O(n/p) (see [16]). We then start merging the highest-score matrices in parallel using the workoptimal parallel highest-score matrix multiplication algorithm. Consider level l ∈ [0 : log p] of the merging process, and let r = 2l . We again use the fact that the resulting highest score matrix for comparing a permutation string and a permutation-substring of length 2rn/p can be stored in O(2rn/p) space. Furthermore, each merge is performed by a group of 2r processors. By taking the sum over all r corresponding p levels of the merging process, we get to the log 1.5 (2rn/p) /(2r)) = O(n1.5 /p) and communication computation cost W = O( √ r √ cost H = O( r 2rn/p/( 2r)) = O(n/ p). This analysis includes the top level of the merging tree where r = p. The number of supersteps S = O(log p), as the merging tree contains log p levels. Theorem 2. The semi-local LCS problem for permutation strings of length n √ can be solved using scalable memory cost M = O(n/ p) on each processor. √ √ Proof. Because H = O(n/ p), the memory cost is lower-bounded by O(n/ p). √ The recursive partitioning procedure only requires memory O(n/ p) for blocks √ √ of size n/ p × n/ p because each new level of the recursion can re-use the memory used by the data for the previous level. It remains to show that we only need to store a fraction of n/p of both input strings on each processor. This is obviously the case for input string x, which is partitioned into disjoint substrings of length n/p. However, we must still show that it is not necessary for each processor to store the entire other input string y. Initially, each processor q ∈ ¯ q = y(q−1)· np +1 . . . yq· np . ¯q = x(q−1)· np +1 . . . xq· np and y [1 : p] holds substrings x ¯q in y, we need to find In order to determine the position of each character from x the sorting permutation of y, after which each processor can retrieve the positions ¯q of x. In the comparison-based model, this is of all characters in its fraction x possible in BSP cost W = O((n log n)/p), H = O(n/p) and S = O(1) supersteps e.g. using parallel sorting by regular sampling (see [17,18]). The computation
Parallel Longest Increasing Subsequences in Scalable Time and Memory
183
and communication costs for this sorting step are dominated by the costs of the iterated highest-score matrix multiplication procedure shown in Theorem 1. After sorting, we can redistribute the data using communication O(n/p) such that each processor holds all matches for matching its substring of x against y. At no point in this initial sorting and redistribution procedure do we need to store more than O(n/p) elements of data on any processor. Therefore, the memory √ √ cost is dominated by the cost for carrying out the recursion on the n/ p×n/ p√ sized blocks of the output highest-score matrix, which is O(n/ p). We have now shown that our algorithm is scalable in computation, communication and memory for semi-local permutation string comparison. However, the question remains what speedup we can achieve for LIS computation. For estimating this speedup, consider the following simple performance model. We compute the LIS of a sequence with n elements using p processors. We fix a constant g which specifies how long data transmission takes in comparison to computation4 . We consider two different input data distributions. First, we assume that each processor initially holds an equal fraction of the sequence, which could e.g. be the case if the sequence is a result of a previous parallel computation. In the other distribution, the entire sequence is known to every processor before starting the computation. Let r be the record size which specifies the size of each element in the input sequence. If the input sequence is distributed in equal fractions, we have account for bseq = rn data elements which need to be distributed to all processors first. In the parallel algorithm, we need only to compensate for having a distributed input by adding a term bpar = r · n/p, as each processor only needs a n/p-sized fraction of the input. If the input sequence is known to all processors in advance, we have bseq = bpar = 0. The running time for the sequential computation is then given by wseq = n log n + gbseq (neglecting constant factors, as we are only looking at the performance ratio compared to the parallel algorithm). The running time of the parallel computation using p √ processors is wpar(p) = 21.5 · n1.5 /p + g · (p + n/ p + bpar ).5 In our estimation, we assume g = 10, such that transmitting one element of data takes ten times as long as one step of the LIS computation (i.e. we assume that the network connecting the processors is very fast), and further that r = 2 (i.e. each sequence element requires the same amount of memory as two integers). Figures 1 and 2 show the speedup wseq /wpar(p) for various problem sizes obtained using this estimation. We see that speedup can be achieved under these conditions. This speedup is below the ideal linear speedup since our algorithm is restricted by the fact that the communication cost does not scale optimally, but √ only by a factor √ of 1/ p. Also, the number of processors must be bounded from below by Ω( n/ log n) in order to achieve speedup over the fast sequential algo√ rithm. In order to achieve speedup higher than p, a fast communication network is required. If the communication network is too slow, the communication cost of 4 5
In the standard BSP model [14], this parameter is called the inverse bandwidth or communication gap. The factor of 21.5 comes from the fact that the largest highest-score matrices multiplied in our computation are of size 2n × 2n.
184
P. Krusche and A. Tiskin
Speedup
Speedup
8
Seq.
3
Seq.
p4
6
p4 2
p16
p16
4 p32
p32 1
2
p64
1000
10 000
100 000
Sequence Length 1 000 000
p64
1000
10 000
100 000
Sequence Length 1 000 000
Fig. 1. Speedup estimation if sequence is Fig. 2. Speedup estimation if sequence is distributed in equal fractions known to all processors initially
the highest-score matrix multiplication algorithm becomes dominant, particularly since it is necessary to use many processors to achieve speedup over sequential computation for long sequences. Improving the method to achieve perfectly scalable communication as well as computation would therefore greatly improve its overall scalability. We would like to point out that these speedup estimations only serve the purpose of evaluating the practical potential of a stronger theoretical result. Our main aim is to show a first approach to parallel LIS computation which is scalable, and pointing out that despite the LIS problem being well-studied, no algorithms had been proposed so far which are scalable and can compete with the fastest sequential method.
5
Conclusions and Outlook
In this paper, we have presented a new parallel algorithm for semi-local permutation string comparison. An important application for this algorithm is LIS computation. Although the computation cost for our algorithm is not asymptotically work-optimal in comparison the the best sequential algorithm for the LIS problem, it is in fact the first coarse grained parallel algorithm which achieves any scalability for the problem. All previous approaches either have restrictive slackness conditions for work-optimality, or only have very limited theoretical speedup over just running sequentially. For LIS computation, using or algorithm can be of advantage for computations in which many processors exist and are connected by a very fast interconnection network (e.g. to achieve at least some speedup on a shared memory machine), or if either the input data or the resulting LIS needs to be distributed to all processors. It remains an open question whether it is possible to obtain a generally workoptimal parallel algorithm for the LIS problem running in time O((n log n)/p) in the comparison-based model. We believe that our method can be improved in two different ways. If the sequential algorithm for highest-score matrix multiplication can be improved to run in o(n1.5 ), speedup would be improved for sequences long enough such that the computational work becomes dominant.
Parallel Longest Increasing Subsequences in Scalable Time and Memory
185
Furthermore, improving the communication cost of the parallel algorithm would allow to use more processors to speed up the computation. Therefore, we believe that our approach has the potential to eventually yield a first truly work-optimal algorithm for LIS computation.
References 1. Erd˝ os, P., Szekeres, G.: A combinatorial problem in geometry. Compositio Math. 2, 463–470 (1935) 2. Bespamyatnikh, S., Segal, M.: Enumerating longest increasing subsequences and patience sorting. Information Processing Letters 76, 7–11 (2000) 3. McColl, W.F.: General purpose parallel computing. In: Lectures on parallel computation, pp. 337–391. Cambridge University Press, New York (1993) 4. Garcia, T., Myoupo, J., Sem´e, D.: A work-optimal CGM algorithm for the longest increasing subsequence problem. In: Proc. PDPTA (2001) 5. Nakashima, T., Fujiwara, A.: A cost optimal parallel algorithm for patience sorting. Parallel Processing Letters 16(1), 39–52 (2006) 6. Sem´e, D.: A CGM algorithm solving the longest increasing subsequence problem. In: Gavrilova, M.L., Gervasi, O., Kumar, V., Tan, C.J.K., Taniar, D., Lagan´ a, A., Mun, Y., Choo, H. (eds.) ICCSA 2006. LNCS, vol. 3984, pp. 10–21. Springer, Heidelberg (2006) 7. Hunt, J.W., Szymanski, T.G.: A fast algorithm for computing longest common subsequences. Communications of the ACM 20(5), 350–353 (1977) 8. Wagner, R.A., Fischer, M.J.: The string-to-string correction problem. J. ACM 21(1), 168–173 (1974) 9. McColl, W.F.: Scalable Computing. In: van Leeuwen, J. (ed.) Computer Science Today: Recent Trends and Developments. LNCS, vol. 1000, pp. 46–61. Springer, Heidelberg (1995) 10. Krusche, P., Tiskin, A.: Efficient parallel string comparison. In: ParCo. NIC Series, vol. 38, pp. 193–200. John von Neumann Institute for Computing (2007) 11. Tiskin, A.: Longest common subsequences in permutations and maximum cliques in circle graphs. In: Lewenstein, M., Valiente, G. (eds.) CPM 2006. LNCS, vol. 4009, pp. 270–281. Springer, Heidelberg (2006) 12. Apostolico, A., Atallah, M.J., Larmore, L.L., McFaddin, S.: Efficient parallel algorithms for string editing and related problems. SIAM J. Comp. 19(5), 968–988 (1990) 13. Tiskin, A.: Semi-local longest common subsequences in subquadratic time. J. Disc. Alg. 6(4), 570–581 (2008) 14. Valiant, L.G.: A bridging model for parallel computation. Communications of the ACM 33(8), 103–111 (1990) 15. Bisseling, R.: Parallel Scientific Computation: A Structured Approach Using BSP and MPI. Oxford University Press, Oxford (2004) 16. Tiskin, A.: Semi-local string comparison: Algorithmic techniques and applications. Extended version of [19], arXiv: 0707.3619v5 17. Shi, H., Schaeffer, J.: Parallel sorting by regular sampling. J. Parallel and Distributed Computing 14(4), 361–372 (1992) 18. Tiskin, A.: The bulk-synchronous parallel random access machine. Theoretical Computer Science 196(1-2), 109–130 (1998) 19. Tiskin, A.: Semi-local string comparison: Algorithmic techniques and applications. Mathematics in Computer Science 1(4), 571–603 (2008)
A Scalable Parallel Union-Find Algorithm for Distributed Memory Computers Fredrik Manne and Md. Mostofa Ali Patwary Department of Informatics, University of Bergen, N-5020 Bergen, Norway {Fredrik.Manne,Mostofa.Patwary}@ii.uib.no
Abstract. The Union-Find algorithm is used for maintaining a number of non-overlapping sets from a finite universe of elements. The algorithm has applications in a number of areas including the computation of spanning trees, sparse linear algebra, and in image processing. Although the algorithm is inherently sequential there has been some previous efforts at constructing parallel implementations. These have mainly focused on shared memory computers. In this paper we present the first scalable parallel implementation of the Union-Find algorithm suitable for distributed memory computers. Our new parallel algorithm is based on an observation of how the Find part of the sequential algorithm can be executed more efficiently. We show the efficiency of our implementation through a series of tests to compute spanning forests of very large graphs. Keywords: disjoint sets, parallel Union-Find, distributed memory.
1
Introduction
The disjoint-set data structure is used for maintaining a number of nonoverlapping sets consisting of elements from a finite universe. Its uses include among other, image decompositions, the computation of connected components and minimum spanning trees in graphs, and is also taught in most algorithm courses. The algorithm for implementing this data structure is often referred to as the Union-Find algorithm. More formally, let U be a collection of n distinct elements and let Si denote a set of elements from U . Two sets {S1 , S2 } are disjoint if S1 ∩ S2 = ∅. A disjoint set data structure maintains a collection {S1 , S2 , . . . , Sk } of disjoint dynamic sets selected from U . Each set is identified by a representative x, which is usually some member of the set. The two main operations are then to find which set a given element belongs to by locating its representative element and also to create a new set from the union of two existing sets. The underlying data structure of each set is typically a rooted tree where the element in the root vertex is the representative of the set. Using the two techniques Union-by-rank and path compression the running time of any combination of m Union and Find operations is O(nα(m, n)) where α is the very slowly growing inverse Ackerman function [4]. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 186–195, 2010. c Springer-Verlag Berlin Heidelberg 2010
Scalable Parallel Union-Find Algorithm for Distributed Memory Computers
187
From a theoretical point of view using the Union-Find algorithm is close to optimal. However, for very large problem instances such as those that appear in scientific computing this might still be too slow. This is especially true if the Find-Union computation is applied multiple times and take up a significant amount of the total time. It might also be that the underlying problem is too large to fit in the memory of one processor. One recent application that makes use of repeated applications of the Union-Find algorithm is a new algorithm for computing Hessian matrices using substitution methods [7]. Hence, designing parallel algorithms is necessary to keep up with the very large problem instances that appear in scientific computing. The first such effort was by Cybenko et al. [5] who presented an algorithm using the Union-Find algorithm for computing the connected components of a graph and gave implementations both for shared memory and distributed memory computers. The distributed memory algorithm duplicates the vertex set and then partitions the edge set among the processors. Each processor then computes a spanning forest using its local edges. In log p steps, where p is the number of processors, these forests are then merged until one processor has the complete solution. However, the experimental results from this algorithm were not promising and showed that for a fixed size problem the running time increased with the number of processors used. Anderson and Woll also presented a parallel Union-Find algorithm using waitfree objects suitable for shared memory computers [1]. In doing so they showed that parallel algorithms using concurrent Union-operations risk creating unbalanced trees. However, they did not produce any experimental results for their algorithm. We note that there exists an extensive literature on designing parallel algorithms for computing a spanning forest or the connected components of a graph. However, up until the recent paper by Bader and Cong [3] such efforts had failed to give speedup on arbitrary graphs. In [3] the authors present a novel scalable parallel algorithm for computing spanning forests on a shared memory computer. Focusing on distributed memory computers is of importance since these have better scalability than shared memory computers and thus the largest systems tend to be of this type. However, their higher latency makes distributed memory computers more dependent on aggregating sequential work through the exploitation of locality. The current work presents a new parallel Union-Find algorithm for distributed memory computers. The algorithm operates in two stages. In the first stage each processor performs local computations in order to reduce the number of edges that need to be considered for inclusion in the final spanning tree. This is similar to the approach used in [5], however, we use a sequential Union-Find algorithm for this stage instead of BFS. Thus when we start the second parallel stage each processor has a Union-Find type forest structure that spans each local component. In the second stage we merge these structures across processors to obtain a global solution. In both the sequential and the parallel stage we make use of a
188
F. Manne and Md. M.A. Patwary
novel observation on how the Union-Find algorithm can be implemented. This allows both for a faster sequential algorithm and also to reduce the amount of communication in the second stage. To show the feasibility and efficiency of our algorithm we have implemented several variations of it on a parallel computer using C++ and MPI and performed tests to compute spanning trees of very large graphs using up to 40 processors. Our results show that the algorithm scales well both for real world graphs and also for small-world graphs. The paper is organized as follows. In Section 2 we briefly explain the sequential algorithm and also how it can be optimized. In Section 3 we describe our new parallel algorithm, before giving experimental results in Section 4.
2
The Sequential Algorithm
In the following we first outline the standard sequential Union-Find algorithm. We then point out how it is possible to speed up the algorithm by paying attention to the rank values. This is something that we will make use of when designing our parallel algorithm. The standard data structure for implementing the Union-Find algorithm is a forest where each tree represents a connected set. To implement the forest each element x has a pointer p(x) initially set to x. Thus each x starts as a set by itself. The two operations used on the sets are then F ind(x) and U nion(x, y) where x and y are distinct elements. F ind(x) returns the root of the tree that x belongs to. This is done by following pointers starting from x. U nion(x, y) merges the two trees that x and y belong to. This is achieved by making one of the roots of x and y point to the other. With these operations the connected components of a graph G(V, E) can be computed as shown in Algorithm 1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.
S=∅ for each x ∈ V do p(x) = x end for for each (x, y) ∈ E do if F ind(x) = F ind(y) then U nion(x, y) S = S ∪ {(x, y)} end if end for
Algorithm 1. The sequential Union-Find algorithm When the algorithm terminates the vertices of each tree will consist of a connected component and the set of edges in S define a spanning forest on G. There are two standard techniques for speeding up the Union-Find algorithm. The first is Union-by-rank. Here each vertex is initially given a rank of 0. If two sets are to be merged where the root elements are of equal rank then the rank of
Scalable Parallel Union-Find Algorithm for Distributed Memory Computers
189
the root element of the combined set will be increased by one. In all other Union operations the root with the lowest rank will be set to point to the root with the higher rank while all ranks remain unchanged. Note that this ensures that the parent of a vertex x will always have higher rank than the vertex x itself. The second technique is path compression. In its simplest form, following any Find operation, all traversed vertices will be set to point to the root. This has the effect of compressing the path and making subsequent Find operations using any of these vertices faster. Note that even when using path compression the rank values will still be strictly increasing when moving upwards in a tree. Using the techniques of Union-by-rank and path compression the running time of any combination of m Union and Find operations is O(nα(m, n)) where α is the very slowly growing inverse Ackerman function [4]. We now consider how it is possible to implement the Union-Find algorithm in a more efficient way. It is straight forward to see that one can speed up Algorithm 1 by storing the results of the two Find operations and use these as input to the ensuing Union operation which then only has to determine which of the two root vertices should point to the other. In the following we describe how it is possible in certain cases to terminate the Find operation before reaching the root. Let the rank of a vertex z be denoted by rank(z). Consider two vertices x and y belonging to different sets with roots rx and ry respectively where rank(rx ) < rank(ry ). If we find rx before ry then it is possible to terminate the search for ry as soon as we reach an ancestor z of y where rank(z) = rank(rx ). This follows since the rank function is strictly increasing and we must therefor have rank(ry ) > rank(rx ) implying that ry = rx . At this point it is possible to join the two sets by setting p(rx ) = p(z). Note that this will neither violate the rank property nor will it increase the asymptotic time bound of the algorithm. However, if we perform F ind(y) before F ind(x) we will not be able to terminate early. To avoid this we perform the two Find operations in an interleaved fashion by always continuing the search from the vertex with the lowest current rank. In this way the Find operation can terminate as soon as one reaches the root with the smallest rank. We label this as the zigzag Find operation as opposed to the classical Find operation. The zigzag Find operation can also be used to terminate the search early when the vertices x and y belong to the same set. Let z be their lowest common ancestor. Then at some stage of the zigzag Find operation the current ancestors of x and y will both be equal to z. At this point it is clear that x and y belong to the same set and the search can stop. We note that the zigzag Find operation is similar to the contingently unite algorithm as presented in [9], only that we have extracted out the specifics of the path compression technique.
3
The Parallel Algorithm
In the following we outline our new parallel Union-Find algorithm. We assume a partitioning of both the vertices and the edges of G into p sets each,
190
F. Manne and Md. M.A. Patwary
V = {V0 , V1 , . . . , Vp−1 } and E = {E0 , E1 , . . . , Ep−1 } with the pair (Vi , Ei ) being allocated to processor i, 0 ≤ i < p. If v ∈ Vi (or e ∈ Ei ) processor i owns v (or e) and v (or e) is local to processor i. Any processor i that has a local edge (v, w) ∈ Ei such that it does not own vertex v will create a ghost vertex v as a substitution for v. We denote the set of ghost vertices of processor i by Vi . Thus an edge allocated to processor i can either be between two vertices in Vi , between a vertex in Vi and a vertex in Vi , or between two vertices in Vi . We denote the set of edges adjacent to at least one ghost vertex by Ei . The algorithm operates in two stages. In the first stage each processor performs local computations without any communication in order to reduce the number of edges that need to be considered for the second final parallel stage. Due to space considerations we only outline the steps of the algorithm and neither give pseudo-code nor a formal proof that the algorithm is correct. Stage 1. Reducing the input size Initially in Stage 1 each processor i computes a spanning forest Ti for its local vertices Vi using the local edges Ei − Ei . This is done using a sequential FindUnion algorithm. It is then clear that Ti can be extended to a global spanning forest for G. Next, we compute a subset Ti of Ei such that Ti ∪ Ti form a spanning forest for Vi ∪ Vi . We omit the details of how Ti can be computed efficiently but note that this can be done without destroying the structure of Ti . The remaining problem is now to select a subset of the edges in Ti so as to compute a global spanning forest for G. Stage 2. Calculating the final spanning forest The underlying data structure for this part of the algorithm is the same as for the sequential Union-Find algorithm, only that we now allow trees to span across several processors. Thus a vertex v can set p(v) to point to a vertex on a different processors other than its own. The pointer p(v) will in this case contain information about which processor owns the vertex being pointed to, its local index on that processor, and also have a lower bound on its rank. Each ghost vertex v will initially set rank(v ) = 0 and p(v ) = v. Thus the connectivity of v is initially handled through the processor that owns v. For the local vertices the initial p() values are as given from the computation of Ti . We define the local root l(v) as the last vertex on the Find-path of v that is stored on the same processor as v. If in addition l(v) has p(l(v)) = l(v) then l(v) is also a global root. In the second stage of the algorithm processor i iterates through each edge (v, w) ∈ Ti to determine if this edge should be part of the final spanning forest or not. This is done by issuing a Find-Union query (FU) for each edge. A FUquery can either be resolved internally by the processor or it might have to be sent to other processors before an answer is returned. To avoid a large number of small messages a processor will process several of its edges before sending and receiving queries. A computation phase will then consist of first generating new
Scalable Parallel Union-Find Algorithm for Distributed Memory Computers
191
FU-queries for a predefined number of edges in Ti and then to handle incoming queries. Any new messages to be sent will be put in a queue and transmitted in the ensuing communication phase. Note that a processor might have to continue processing incoming queries after it has finished processing all edges in Ti . In the following we describe how the FU-queries are handled. A FU-query contains information about the edge (v, w) in question and also to which processor it belongs. In addition the FU-query contains two vertices a and b such that a and b are on the Find-paths of v and w respectively. The query also contains information about the rank of a and b and if either a or b is a global root. Initially a = v and b = w. When a processor receives (or initiates) a FU-query it is always the case that it owns at least one of a and b. Assume that this is a, we then label a as the current vertex. Then a is first replaced by p(l(a)). There are now three different ways to determine if (v, w) should be part of the spanning forest or not: i) If a = b then v and w have a common ancestor and the edge should be discarded. ii) If a = b, p(a) = a, and rank(a) < rank(b) then p(a) can be set to b and thus including (v, w) in the spanning forest. iii) If a = b, rank(a) = rank(b), p(a) = a, while b is marked as also being a global root then p(a) can be set to b while a message is sent to b to increase its rank by one. To avoid that a and b concurrently sets each other as parents in Case iii) we associate a unique random number r() with each vertex. Thus we must also have r(a) < r(b) before we set p(a) = b. If a processor i reaches a decision on the current edge (v, w), it will send a message to the owner of the edge about the outcome. Otherwise processor i will forward the updated FU-query to a processor j (where j = i) such that j owns at least one of a and b. In the following we outline two different ways in which the FU-queries can be handled. The difference lies mainly in the associated communication pattern and reflects the classical as opposed to the zigzag Union-Find operation as outlined in Section 2. In the classical parallel Union-Find algorithm a is initially set as the current vertex. Then while a = p(a) the query is forwarded to p(a). When the query reaches a global root, in this case a, then if b is marked as also being a global root, rules i) through iii) are applied. If these result in a decision such that the edge is either discarded or p(a) is set to b then the query is terminated and a message is sent back to the processor owning the edge in question. Otherwise, the query is forwarded to b where the process is repeated (but now with b as the current vertex). In the parallel zigzag algorithm a processor that initiates or receives a FUquery will always check all three cases after first updating the current vertex z with l(z). If none of these apply the query is forwarded to the processor j which owns the one of a and b marked with the lowest rank and if rank(a) = rank(b) the one with lowest r value. Note that if v and w are initially in the same set then a query will always be answered as soon as it reaches the processor that owns the lowest common ancestor of v and w. Similarly, if v and w are in different
192
F. Manne and Md. M.A. Patwary
sets the query will be answered as soon as the query reaches the global root with lowest rank. Since FU-queries are handled concurrently it is conceivable that a vertex z ∈ {a, b} has seized to be a global root when it receives a message to increase its rank (if Case iii) has been applied). To ensure the monotonicity of ranks z then checks, starting with w = p(z), that rank(w) is strictly greater than the updated rank of z. If not we increase rank(w) by one and repeat this for p(w). Note that this process can lead to extra communication. Similarly as for the algorithm in [1] it is possible that unbalanced trees are created with both parallel communication schemes. This can happen if more than two trees with the same rank are merged concurrently such that one hangs of the other. When a processor i receives a message that one of its edges (v, w) is to be part of the spanning forest it is possible to initiate a path compression operation between processors. On processor i this would entail to set l(v) (and l(w)) to point to the new root which would then also have to be included in the return message. Since there could be several such incoming messages for l(v) and these could arrive in an arbitrary order we must first check that the rank of the new root is larger than the rank that i has stored for p(l(v)) before performing the compression. If this is the case then it is possible to continue the compression by sending a message to p(l(v)) about the new root. We label these schemes as either 1-level or full path compression.
4
Experiments
For our experiments we have used a Cray XT4 distributed memory parallel machine with AMD Opteron quad-core 2.3 GHz processors where each group of four cores share 4 GB of memory. The algorithms have been implemented in C++ using the MPI message-passing library. We have performed experiments using both graphs taken from real application as well as on different types of synthetic graphs. In particular we have used application graphs from areas such as linear programming, medical science, structural engineering, civil engineering, and automotive industry [6,8]. We have also used small-world graphs as well as random graphs generated by the GTGraph package [2]. Table 1 give properties of the graphs. The first nine rows contains information about the application graphs while the final two rows give information about the small-world graphs. The first 5 columns give structural properties about the graphs while the last two columns show the time in seconds for computing a spanning forest using Depth First Search (DFS) and the sequential zigzag algorithm (ZZ). We have also used two random graphs both containing one million vertices and respectively, 50 and 100 million edges. Note that all of these graphs only contains one component. Thus the spanning forest will always be a tree. Our first results concern the different sequential algorithms for computing a spanning forest. As is evident from Table 1 the zigzag algorithm outperformed
Scalable Parallel Union-Find Algorithm for Distributed Memory Computers
193
Table 1. Properties of the graphs Name m t1 cranksg2 inline 1 ldoor af shell10 boneS10 bone010 audi spal 004 rmat1 rmat2
|V | 97578 63838 503712 952203 1508065 914898 986703 943695 321696 377823 504817
|E| 4827996 7042510 18156315 22785136 25582130 27276762 35339811 38354076 45429789 30696982 40870608
Max Deg 236 3422 842 76 34 80 80 344 6140 8109 10468
Avg Deg 98.95 220.64 72.09 47.86 33.93 59.63 71.63 81.28 282.44 162.49 161.92
DFS ZZ 0.12 0.06 0.15 0.03 0.57 0.26 0.71 0.47 1.04 0.37 0.86 0.38 1.05 0.47 1.20 0.33 1.33 0.66 2.07 1.34 2.71 1.81
the DFS algorithm. A comparison of the different sequential Union-Find algorithms on the real world graphs is shown in the upper left quadrant of Figure 1. All timings have been normalized relative to the slowest algorithm, the classical algorithm (CL) using path compression (W). As can be seen, removing the path compression (O) decreases the running time. Also, switching to the zigzag algorithm (ZZ) improves the running time further, giving approximately a 50% decrease in the running time compared to the classical algorithm with path compression. To help explain these results we have tabulated the number of “parent chasing” operations on the form z = p(z). These show that the zigzag algorithm only executes about 10% as many such operations as the classical algorithm. However, this does not translate to an equivalent speed up due to the added complexity of the zigzag algorithm. The performance results for the synthetic graphs give an even more pronounced improvement when using the zigzag algorithms. For these graphs both zigzag algorithms outperforms both classical algorithms and the zigzag algorithm without path compression gives an improvement in running time of close to 60% compared to the classical algorithm with path compression. Next, we present the results for the parallel algorithms. For these experiments we have used the Mondrian hypergraph partitioning tool [10] for partitioning vertices and edges to processors. For most graphs this has the effect of increasing locality and thus enabling to reduce the size of Ti in Stage 1. In our experiments T = ∪i Ti contained between 0.1% and 0.5 % of the total number of edges for the application graphs, between 1 % and 6 % for the small-world graphs, and between 2 % and 36 % for the random graphs. As one would expect these numbers increase with the number of processors. In our experiments we have compared using either the classical or the zigzag algorithm, both for the sequential computation in Stage 1 and also for the parallel computation in Stage 2. We note that in all experiments we have only used level1 path compression in the parallel algorithms as using full compression, without exception, slowed down the algorithms.
194
F. Manne and Md. M.A. Patwary Sequential spanning tree algorithms
Parallel spanning tree algorithms using 4 processors
100 100
90 90
80
Time in %
Time in %
80
70
60
70
60
50
P−CL S−CL−W
S−CL−W
50
P−ZZ S−CL−W
S−ZZ−W 40
P−ZZ S−ZZ−W
S−CL−O
P−ZZ S−CL−O
S−ZZ−O 30
m_t1
cranksg2 inline_1
ldoor af_shell10boneS10 bone010
P−ZZ S−ZZ−O
40
audi
spal_004
m_t1
cranksg2 inline_1
ldoor
Graphs
af_shell10 boneS10 bone010
audi
spal_004
Graphs
Parallel spanning tree algorithms using 8 processors
Speedup of parallel spanning tree algorithms
100
16
14
90
12
10
Speedup
Time in %
80
70
8
6
60
P−CL S−CL−W
spal_004
4
audi
P−ZZ S−CL−W
50
P−ZZ S−ZZ−W
inline_1
2
P−ZZ S−CL−O P−ZZ S−ZZ−O
40
m_t1
cranksg2 inline_1
ldoor af_shell10boneS10 bone010
Graphs
0 audi
spal_004
1
4
8
16
24
32
40
Processors
Fig. 1. Performance results: S - Sequential algorithm, P- Parallel algorithm, CL Classical Union-Find, ZZ - zigzag Union-Find, W - With path compression, O - Without path compression
How the improvements from the sequential zigzag algorithm are carried into the parallel algorithm can be seen in the upper right and lower left quadrant of Figure 1. Here we show the result of combining different parallel algorithms with different sequential ones when using 4 and 8 processors. All timings have again been normalized to the slowest algorithm, the parallel classical algorithm (PCL) with the sequential classical algorithm (S-CL), and using path compression (W). Replacing the parallel classical algorithm with the parallel zigzag algorithm while keeping the sequential algorithm fixed gives an improvement of about 5% when using 4 processors. This increases to 14% when using 8 processors, and to about 30% when using 40 processors. This reflects how the running time of Stage 2 of the algorithms becomes more important for the total running time as the number of processors are increased. The total number of sent and forwarded FU-queries is reduced by between 50% and 60% when switching from the parallel classical to the parallel zigzag algorithm. Thus this gives an upper limit on the possible gain that one can obtain from the parallel zigzag algorithm over the parallel classical algorithm. When keeping the parallel zigzag algorithm fixed and replacing the sequential algorithm in Step 1 we get a similar effect as we did when comparing the
Scalable Parallel Union-Find Algorithm for Distributed Memory Computers
195
sequential algorithms, although this effect is dampened as the number of processors is increased and Step 1 takes less of the overall running time. The figure in the lower right corner shows the speedup on three large matrices when using the best combination of algorithms, the sequential and parallel zigzag algorithm. As can be seen the algorithm scales well up to 32 processors at which point the communication in Stage 2 dominates the algorithm and causes a slowdown. Similar experiments for the small-world graphs showed a more moderate speedup peaking at about a factor of four when using 16 processors. The random graphs did not obtain speedup beyond 8 processors and even for this configuration the running time was still slightly slower than for the best sequential algorithm. We expect that the speedup would continue beyond the current numbers of processors for sufficiently large data sets. To conclude we note that the zigzag Union-Find algorithm achieves considerable savings compared to the classical algorithm both for the sequential and the parallel case. However, our parallel implementation did not achieve speedup for the random graphs, as was the case for the shared memory implementation in [3]. This is mainly due to the poor locality of such graphs.
References 1. Anderson, R.J., Woll, H.: Wait-free parallel algorithms for the union-find problem. In: Proceedings of The Twenty-Third Annual ACM Symposium on Theory of Computing (STOC 1991), pp. 370–380 (1991) 2. Bader, D.A., Madduri, K.: GTGraph: A synthetic graph generator suite (2006), http://www.cc.gatech.edu/~ kamesh/GTgraph 3. Bader, D.J., Cong, G.: A fast, parallel spanning tree algorithm for symmetric multiprocessors (smps). Journal of Parallel and Distributed Computing 65, 994– 1006 (2005) 4. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. The MIT Press, Cambridge (2001) 5. Cybenko, G., Allen, T.G., Polito, J.E.: Practical parallel algorithms for transitive closure and clustering. International Journal of Parallel Computing 17, 403–423 (1988) 6. Davis, T.A.: University of Florida sparse matrix collection. ACM Transactions on Mathematical Software (submitted) 7. Gebremedhin, A.H., Tarafdar, A., Manne, F., Pothen, A.: New acyclic and star coloring algorithms with applications to computing hessians. SIAM Journal on Scientific Computing 29, 515–535 (2007) 8. Koster, J.: Parasol matrices, http://www.parallab.uib.no/projects/parasol/data 9. Tarjan, R.E., van Leeuwen, J.: Worst-case analysis of set union algorithms. J. ACM 31, 245–281 (1984) 10. Vastenhouw, B., Bisseling, R.H.: A two-dimensional data distribution method for parallel sparse matrix-vector multiplication. SIAM Review 47, 67–95 (2005)
Extracting Both Affine and Non-linear Synchronization-Free Slices in Program Loops Wlodzimierz Bielecki and Marek Palkowski Faculty of Computer Science, Technical University of Szczecin, 70210, Zolnierska 49, Szczecin, Poland {bielecki,mpalkowski}@wi.ps.pl http://kio.wi.zut.edu.pl/
Abstract. An approach is presented permitting for extracting both affine and non-linear synchronization-free slices in program loops. It requires an exact dependence analysis. To describe and implement the approach, the dependence analysis by Pugh and Wonnacott was chosen where dependences are found in the form of tuple relations. The approach is based on operations on integer tuple relations and sets and it has been implemented and verified by means of the Omega project software. Results of experiments with the UTDSP benchmark suite are discussed. Speed-up and efficiency of parallel code produced by means of the approach is studied. Keywords: synchronization-free slices, free-scheduling, parallel shared memory programs, dependence graph.
1
Introduction
Microprocessors with multiple execution cores on a single chip are the radical transformation taking place in the way that modern computing platforms are being designed. Hardware industry is moving toward the direction of multi-core programming and parallel computing. This opens new opportunities for software developers [1]. Multithreaded coarse-grained applications allow the developer to exploit the capabilities provided by the underlying parallel hardware platform including multi-core processors. Coarse-grained parallelism is obtained by creating a thread of computations on each execution core (or processor) to be executed independently or with occasional synchronization. Different techniques have been developed to extract synchronization-free parallelism available in loops, for example, those presented in papers [2,3,4,5,6]. Unfortunately, none of those techniques extracts synchronization-free parallelism when the constraints of sets representing slices are non-linear. In our recent work [7,8], we have proposed algorithms to extract coarse-grained parallelism represented with synchronization-free slices described by non-linear constraints. But they are restricted to extracting slices being represented by a graph of the chain or tree topology only. R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 196–205, 2010. c Springer-Verlag Berlin Heidelberg 2010
Extracting Affine and Non-linear Synchronization-Free Slices in Loops
197
The main purpose of this paper is to present an approach that permits us to extract synchronization-free slices when they are represented i) by both affine and non-linerar constraints, ii) not only by the chain or tree topology but also by a graph of an arbitrarily topology. The presented algorithm is applicable to the parameterized both uniform and non-uniform arbitrary nested loop.
2
Background
In this paper, we deal with affine loop nests where, for given loop indices, lower and upper bounds as well as array subscripts and conditionals are affine functions of surrounding loop indices and possibly of structure parameters, and the loop steps are known constants. Two statement instances I and J are dependent if both access the same memory location and if at least one access is a write. I and J are called the source and destination of a dependence, respectively, provided that I is lexicographically smaller than J (I≺J, i.e., I is always executed before J). Our approach requires an exact representation of loop-carried dependences and consequently an exact dependence analysis which detects a dependence if and only if it actually exists. To describe and implement our algorithm, we chose the dependence analysis proposed by Pugh and Wonnacott [11] where dependences are represented by dependence relations. A dependence relation is a tuple relation of the form [input list ]→[output list]: formula, where input list and output list are the lists of variables and/or expressions used to describe input and output tuples and formula describes the constraints imposed upon input list and output list and it is a Presburger formula built of constraints represented with algebraic expressions and using logical and existential operators. We use standard operations on relations and sets, such as intersection (∩), union (∪), difference (-), domain (dom R), range (ran R), relation application (S = R(S): e ∈S iff exists e s.t. e→e ∈R,e∈S), transitive closure of relation R, R*. In detail, the description of these operations is presented in [11,12,13]. Definition 1. An iteration space is a set of all statement instances that are executed by a loop. Definition 2. An ultimate dependence source is the source that is not the destination of another dependence. Defnition 3. Given a dependence graph, D, defined by a set of dependence relations, S, a slice is a weakly connected component of graph D, i.e., a maximal subgraph of D such that for each pair of vertices in the subgraph there exists a directed or undirected path. Definition 4. Given a relation R, set of common dependence sources, CDS, is defined as follows
198
W. Bielecki and M. Palkowski
CDS := {[e] : Exists(e ,e , s.t., e = R−1 (e ) = R−1 (e ) && e ,e ∈ range(R) && e =e ) }. Definition 5. Given a relation R, set of common dependence destinations, CDD, is defined as follows CDD := {[e] : Exists(e ,e s.t., e = R(e ) = R(e ) && e ,e ∈ domain(R) && e =e )}. We distinguish the following three kinds of the dependence graph topology i) chain when CDS = φ and CDD = φ, ii) tree when CDS = φ and CDD = φ, iii) multiple incoming edges graph - MIE graph when CDD = φ. A schedule is a mapping that assigns a time of execution to each statement instance of the loop in such a way that all dependences are preserved, that is, mapping s:I→Z such that for any dependent statement instances si1 and si2 ∈ I, s(si1)<s(si2) if si1 ≺ si2 [9]. Definition 7 [9]. A free schedule assigns statement instances as soon as their operands are available, that is, mapping σ:I→Z such that 0 if thereis no p ∈ I s.t. p ≺ p σ(p) = 1 + max(σ(p )), p ∈ I, p ≺ p . The free schedule is the ”fastest” schedule possible. Its total execution time is Tf ree = 1 + max(σf ree (p), p ∈ I) .
3
Motivating Example
To explain troubles we deal with non-linear slices, let us consider the following example for i = 1 to n for j = 1 to n a(i,j) = a(2*i,j) : a(i,j-1) endfor endfor Dependences in this loop are represented by the following relations R1 :=[i,j] → [2i,j]: 1i && 2i n && 1 j < n , R2 :=[i,j] → [i,j+1] : 1 i n && 1 j < n . From Figure 1, illustrating dependences in the loop for n=6, we can see that there exist 3 synchronization-free slices represented by the following sets of statement instances sk := {[i, j] : 1 ≤ i ≤ n && 1 ≤ j ≤ n && Exists α : i = 2α ∗ zk }, where k=1,2,3 and z1 =1, z2 =3, z3 =5. The constraints of the sets above are non-linear. We are unaware of any technique permitting for generating code for
Extracting Affine and Non-linear Synchronization-Free Slices in Loops
199
Fig. 1. Iteration space and dependences for the motivating example, n=6
such sets. Our previous algorithms [7,8] allow one to parallelize only such loops whose dependence graphs are chains or trees. The dependence graph for the motivating example is a multiple incoming edge graph. Below we propose an algorithm permitting for generating code enumerating statement instances of both affine and non-linear slices.
4
Algorithm
The idea of the algorithm presented below is the following. First, using a set of dependence relations, we check whether the topology of a dependence graph is a MIE graph. If so, relations are modified so that they do not represent any common dependences destination. Then we check whether such modifications are permitted. If so, using modified relations, we generate code that preserves all dependences represented by the original relations. Algorithm. Extracting both affine and non-linear synchronization-free slices. Input: Set of dependence relations Ri , 1iq, S; set of slice sources, Sour, calculated by means of the technique presented in [7]. Output: code scanning synchronization-free slices when possible. 1 Using set S, calculate the union of relations R:= Ri . 1≤i≤q
2 Using R, calculate set of common dependence destinations, CDD. If CDD = φ then S =S, go to step 5 /* satisfying this condition means that the dependence graph described by R is not a MIE graph */ 3 For each relation Ri , 1iq from set S do 3.1 Form relation R i := Ri ={[e]→[e ] : constraints} and insert in its constraints the following constraint: && e ∈ range Ri && e ∈ / CDD, /* this constraint causes that R i does not describe any common dependence destination*/ 3.2 If the following condition Ri - R i = φ AND R i - Ri = φ, /* satisfying this condition means that Ri and R i are the same */ is not satisfied, then do
200
W. Bielecki and M. Palkowski
3.2.1 Create a new set of relations S , where Ri is replaced for R i 3.2.2 Calculate R as the union of all relations contained in set S 3.2.3 If R+(Sour) -R +(Sour)= φ and R +(Sour)- R+(Sour)= φ, /*satisfying this condition means that the transitive closure of R and that of R are the same */ where R+ is the transitive closure of R, then replace Ri for R i in set S, 4 Using the modified set S , calculate the union of relations R:= Ri ; 1≤i≤q
using R, calculate set of common dependence destinations, CDD, and check whether CDD = φ; if not, the end, the algorithm fails to extract slices. 5 Generate parallel code for set of dependence relations S using the algorithm presented in [8]. Proof. To prove the correctness of the algorithm above, we should demonstrate that the resulting code produced by the algorithm preserves all the dependences in the original loop. With this purpose, let us remind that code produced by the technique presented in paper [8] assigns statement instances to be executed according to free-scheduling. To show that the free scheduling is the same for both a set of modified relations and a set of original relations, we observe that time k when a given statement instance, I, is executed according to free-scheduling, is defined by the number of vertices on the maximal path from the source of a slice, S, to I (a path in a graph is a sequence of vertices such that from each of its vertices there is an edge to the next vertex in the sequence). Because satisfying the condition R + (Sour) − R + (Sour) = φ and R + (Sour) − R + (Sour) = φ guarantees that for any vertex I, the maximal path from S to I is the same for the both the dependence graph formed by a set of original dependence relations and that formed by a set of modified relations produced by the algorithm, we can conclude that the free scheduling is the same for both the original dependence graph and the dependence graph produced by the algorithm, i.e., the resulting code, produced by the algorithm, preserves all the dependences in the original loop. From the proof above, it is clear when the algorithm fails to produce parallel code. This takes place when the algorithm is not able to produce a non-MIE dependence graph or the free scheduling for a produced non-MIE graph is different from that produced for the original dependence graph. In the motivating example, each ultimate dependence source is the source of a slice (the dependence graph does not contain any pair of ultimate dependence sources that is connected by the undirected path in the dependence graph). So, sources of slices can be calculated as Domain(R)-Range(R). In papers [7,8], we proposed algorithms to generate while loop based code scanning non-linear slices of the chain and /or tree topology. Each iteration of a while loop executes all the statement instances whose operands have already been calculated and prepares a set of statement instances to be executed at the next iteration. Statement
Extracting Affine and Non-linear Synchronization-Free Slices in Loops
201
instances are executed as soon as their operands are available, preserving in such a way all dependences exposed for an original loop. Such a schedule (assigning a time of execution to each operation of the loop) is known to be freeschedule [9]. Below we demonstrate applying the algorithm to the motivating example. Set S of dependence relations is the following R1 :={[i,j] → [2i,j]: 1i && 2i n && 1 j < n }, R2 :={[i,j] → [i,j+1] : 1 i n && 1 j < n }. A set of sources, Sour, is calculated by the algorithm presented in [7] Sour := {[i,1]: Exists ( alpha : 2alpha = 1+i && 1 i n, 2n-3}. 1 R :=R1 ∪R2 ={[i,1]→[2i,1]: 1i && 2in && 1j < n }∪{[i,j]→[i,j+1]:1in &&1j
202
W. Bielecki and M. Palkowski
R+(Sour)-R +(Sour) = {[i,j] : 2 < j n && 1 i n && Exists(alpha : 2alpha = i)} = φ /* transitive closure of R and that of R are not the same */ S = {R 1, R2 } 4 CDD = φ. The dependence graph is represented by the tree (Fig. 2). 5 Using the technique presented in [8], we generate the following parallel code. parfor(t=1;t<=min(n,2*n-3);t+=2) { I=[t,1]; Add I to S’; /* I is the source of a tree */ while(S’!= null) { S_tmp=null; (par)foreach(vector I=[i,j] in S’) { s1(I); /* the original loop statement */ /* to be executed at iteration I */ if(j==1 && 1<=i && 2*i<=n){ ip = 2*i; jp = 1; add J=[ip,jp] to S_tmp; } if(1<=i && i<=n && 1<=j&&j
Fig. 2. Iteration space and dependences for the motivating example after applying the algorithm
5
Experiments
We have studied the parallel code above to examine what is its speedup and efficiency. We have carried experiments on a multiprocessor machine (2x Intel
Extracting Affine and Non-linear Synchronization-Free Slices in Loops
203
Quad Core, 1.6 GHz, 2 GB RAM, Ubuntu Linux) using the OpenMP API [15]. The time of the execution of the original loop and that of the code produced by our algorithm on one processor as well as speed-up, S, and efficiency, E, for 2,4, and 8 processors are presented in Table 1. The following conclusions can be made analyzing the data in Table 1. There is no considerable difference between the time of the execution of the original sequential for loop and that of the while loop produced by the presented algorithm. The produced parallel code is characterized by positive speed-up( S>1). The efficiency of the parallel code increases as the volume of computations executed by the produced while loop increases. Table 1. Results for the motivating example N 250 500 1000 1500 2000
1 CPU for [s] while [s] 0,014 0,016 0,056 0,064 0,224 0,257 0,504 0,573 0,922 1,006
2 CPU S E 1,273 0,636 1,436 0,718 1,659 0,830 1,732 0,866 1,787 0,893
4 CPU S E 1,167 0,292 2,074 0,519 3,027 0,757 3,170 0,792 3,453 0,863
8 CPU S E 1,077 0,135 2,667 0,333 4,148 0,519 5,538 0,692 6,188 0,773
We have also studied UTDSP benchmarks [14] to recognize what is the dependence graph topology and what is the effectiveness of the presented algorithm. The UTDSP Benchmark Suite was created to evaluate the quality of code generated by a high-level language (such as C) compiler targeting a programmable digital signal processor (DSP). We have considered only such loops for which Petit [12] was able to carry out a dependence analysis. Table 2. Results for UTDSP loops All loops Topology of dependence graph MIE graphs tree chain Before using the algorithm 14 78% 1 5% 3 17% 18 After using the algorithm 5 27% 2 11% 11 62%
Table 2 presents the number and percentage of loops with different dependence graph topologies. MIE graphs occur in 14 of 18 graphs. After using the presented approach, the number of loops with that topology was reduced to 5. For 14 MIE graphs, the approach is able to transform 9 of them to the tree or chain topology. We examined parallel codes produced on the basis of the presented approach for UTDSP loops. Experimental results were similar for the loops under examination. Due to the lack of space, we attach experimental results only for the edge detect application from the UTDSP suite. Dependences in this loop are represented by a MIE graph. After applying the proposed algorithm, dependences
204
W. Bielecki and M. Palkowski Table 3. Experimental results for the UTDSP edge detect application N 1000 1500 2000 2500 3000
1 CPU for[s] while [s] 0,047 0,061 0,107 0,137 0,188 0,246 0,294 0,382 0,428 0,550
2 CPU S E 1,309 0,654 1,446 0,723 1,440 0,720 1,431 0,716 1,507 0,754
4 CPU S E 1,521 0,380 2,099 0,525 2,806 0,701 2,786 0,696 2,828 0,707
8 CPU S E 1,842 0,230 2,301 0,288 3,388 0,423 3,406 0,426 4,389 0,549
are represented by chains. The time, speed-up and efficiency of a parallel program (generated by approach [7]) are presented in Table 3. Analysing the data in this table, we can conclude that the presented approach can be applied for producing efficient parallel code for real-life loops.
6
Related Work
The affine transformation framework, considered in papers [4,5,6] unifies a large number of previously proposed loop transformations. However, it fails to extract synchronization-free parallelism when slices are described by non-linear constraints. Our previous papers [7,8] are devoted to the extraction of non-linear synchronization-free slices being represented by graphs of the chain or tree topology only. The techniques presented in those papers fail to extract slices being represented by multiple incoming edges(MIE) graphs. The main contribution of this paper is the demonstration how to extract synchronization-free slices described by both affine and non-linear constraints and being represented by MIE graphs. The algorithm presented in this paper was implemented by means of the Omega library. Carried out experiments by means of a tool developed [16] demonstrate the correctness and satisfactory efficiency of parallel code produced on the basis of the algorithm presented in this paper.
7
Conclusion and Future Work
We presented a way how to extract synchronization-free slices represented by both affine and non-linear constraints. Code generated by the presented algorithm assigns loop statement instances to be executed according to the free scheduling policy. The algorithm successes when eliminating common dependence destinations does not violate the free scheduling valid for an original dependence graph presenting exact dependences in a program loop. The algorithm can be applied to produce parallel coarse-grained programs for parallel shared memory computers. Our future research direction is to derive techniques for extracting both synchronization-free slices and slices requiring synchronization and being represented by graphs of an arbitrary topology.
Extracting Affine and Non-linear Synchronization-Free Slices in Loops
205
References 1. Akhter, S., Roberts, J.: Multi-core Programming, Intel Books by Engineers, for Engineers (2006) 2. Allen, R., Kennedy, K.: Optimizing Compilers for Modern Architectures, p. 790. Morgan Kaufmann, San Francisco (2001) 3. Banerjee, U.: Unimodular transformations of double loops. In: Proceedings of the Third Workshop on Languages and Compilers for Parallel Computing, pp. 192–219 (1990) 4. Feautrier, P.: Toward automatic distribution. Journal of Parallel Processing Letters 4, 233–244 (1994) 5. Lim, W., Cheong, G.I., Lam, M.S.: An affine partitioning algorithm to maximize parallelism and minimize communication. In: Proceedings of the 13th ACM SIGARCH International Conference on Supercomputing (1999) 6. Darte, A., Robert, Y., Vivien, F.: Scheduling and Automatic Parallelization. Birkhuser, Boston (2000) 7. Bielecki, W., Beletska, A., Palkowski, M., San Pietro, P.: Extracting synchronization-free chains of dependent iterations in non-uniform loops. In: 14th International Multi-Conference on Advanced Computer Systems, Polish Journal of Environmental Studies, Miedzyzdroje, vol. 16(5B), pp. 165–171 (2007) 8. Bielecki, W., Beletska, A., Palkowski, M., San Pietro, P.: Extracting synchronization-free trees composed of non-uniform loop operations. In: Bourgeois, A.G., Zheng, S.Q. (eds.) ICA3PP 2008. LNCS, vol. 5022, pp. 185–195. Springer, Heidelberg (2008) 9. Boulet, P., Darte, A., Silber, G., Vivien, F.: Loop paralellelization algorithms: from parallelism extraction to code generation, Technical Report 97-17, LIP, ENS-Lyon, France, p. 30 (June 1997) 10. The Omega project, http://www.cs.umd.edu/projects/omega 11. Pugh, W., Wonnacott, D.: An exact method for analysis of value-based array data dependences. In: The Sixth Annual Workshop on Programming Languages and Compilers for Parallel Computing. Springer, Heidelberg (1993) 12. Kelly, W., Maslov, V., Pugh, W., Rosser, E., Shpeisman, T., Wonnacott, D.: The omega library interface guide. Technical report, College Park, MD, USA (1995) 13. Pugh, W., Rosser, E.: Iteration space slicing and its application to communication optimization. In: Proceedings of the International Conference on Supercomputing, pp. 221–228 (1997) 14. UTDSP benchmark suite, http://www.eecg.toronto.edu/~ corinna/DSP/infrastructure/UTDSP.html 15. OpenMP API, http://www.openmp.org 16. Implementation of the presented approach source code, and documentation, http://detox.wi.ps.pl/SFS_Project
A Flexible Checkpoint/Restart Model in Distributed Systems Mohamed-Slim Bouguerra1,2, Thierry Gautier2 , Denis Trystram1 , and Jean-Marc Vincent1 1
Grenoble University, ZIRST 51, avenue Jean Kuntzmann 38330 MONTBONNOT SAINT MARTIN, France 2 INRIA Rhone-Alpes, 655 avenue de l’Europe Montbonnot-Saint-Martin 38334 SAINT ISMIER, France {mohamed-slim.bouguerra,thierry.gautier,denis.trystram, jean-marc.vincent}@imag.fr
Abstract. Large scale applications running on new computing platforms with thousands of processors have to face with reliability problems. The failure of a single processor will cause the entire execution to fail. Most existing approaches to guarantee reliable executions are based on fault tolerance mechanisms. Coordinated checkpointing is one of the most popular technique to deal with failures in such platforms. This work presents a new model of coordinated Checkpoint/Restart mechanism for several types of computing platforms. The model is parametrized by the process failure distribution, the cost to save a global consistent state of processes and the number of computational resources. Through mathematical analysis of reliability, we apply this new model to compute the optimal interval between checkpoint times in order to minimize the average completion time. Model independency from the type of the failure law makes it completely flexible. We show that such a model may be used to reduce the checkpoint rate up to 20% in same cases and up to factor 4 the total overhead in same cases. Finally, we report some experiments based on simulations for random failure distributions corresponding to the two most popular laws, namely, the Poisson’s process and Weibull’s law. Keywords: Fault tolerance - Reliability modeling - Checkpointing.
1
Introduction
Today the most powerful computing systems involve more and more processors. Reliability is a crucial issue to address while running applications on such large systems because the failure rate grows with the system size. From the IBM source, the BlueGene/L system with 65,536 nodes is expected to have a mean time between failure less than 20 hours [1][2][3]. The users expect several processors to fail during the execution of their applications. Hence, it is necessary to develop strategies for providing a reliable completion of large applications. Fault-tolerant systems have been extensively studied on various computing or R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 206–215, 2010. c Springer-Verlag Berlin Heidelberg 2010
A Flexible Checkpoint/Restart Model in Distributed Systems
207
real-time systems and any approaches have been proposed for dealing with failures. We focus in this work on the coordinated Checkpoint/Restart approach which is one of the most popular mechanism used in practice [4][5][6][7][8]. In checkpointing-based approaches, the state of computations is saved periodically on a reliable storage [9]. Informally, checkpointing is a technique that enables to reduce the completion time of the application by saving intermediate states in a stable storage, and then, to restore from the last stored state when a failure occurs. In the case of distributed computations, checkpointing methods differ from each other in the way the processes are coordinated in order to capture the global state in a consistent manner or not. Coordinated checkpointing requires that all processes coordinate the construction of a consistent global state before they write the individual checkpoints in the stable storage. After one or many failures, the restart mechanism sets up each processor from the last checkpoint and then, it schedules again the tasks of the crashed processors on new ones. In this paper, we focus on the minimization of both the completion time of an application and the Checkpoint/Restart overhead by determining the optimal interval between two consecutive checkpoints. The main contribution of this work is to derive a new probabilistic model of the execution time of parallel applications with stochastic processes. This explicit analytical cost model enables to address and solve several interesting problems. First, like most of other models, it can be used to determine the optimal interval length between two successive checkpoints that minimizes the checkpoint overhead. Moreover, it is possible to take into account a variable checkpoint cost that depends on residual workload. This issue is not detailed here however, the details can be found in the technical report [10]. Finally, the proposed model can be adapted to several types of computations since it is independent from the failure law type, this make it very flexible. We show how to use it for the two most popular laws (Poisson’s process and Weibull distribution). The rest of this paper is organized as follows: Section 2 briefly recalls and discusses the other existing models for coordinated checkpointing on parallel platforms. Section 3 describes the proposed analytical model that expresses the expected completion time. In Section 4, we detail how to apply this model to two case-studies for the distribution of failure occurrences with Poisson’s process and Weibull’s law. Before concluding, we report in Section 5 some experiments based on simulations for assessing the model in several scenarii.
2
Review of Related Works
Let us first recall the principle of the coordinated checkpoint/restart protocol proposed by Chandy et al. [9]. It served as a basis of many implementations of fault tolerant systems for high performance computing. Coordinated checkpointing requires that all processes periodically coordinate the construction of a consistent global state before writing the state of individual checkpoints in a stable storage. Based on simulation results, Elnozahy et al. [5] showed that this approach is the most effective fault tolerance mechanism in large-scale parallel
208
M.-S. Bouguerra et al.
platforms. The aims of fault tolerance models is to find strategies that minimize the completion time of the execution. Young [8] proposed a checkpoint/restart model where the failures follow an exponential law. Moreover, checkpointing is assumed to be fault-free and to have a constant execution time. The basic periodic version of this model has been recently extended by Oliner et al. [7]. Daly proposed also in [4] an extension of Young’s model where failures can occur during checkpointing and he derived a higher order of approximation. Both Young’s and Daly’s models are able to compute an optimal checkpoint interval that minimizes the completion time considering constant checkpoint times. Other works from Geist et al. [11], Plank et al. [12] studied stochastic models and determined an optimal checkpoint date that minimizes a cost function which corresponds to maximize the availability of the system. Yudan et al. [6] proposed a stochastic model and an optimal checkpoint interval which does not depend on the specific failure law, such as the Poisson’s process used in [11] [12]. Moreover, in this model, it is possible to compute the optimal checkpoint interval that minimizes the expected wasting time in the protocol itself (checkpoint overhead, restart time and rollback time). Another variant of this model is proposed in [13], that modelizes the incremental checkpoint under Poisson’s process. We present an analytical model that expresses the expected completion time with respect to the distribution law of failures taking into account the checkpoint overheads. The most interesting differences in our model, is that it does not depend on a particular failure law and it reduces both of the checkpoint rate and total wast time. Finally it takes into account the variable size of checkpoint as an input of the problem and this issue is addressed in [10].
3
Probabilistic Execution Model
For modeling the execution of a parallel application in presence of failures and with Checkpoint/Restart mechanism, the scheme of modeling is organized as follows. Firstly, a basic model that describe the parallel application in presence of failures is presented. This first model allows to compute the expected completion time of the execution with a formal method. Finally, using this abstraction without checkpointing, we derive a global model that contains the Checkpoint/Restart mechanism. It allows, to establish a global formula that expresses the expected completion time of the execution. 3.1
Parallel Application Model
A parallel application is modeled by a residual computation work function denoted by ωt which is the amount of work remaining at the time index t. If we draw this variation, the curve would be divide into two parts. The first part is characterized by a linear speed-up with a slope which depends on the number of computing processors m and overhead factor α (due to the parallelism). In this phase, all the processors are busy. The second part presents a less stiff curve. This means that there is not enough work for all the processors and some idle
A Flexible Checkpoint/Restart Model in Distributed Systems ωt phase of startup ω0 R R
Wt 1 R ω0
phase of recovery R
Z3
R1
209
R1
I1
Phase of Checkpoint C (I1 )
*
$R^{k−1}$
Rk−1
Rk−1
Ik−1 Rk
Rk
C (I1 + I2 + · · · + Ik−1 )
Failure Ik
X1
X2
X11
X3
X21
X31
X1k−1
Z1
Z2
X2k−1
X3k−1
X1k
X2k t
1 Tend
Tend t
k−1 Tend
k Tend
Tend
Fig. 1. Execution Model without check- Fig. 2. Model of execution with the Checkpoint mechanism point/Restart mechanism
time appear. These two parts will be determined by the amount of parallelism in the target application. In this work we consider only the first part with a linear speed-up until the completion of the application. This is not really a limitation for large scale applications. Under this hypothesis we have: ωt = ω0 − αmt. 3.2
Execution Model without Checkpoint Mechanism
To establish the first fundamental theorem, we start by studying the execution without checkpoint mechanism, where the execution is restarted from the beginning after a failure (see Figure 1). This figure models the following renewal process. At the first step, there is a start-up in where the time is elapsed but the residual workload does not decrease. Then, the computation is started and the residual workload is decreased with a speed-up depending on the number of computing nodes. Finally, when a failure comes into one of these preview phases, all the computed work is lost. Thus, the application is restarted from the beginning after a duration R which denotes the restart cost. For modeling the failures process, we consider mainly failures that affect the hardware or software system. Also, we assume the reliability of the failures detection tool and the time of failures detection is negligible. Then, let us suppose that failures do not propagate hence, the time between failures on different processors in the system are independent and identically distributed. From the above assumptions, we deduce that {Xi } process (which denote the ith inter-arrival of failures) is a renewal process and {Xi } are positive independent and identically distributed random variables with distribution function F (t) and probability density function f (t). We recall that in this case the recovery process is considered as a deterministic process with duration R. From this abstraction, we deduce an analytical model that expresses the expected completion time Tend . Let us consider a number of failures N during the execution such as: {N = ω0 + R}. Consequently N is a stopping time process associated n} = {Xn αm to the process {Xi }. Using Wald’s equation [14] we can establish the following theorem.
210
M.-S. Bouguerra et al.
Theorem 1. Let Tend be the completion time of the execution and p the probaω0 bility of failure during the execution with p = F ( αm + R) then: 1 ω0 1 E(Tend ) = E(X1 )+ +R−E(XN ) = ω0 1−p αm 1 − F ( αm + R)
ω0 αm +R
1−F (x)dx. 0
proof in [10]. 3.3
Execution Model with Checkpoint Mechanism
The next step consists in inserting the Checkpoint/Restart mechanism to the basic model. Let us divide the initial amount of work into k intervals denoted j denote the completion time of amount of work I1 , I2 , · · · , Ik such as each Tend Ij . Therefore, the global execution model is drawn in Figure 2. At the first step, there is a start-up or a restart phase in where the time is elapsed but the residual workload does not decrease. Then, the computation is started and the residual workload is decreased with a speed-up depending on the number of computing nodes. After an interval of work denoted by Ij a checkpoint is triggered. Finally, when a failure comes into one of these preview phases, all the computed work after the last checkpoint will be lost. Thus, the application is restarted from the last checkpoint. In this model we consider that the checkpoint process is a deterministic process and its cost depends only on the amount of the work already done. We briefly recall that based on the design of the coordinated checkpoint mechanism, the data to save are the current state of each process and the computing data results associated to this one. More precisely, the amount of this data will depend on the work already done. Therefore, let sj be the amount of work already done, where sj = ji=1 Ii , the checkpoint cost function is denoted by C(sj ). Also, the restart process is considered as a deterministic process and its cost depends on the amount of data saved in the last checkpoint. It is denoted j−1 by R(rj ) where rj = i=1 Ii is the work already done before the last checkpoint. Finally, we consider that the phases of checkpoint and recovery may also suffer from failures. From the above assumptions, we deduce that {Xij } process (which denotes the ith inter-arrival of failures in the j th interval of checkpoint) is a renewal process, thus {Xij } are positive independent and identically distributed random variables with distribution function F (t) and probability density function f (t). Then, the main idea is to use the basic model (without checkpoint mechanism) for each Ij , as Ij is the amount of initial workload that should be done between the checkpoint number j − 1 and j. Using this abstraction and under the these hypothesis we establish the following theorem: Theorem 2. Let k be the number of checkpoints and ηj be the amount of work k ω0 between each checkpoint where ηj = Ij + C(sj ) + R(rj ). Such as j=1 Ij = αm then: E(Tend ) = E(X1 )
k j=1
1 j + ηj − E(XN ), 1 − pj j=1 k
where pj = F (ηj ).
A Flexible Checkpoint/Restart Model in Distributed Systems
211
Proof in [10]. Hence, Theorem 2 expresses the expected value of the completion time. It is worth noting it does not depend on the type of failures law which confirms the generality and flexibility of the proposed model. Therefore, in order to find the optimal amount of work between each checkpoint (Iˆ1 , Iˆ2 , · · · , Iˆk ), we have to minimize the equation of Theorem 2.
4
Case Studies
In this section, we propose two case studies. In the first one, we express the optimal checkpoint interval when the failures process is modeled by a Poisson’s process. Then, in the second one we also express the suboptimal solution when the failures followed a Weibull’s law. 4.1
Poisson’s Process Failures
One of the most common method to model the failures on computer devices, is the Poisson’s process with a constant rate denoted by λ. In this case the expected completion time is given by this expression. 1 ληj [e − 1] where ηj = Ij + C(sj ) + R(rj ). λ j=1 k
E(Tend ) =
(1)
Then, considering the cost of the checkpoint barrier as a constant cost such that C(s) = C and R(r) = R, the following lemma is established. Lemma 1. When the cost of Checkpoint and restart is constant, then the optiω0 mal interval between each checkpoint is αmk where k is the checkpoint number. Proof in [10]. Then after finding the optimal interval between each checkpoint using Lemma 1, Equation (1) becomes: k λ( ω0 +C+R) (e αmk − 1). (2) λ Finally, to minimize this equation we use the Lambert’s function denoted by W. Then, the optimal interval between checkpoint is: ω0 = 1 + W(−e−1−λ (C+R) ) (3) αmk Notice that Lambert’s function could be computed numerically using Taylor’s series, (more information about the proof can be found in [10]). E(Tend ) =
4.2
Using Weibull’s Law
In this second case study, a Weibull’s law with shape parameter β and scale parameter λ is used to model the failure distribution occurrences. Then, the expected completion time is given by: ηj k β (ληj )β E(Tend ) = e e−(λx) dx. (4) j=1
0
212
M.-S. Bouguerra et al.
Considering the case where a constant cost is spend to checkpoint the global application. Consequently, the following lemma can be established: Lemma 2. If the failure distribution is a Weibull’s law and the checkpoint/ ω0 restart cost is constant, the suboptimal interval between each checkpoint is αmk , where k is the number of checkpoints. Proof details in [10]. Then, using Lemma 2 the new expression of the expected completion time becomes: ω
ke(λ( αmk +C+R))
β
ω αmk +C+R
β
e−(λx) dx
(5)
0
ˆ the Brent’s numerical method Hence, to minimize Equation (5) in order to find k, from the GNU scientific library is used to find the minimum. More informations about the numerical method source code are in [10].
5 5.1
Simulations Simulations Scenario
In this simulation we propose 3 different scenario to validate and to compare the proposed model. In the first and the second scenario a Poisson’s process is used to generate random failures time with respect to different failure rates. Then, in the last scenario we use a Weibull’s law to generate the failures time using the same value for the parametres λ and β described in [6]. These Weibull’s parameters are the result of a statical analysis in [6] of failures log-files from the LLNL ASC White Computer system during the one year period June, 2003 to May, 2004. The best fitted distribution to the data is the Weibull distribution with the shape parameter of β = 0.5094 and the scale parameter of 1 . λ = 20.584 Then, to compare the model, we consider the Daly’s model [4] in the first and in the second scenario then, in the 3th scenario we consider the model proposed in [6] to compare our model when the failures follow a Weibull distribution. The main objective of these simulations is to study the behaviors of the model with respect to the variation of one form the different parameters i,e, (Failure rate, Checkpoint cost, Initial amount of work). Therefore, as scenario, we suppose that one parameter varies in a given interval and the others are constant. The different scenario configurations are in Table 1. For instance, in the first scenario the failure rate is increased in [ 12 , 96] by step of 5 while the checkpoint cost and the initial amount of work are constant. Then, for each setup we perform 104 simulations of execution with random failures. We notice that the same simulation is performed for both constant checkpoint cost 15 and 10 minutes only for scenario 3. Finally, It should be noted that the confidence interval reported in all simulations are at 95%.
A Flexible Checkpoint/Restart Model in Distributed Systems
213
Table 1. Scenario configurations Scenario Failure Law Failure Rate (per day) Checkpoint Cost (mins) Amount Of Work 1 Poisson’s Process λ ∈ [1/2, 96] C = 10 ω0 /αm = 10 days 2 Poisson’s Process λ = 1/2 C ∈ [1/2, 96] ω0 /αm = 7 days 3 Weibull λ = 1/20.584, β = 0.5094 C = 10, C = 15 ω0 /αm ∈ [100, 2500] hrs
5.2
Results and Discussions
Figure 3 reports the variation of the average completion time versus the variation of the failures rate, with respect to the configuration of scenario 1. As it can be seen in this figure both models perform the same average completion time and the same confident interval. Consequently, our model has the same performance in term of completion time as Daly’s model when λ highly increases. The second result with configuration 1 is presented in Figure 4, that shows the variation of the checkpoint rate versus the failure rate. This figure reveals that our model outperforms Daly’s model in term of checkpoint rate when λ becomes very large. Indeed, the proposed model has the same average completion time as Daly’s model even though it reduces the checkpoint rate by to 20%. It should be noted that decreasing in the checkpoint rate leads to lower I/O request and communication on the system in terms of network communication and stable storage writing as well. Secondly, to explain more why both models have the same completion time and FCM outperforms Daly’s model in term of checkpoint rate. The average overhead due to failure and the average overhead due to checkpoint are depicted versus the variation of the checkpoint cost in Figure 5, for FCM in contrast to Daly’s model. Hence, as it can be seen in this figures that the proposed model performs less checkpoint (discontinuous curve with square marker) than Daly’s model (discontinuous curve without marker) but FCM loses more work due to failure then Daly’s model. That means both models have the same wast time (Checkpoint overhead in addition to re-exectude work due to failures), consequently they have the same average completion time. These outcomes show that the proposed model (FCM) outperforms the Daly’s model in terms of system I/O overhead while the completion time performance dose not change which is very interesting features. In fact, this is due to the assumption on the checkpoint cost that should be low in Daly’s model. As a conclusion of the performances analysis, let us emphasize that both models have the same average completion time and confident interval which determines the stability of the model. Indeed, FCM reduces very well the checkpoint rate by to 20% and it is independent from the type of the failures law which make it flexible while Daly’s model considers only the Poisson’s process. Finally in the 3th scenario, we intend to compare the average wast time versus the variation of the initial amount of work with the model proposed in [6]. The result of this simulation is depicted in Figure 6, in which the same scenario is repeated for two values of checkpoint cost. This figure reveals for the both values of checkpoint cost, that FCM reduces by a factor of 3 the total wast time and reduces as well the confident intervals especially when the initial amount of work is relatively large. This is due to the point-fixe method
214
M.-S. Bouguerra et al.
30
1600
FCM J.Daly
FCM J.Daly
25 1200
Checkpoint number
Average completion time (days)
1400
20
15
10
1000
800
600
400 5 200
0 0
10
20
30
40
50
60
70
80
90
0 0
100
10
20
30
Failures rate per day
40
50
60
70
80
90
100
Failures rate per day
Fig. 3. Variation of the average completion times (Scenario 1)
Fig. 4. Variation of the optimal checkpoint number (Scenario 1)
1.6
300
MCM lost work J.Daly lost work MCM checkpoint overhead J.Daly checkpoint overhead
1.4
250
FCM (15 mins) Yudan et al (15 mins) Yudan et al (10 mins) FCM (10 mins)
Average wast time per hrs
Mean of lost time (Day)
1.2
1
0.8
0.6
200
150
100
0.4 50
0.2
0
0
10
20
30
40
50
60
70
80
90
100
Cost of checkpoint (minute)
Fig. 5. Average Checkpoint / Lost work overhead (Scenario 2)
0 0
500
1000
1500
2000
2500
Initial amount of work per hrs
Fig. 6. Variation of the average wast time (Scenario 3)
used by authors to predict the time between the last checkpoint and the next failures. Consequently, FCM reduces the average completion time by reducing the total wast time and it outperforms the model in [6] in terms of stability by reducing the confident intervals.
6
Conclusions
We presented in this paper a new flexible mathematical model for checkpointing applications in large-scale computing platforms. The main result was to establish a general analytical model which expresses the expected completion time of the application in distributed systems. We showed that it is possible to use this model with various probability laws that modelize the failures process, e.g, the Poisson’s process or the Weibull’s law. Moreover, this model is derived, for determining the optimal interval between checkpoints considering the checkpoint cost and
A Flexible Checkpoint/Restart Model in Distributed Systems
215
the wast time. A performance analysis with others existing models revealed that our model reduces the checkpoint rate, the strain on I/O bandwidth and the total wast time in terms of expectation and variance. We are currently working on implementing this mechanism in a real system. For this purpose, it would be interest to take into account a variable number of processors (instead of unbounded number of processors) since, it is not reasonable to assume that enough number of spare processors are always available.
References 1. Adiga, N., et al.: An Overview of the BlueGene/L Supercomputer. In: ACM/IEEE 2002 Conference on Supercomputing, p. 60 (2002) 2. Schroeder, B., Gibson, G.A.: A large-scale study of failures in high-performance computing systems. In: DSN 2006: Proceedings of the International Conference on Dependable Systems and Networks, Washington, DC, USA, pp. 249–258 (2006) 3. Hacker, T.J., Romero, F., Carothers, C.D.: An analysis of clustered failures on large supercomputing systems. J. Parallel Distrib. Comput. 69(7), 652–665 (2009) 4. Daly, J.T.: A higher order estimate of the optimum checkpoint interval for restart dumps. Future Generation Computer Systems 22(3), 303–312 (2006) 5. Elnozahy, E.N., Plank, J.S.: Checkpointing for peta-scale systems: A look into the future of practical rollback-recovery. IEEE Trans. Dependable Secur. Comput. 1(2), 97–108 (2004) 6. Liu, Y., Nassar, R., Leangsuksun, C., Naksinehaboon, N., Paun, M., Scott, S.: An optimal checkpoint/restart model for a large scale high performance computing system. In: IEEE International Symposium on Parallel and Distributed Processing, pp. 1–9 (2008) 7. Oliner, A.J., Rudolph, L., Sahoo, R.K.: Cooperative checkpointing: a robust approach to large-scale systems reliability. In: Proceedings of The 20th Annual International Conference on Supercomputing, pp. 14–23. ACM, New York (2006) 8. Young, J.W.: A first order approximation to the optimum checkpoint interval. ACM Commun. 17(9), 530–531 (1974) 9. Chandy, K.M., Lamport, L.: Distributed snapshots: determining global states of distributed systems. ACM Trans. Comput. Syst. 3(1), 63–75 (1985) 10. Bouguerra, M.S., Gautier, T., Trystram, D., Vincent, J.M.: A new flexible checkpoint/restart model. Technical report, RR-6751, INRIA (2008) 11. Geist, R., Reynolds, R., Westall, J.: Selection of a checkpoint interval in a criticaltask environment. IEEE Transactions on Reliability 37, 395–400 (1988) 12. Plank, J.S., Thomason, M.G.: The average availability of parallel checkpointing systems and its importance in selecting runtime parameters. In: 29th International Symposium on Fault-Tolerant Computing, pp. 250–259 (1999) 13. Naksinehaboon, N., Liu, Y., Leangsuksun, C., Nassar, R., Paun, M., Scott, S.: Reliability-Aware Approach: An Incremental Checkpoint/Restart Model in HPC Environments. In: IEEE International Symposium on Cluster Computing and the Grid, pp. 783–788 (2008) 14. Tijms, H.C.: A First Course in Stochastic Models. John Wiley, Chichester (2003)
A Formal Approach to Replica Consistency in Directory Service Jerzy Brzeziński, Cezary Sobaniec, and Dariusz Wawrzyniak Poznan University of Technology, Institute of Computing Science Piotrowo 2, 60-965 Poznan, Poland {Jerzy.Brzezinski,Cezary.Sobaniec,Dariusz.Wawrzyniak}@put.edu.pl
Abstract. Replication is commonly applied in the implementation of directory service as a way to improve availability and efficiency. However, it raises the problem of consistency maintenance, especially important in the case of slow propagation of changes. The delay in propagation may lead to different anomalies in directory service usage. This paper presents an analysis of consistency conditions that are satisfied by the replicated directory entries, and specifies requirements resulting from client’s history of execution (session guarantees) that are necessary in order to meet the same conditions in the view on the client side. Keywords: directory service, replication, consistency.
1
Introduction
Directory service in distributed systems extensively exploits replication as a way to improve availability and efficiency. Availability improvement is the natural consequence of data (directory entries) redundancy. Efficiency in the sense of increased throughput may result from load distribution over multiple servers capable of providing the same data to requesting clients. Besides, in the case of proper replica allocation, the distance between a client and the server that handles it can be shorten, thereby reducing the response time. Replication raises the problem of consistency maintenance. However, this is cost-effective in the case of directory service because most of access requests are queries, which do not change the state of replica. As for the consistency itself, the design of coherence protocol is usually aimed at eventual consistency (e.g. [1]) formulated in [2], and generalised in [3], which states that: if the system stopped issuing new modifications, copies of a replicated data would eventually have the same value. Restrictions on update acceptance and the topology of site interconnections for the propagation of changes (e.g. master-slave), however, may result in stronger consistency. To identify the consistency conditions, different strategies of update propagation are analysed.
Acknowledgment. The research presented in this paper was partially supported by the European Union in the scope of the European Regional Development Fund program no. POIG.01.03.01-00-008/08.
R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 216–225, 2010. c Springer-Verlag Berlin Heidelberg 2010
A Formal Approach to Replica Consistency in Directory Service
217
Another aspect of consistency concerns client-server interaction. Whilst servers are responsible for data exchange to keep a consistent state of replicas, a client can switch from one server to another that is not up-to-date yet. Consequently, the view of directory data on the client side may differ from the view on the server side. Some properties in this aspect — session guarantees (also called data-centric models) — have been formulated in [4]. Here, the influence of session guarantees on the consistency met on the client side is analysed.
2
System Model
The system considered here consists of two kind of sites (sequential processes): clients requesting the directory service and servers storing the replicated directory entries. Clients use the service by updating the entries at servers (writing) or querying about them (reading). A client interacts with one server at a time, but can switch from one server to another during the execution. It is assumed that each server keeps a replica of all entries. In order to exchange the information of updates, servers are interconnected by means of a fixed network infrastructure. Eventually, every update must appear in the history of operations performed by each server. To this end, the server that accepts an update request from a client (i.e. handles the interaction) distributes the changes to other servers. For the sake of formal definitions let us assume the following notation: X — the set of directory entries; wi (xj )v — an event subsequent to the update of entry x ∈ X at a server (replica) Sj issued by a client Ci , defining a new value v for this entry (write and update are used interchangeably); ri (xj )v — an event subsequent to the query on the replica of entry x ∈ X kept by a server Sj , returning a value v to a client Ci (read and query are used interchangeably). If some elements of the notation are not important or clear in the context they are used, they can be omitted. However, to emphasise that any replica of a given entry, say x, is addressed, the notation x∗ is used. Every operation issued in the system raises a few events in the client-server interaction (at the protocol level of abstraction): sending request at a client process and subsequent acceptance at a server process, possibly sending reply at a server process and subsequent acceptance at a client process. Both clients and servers are sequential processes, so all the events appear in some total order, moreover if a reply is expected, no request is sent (by a client) or accepted (by a server) in the interval between the request and corresponding reply. Thus, the order of operations follows the order of events concerning requests, which is formalised by the definitions below. Definition 1 (issue order). An operation o1 precedes an operation o2 in issue order, iff they are requested by the same client, say Ci , and the request to execute C o1 is sent before the request to execute o2 (which is denoted o1 i o2). Issue order is the relation between operations on the client side, and is the consequence of the program performed by a client. In other words, the set of operations is defined by the program of the client process.
218
J. Brzeziński, C. Sobaniec, and D. Wawrzyniak
Definition 2 (execution order). An operation o1 precedes an operation o2 in execution order, iff the requests for these operations are accepted by the same Sj
server, say Sj , and o1 is executed by this server before o2 (denoted o1 o2). Execution order is the relation between operations on the server side. The relation involves the operations that are executed by a server as direct response to a client request, i.e. the operations that come from the interaction with clients handled by the server. However, potentially all updates are observed by each server in the system, because they change the entries, thereby directly or indirectly influence the the results of queries. Thus, each process has its own view of the operations in the system, according to the definition below. Definition 3 (view). Let O be a set of operations, and a total order relation in this set. View, (O, ), is the set O ordered by the relation , provided that the following condition is satisfied: ∀x∈X ∀w(x)v,r(x)v∈O w(x)v r(x)v Moreover, a legal view can be distinguished. Generally, a view is legal if each read operation returns the value defined by the immediately preceding update of the same entry ([5]). Every server operates on its own replica of the set of entries, therefore write operations that appear in its view concern this replica. The operations result either from the direct interaction with clients or from exchange of information with other servers. In either case, a write operation appears in the view of a given server exactly once. Thus the definition for the server side is formalised as follows. Definition 4 (s-legal view). View (O, ) on the server side is legal (s-legal for clarity) if it satisfies the following condition:1 ∀
∃
∀
x∈X w(x)v,r(x)v∈O w(x)u∈O
(u = v ∧ w(x)v w(x)u r(x)v)
(1)
View on the client side is more complicated because a client can switch from one server to another. Since a write operation subsequent to one issue is in the set of operations of each server, it can appear many times in this view. Therefore, the view need not be s-legal. Moreover, the next server can have executed less operations than the previous one or the same operations in a different order. To simplify the identification of operations, it is assumed that a client uses a given server only once. Therefore, the view on the client side includes all write operations on each replica. Understanding of legality in this case is more general. Definition 5 (c-legal view). View (O, ) on the client side is legal (c-legal for clarity) if it satisfies the following 2 conditions: ∀
∀
∃
x∈X r(xj )v∈O w(xj )v∈O
w(xj )v r(xj )v ∧
∃
(u = v ∧ w(xj )v w(x)u r(xj )v)
(2)
w(x)u∈O 1
For the sake of definition simplicity it is assumed that each write issued by a client defines a new (unique) value of a given object.
A Formal Approach to Replica Consistency in Directory Service
219
The set of operations influencing the execution of a process includes (potentially) all write operations and local read operations. In order to define view for a given process, let us decompose the set of operations into the subset of reads and subset of writes, i.e. O = OR ∪ OW . Thereby, let us assume the following R W notation: OCi = OC ∪ OC — the set of operations issued by a client process i i R W Ci ; OSj = OSj ∪ OSj — the set of operations executed by a server Sj , in which OSRj — the set of all read operations executed by a server Sj (as a result of client request), OSRj /Ci ⊆ OSRj — the set of all read operations executed by a server Sj as a result of a request by Ci , OSWj — the set of all write operations executed by a server Sj as the result of both direct request from a client and cooperation with other servers, including a write defining an initial value (not issued by any client). It is clear that there is a correspondence between the sets of operations issued by the clients and the sets of operations executed by the servers. Excluding W an initial value writing, there is a one-to-one mapping between OSWj and i OC i R R R R for each Sj , OC and O / for each C , as well as O and O . C i i Sj Sj j i Ci j i Definition 6 (server’s view). View on the server side for a server Sj is an Sj
s-legal view of the set OSj (denoted (OSj , )). Definition 7 (client’s view). View on the client side for a client Ci is a c-legal Ci
view of the set OCi (denoted (OCi , )).
3
Consistency Models
Consistency models are defined by means of a number of consistency conditions to be satisfied by views of operations. In fact, the definitions are formulated in terms of equivalence between views, i.e. the conditions concern an equivalent view rather than the view itself. Each process has its own view, so in some cases the conditions concern views (or their equivalents) of all processes. Let us start from the definitions of equivalence. Definition 8 (equivalence of write operations). Two write operations — . w(x∗ )v, w(y∗ )u — are equivalent (w(x∗ )v = w(y∗ )u) iff x = y and u = v. The above definition says that write operations are equivalent if (only if) they write the same value to the same logical object (no matter which replica) under the previously made assumption about the uniqueness of written value. The equivalence relation partitions the set of write operations into equivalence classes that form a quotient set. On the server side the set of write operations and its quotient set are in fact the same2 , because each write is executed by a server only once.
2
Formally, there is one-to-one mapping — projection — between OSWj and OSWj /=. .
220
J. Brzeziński, C. Sobaniec, and D. Wawrzyniak
Definition 9 (equivalence of views). Two views of a set of operations, say i
j
(Oi , ) and (Oj , ), are equivalent if all the following conditions hold: (i) OiR = OjR (ii) OiW /=. = OjW /=. j i o1 o2 ⇔ o1 o2 (iii) ∀ o1,o2∈OiR
3.1
Data-Centric Models
The definition of data-centric models may concern both clients and servers. In order to generalise the definitions, it is assumed that a number of processes, say P1 , P2 , . . . , Pn , either clients or servers, cooperate by accessing shared entries, as described in the previous section. In the unified approach to consistency of shared memory presented in [6], some consistency properties are distinguished that are mostly based on process order and data order of operations. Process order is the order of issue (by individual process) resulting from program execution, and data order corresponds to the order of operations observed on an individual entry. Because of the distinction between clients and servers the term local order is used instead of process order. The distinction results from the fact that two operations can be executed by a server in different order than issued by a client. However, the same notation is used for local order on the client side and on the lo server side (i.e. ) assuming that the suitable definition is clear in the context. Definition 10 (local order on the client side). Operations o1 and o2 are C
in local order on the client side iff they are in issue order, i.e.: ∃Ci o1 i o2. Definition 11 (local order on the server side). Operations o1 and o2 are Sj
in local order on the server side iff they are in execution order, i.e.: ∃Sj o1 o2. Definition 12 (data order). Operations on an entry x, o(x)v and o(x)u, are do
in data order (o(x)v o(x)u) iff at least one of the following conditions holds: lo
(i) o(x)v o(x)u
(3)
(ii) v = u ∧ o(x)v ≡ w(x)v ∧ o(x)u ≡ r(x)v po u = v ∧ o(x)v r(x)u ∧ o(x)u ≡ w(x)u (iii) ∃ r(x)u∈O do do (iv) ∃ o1 o ∧ o o2 o ∈O
(4) (5) (6)
Data-centric models are defined by a number of restrictions (consistency conditions) on the order of operations in the view for each process. General assumption is that the view for a given process (or and preserves its equivalence) is s-legal lo
Pi
its own local order, i.e.: ∀Pi ∀o1,o2∈Oi o1 o2 ⇒ o1 o2 .
A Formal Approach to Replica Consistency in Directory Service
221
Combining local order and data order relations, the following consistency models (formalised below) can be defined [6]: PRAM consistency — preserves local order [7]; cache consistency — preserves data order [8]; processor consistency — preserves both local order and data order [8,9]; slow consistency — preserves the intersection of local order and data order [10]. Pi lo PRAM: ∀ o1 o2 ⇒ o1 o2 (7) ∀ Pi o1,o2∈OPi ∪O W
cache: ∀
∀
Pi do o1 o2 ⇒ o1 o2
(8)
∀
Pi lo do o1 o2 ∨ o1 o2 ⇒ o1 o2
(9)
∀
Pi lo do o1 o2 ∧ o1 o2 ⇒ o1 o2
(10)
Pi o1,o2∈OPi ∪O W
processor: ∀
Pi o1,o2∈OPi ∪O W
slow: ∀
Pi o1,o2∈OPi ∪O W
One of the most important models is sequential consistency. The equivalence to sequentially consistent execution is assumed as a general correctness criterion for concurrent system sharing data [11,12]. In addition to local order it preserves total order of all write operations, i.e.: Pi Pi (11) ∀ w1 w2 ∨ ∀ w2 w1 ∀ w1,w2∈O W
3.2
Pi
Pi
Client-Centric Models
Formal definitions of client-centric models covers the dependencies between issue order of operations on the client side and view of their counterparts on the server side, according to the following formulations: Sj C read your writes (RYW): ∀ ∀ w(x)v i r(yj )u ⇒ w(x)v ri (y)u (12) C i Sj
Sj
C
monotonic writes (MW): ∃ w(x)v i w(y)u ⇒ ∀ wi (x)v wi (y)u Ci
(13)
Sj
C
Sj
(14) writes follow reads (WFR): ∃ r(x∗ )v i w(y)u ⇒ ∀ w(x)v wi (y)u Ci Sj Sj Ci monotonic reads (MR): ∀ ∀ r(x∗ )v r(yj )u ⇒ w(x)v ri (y)u (15) C i Sj
Besides, the definitions of session guarantees, presented in [4], are formulated under the assumption that non-commutative modifications of shared data are performed in the same order on each replica. In our system model this means that write operations of one shared object are observed in the same order on each replica, which implies cache consistency on the server side.
222
4
J. Brzeziński, C. Sobaniec, and D. Wawrzyniak
Consistency of Directory Contents
Two general approaches to consistency maintenance of directory contents are considered: master-slave (single master) and multimaster [13,1]. In the masterslave approach slave replicas are available for reading only, thereby update requests are accepted by the master server and then passed to slave servers. This approach simplifies the implementation of coherence protocol because it facilitates the control over the order of update propagation. Especially, the order in which modifications are performed at the master server can be easily preserved at slave servers. The order can be applied to all updates, which implies PRAM consistency (7), or to updates of individual entries, which implies slow consistency (10). However, master server controls all operations, especially the operations on a given entry, thereby ensures data order (def. 12). This in turn implies processor consistency (9), or cache consistency (8), respectively. Moreover, if there is only one master server for the whole domain (no data decomposition), the local order at this server concerns all modification. Consequently, there is the same order of write operations, which implies sequential consistency, because condition (11) is also preserved. This is not the case if directory data are decomposed and separated parts of entries distributed over a number of master servers. The servers independently of one another control the order of operations, which leads to processor consistency or cache consistency. Data decomposition is aimed at exploitation of multiple master servers, nevertheless avoiding conflicts. A general multimaster approach, however, allows for conflicting updates at different servers. In this case, the least requirement for directory contents to avoid persistent divergence is eventual consistency [14]. To achieve eventual consistency in the case of unrelated entries, it is sufficient to apply conflicting updates in the same order on each replica, or in practice, apply Last Writer Wins algorithm [3]. Assuming that two updates are in conflict if they modify the same entry, the order of updates interleaved with possible queries defines data order (def. 12). Consequently eventual consistency in this context implies at least cache consistency.
5
Data-Centric Model on the Client Side
Sec. 4 concerns the view on the server side, and omits the view on the client side. If each client uses the same server all the time, it has the same view that is at its server. As clients can switch from one server to another, a given data-centric model on the server side does not guarantee the same consistency at the client side. The point at issue is which session guarantees are necessary to meet on the client side the data-centric models that is preserved on the server side. Fig. 1 shows an execution in which all session guarantees are preserved. However, the write operations are observed by the servers in different order, which results in the violation of s-legality at the client side. Thus, cache consistency on the server side is the necessary consistency condition to ensure any data-centric model on the client side, which is the assumption for further analysis.
A Formal Approach to Replica Consistency in Directory Service
223
Lemma 1. If all session guarantees (i.e. RYW, MW, MR, WFR) are satisfied in the client-server interaction and the views on the server side are s-legal, the view on the client side is equivalent to an s-legal one. Proof. The proof is only sketched. Let us assume for contradiction that a view Ci
on the client side is not s-legal, i.e. for a client, say Ci : ∃w(x)u,w(x)v∈O w(x)v Ci
w(x)u r(x)v. This means that the client Ci gets v from a server which either has not executed w(x)u yet, or has executed w(x)u before w(x)v. The former case means the violation of RYW or MR, and latter MW or WFR.
C1 w(x)1 XX
S1
S2
C2 w(x)2
X z• w(x)1 X @ = • w(x)2 @ 9 w(x)2• @ R• w(x)1 @ 9 r(x)2 9 r(x)1
Fig. 1. Violation of s-legality
C1
S1
S2
C2 w(x)1
w(x)1• 9
X
XX z r(x)1 X w(x)1• w(x)2 w(x)2 •9
r(x)2 9 9 r(x)1 w(x)2•
XX X z r(x)2 X
Fig. 2. Violation of cache consistency
Most data centric models preserve local order, but its definition differs on the client side and on the server side. The following lemma addresses this problem. Lemma 2. If RYW and MW are satisfied in the client-server interaction, execution order preserves issue order, i.e. if a client Ci issues o1 and o2 during the C
Sj
interaction with Sj , the following condition holds: o1 i o2 ⇒ o1 o2. Proof. MW guarantees that two writes appear in issue order, and RYW guarantees that all previously issued writes appear before the subsequent read. Corollary 1. MW and RYW ensure that issue order is preserved in server view. Execution order has no influence on issue order, however, it influences the view on the server side, thereby on the client side. Lemma 3. If execution order is preserved in server view, and MR guarantee is satisfied in the client-server interaction, a view on the client side (or its equivalence) preserves execution order as well. Proof. MR ensures that each read operation by a client is performed at a server which has executed all write operations appearing in this client’s view. The operations appear in execution order, so this order is preserved in client view. Lemma 4. If all session guarantees are satisfied in the client-server interaction, any data-centric model held on the server side is also met on the client side.
224
J. Brzeziński, C. Sobaniec, and D. Wawrzyniak
Proof. The views on the client side are legal (Lemma 1). Let us assume for contradiction that two writes appear in some order in server view, but there is no equivalent view on the client side that preserves this order. The view on the client side must include some read that does not allow showing the equivalence to the server view, which leads to the violation of some session guarantees. The question is about session guarantees that are important to a given datacentric model. The answer is formulated by the following theorems. Theorem 1. If sequential consistency is preserved on the server side, and RYW, MW and MR guarantees are met in the client-server interaction, sequential consistency is fulfilled on the client side. Proof. Lemma 4 states that all session guarantees applied together are sufficient for any data-centric model. Sequential consistency preserves local order, so — following Lemma 2, Corollary 1, and Lemma 3 — RYW, MW, and MR are necessary session guarantees. As for WFR, it is preserved as a result of sequential consistency at the server side, and need not be checked as a property of session. Theorem 2. If PRAM consistency is preserved on the server side, and RYW, MW and MR guarantees are met in the client-server interaction, PRAM consistency is fulfilled on the client side. Proof. PRAM consistency requires only local order to be preserved, so — following Lemma 2, Corollary 1, and Lemma 3 — RYW, MW, and MR are necessary session guarantees. WFR concerns in fact the relationship between operations that are issued by different processes, so is not necessary. Corollary 2. PRAM consistency on the server side implies in fact processor consistency. If RYW, MW and MR guarantees are met in the client-server interaction, processor consistency is fulfilled on the client side. Proof. Cache consistency is assumed on the server side — in combination with local order (PRAM) processor consistency is preserved. Following the reasoning in the proof of Theorem 1, processor consistency is also met on the client side. Theorem 3. If cache consistency is preserved on the server side, and RYW and MR guarantees are met in the client-server interaction, cache consistency is fulfilled on the client side. Proof. Cache consistency does not require local order. RYW is necessary so as not to miss own writes. MR is necessary to avoid reading a previous value from another server (see Fig. 2).
6
Conclusions
Consistency of data provided by a directory service satisfies sequential consistency, processor consistency, or cache consistency depending on the implementation, configuration, and topology. However, because of the delays in
A Formal Approach to Replica Consistency in Directory Service
225
synchronisation the data accessed by a client on different servers can seem not to satisfy the model preserved on the server side. To meet the consistency model on the client side, a conjunction of session guarantees is necessary. For sequential consistency of processor consistency RYW, MW and MR guarantees are sufficient, and for cache consistency RYW and MW. The formal, declarative specification of consistency allows the users and the administrators to understand, predict, and possibly avoid different anomalies in a directory service.
References 1. Stokes, E., Weiser, R., Moats, R., Huber, R.: Lightweight Directory Access Protocol (version 3) Replication Requirements. RFC 3384 (Informational) (October 2002) 2. Birrell, A.D., Levin, R., Schroeder, M.D., Needham, R.M.: Grapevine: an exercise in distributed computing. ACM Commun. 25(4), 260–274 (1982) 3. Shapiro, M., Bhargavan, K.: The Actions-Constraints approach to replication: Definitions and proofs. Technical Report MSR-TR-2004-14, Microsoft Research (March 2004) 4. Terry, D.B., Demers, A.J., Petersen, K., Spreitzer, M., Theimer, M., Welch, B.W.: Session guarantees for weakly consistent replicated data. In: Proc. of the Third Int. Conf. on Parallel and Distributed Information Systems (PDIS 1994), Austin, USA, September 1994, pp. 140–149. IEEE Computer Society, Los Alamitos (1994) 5. Mizuno, M., Raynal, M., Zhou, J.Z.: Sequential consistency in distributed systems. In: Birman, K.P., Mattern, F., Schiper, A. (eds.) Dagstuhl Seminar 1994. LNCS, vol. 938, pp. 224–241. Springer, Heidelberg (1995) 6. Steinke, R.C., Nutt, G.J.: A unified theory of shared memory consistency. Journal of ACM 51(5), 800–849 (2004) 7. Lipton, R.J., Sandberg, J.S.: PRAM: A scalable shared memory. Technical Report CS-TR-180-88, Dept. of Computer Science, Princeton University (September 1988) 8. Goodman, J.R.: Cache consistency and sequential consistency. Technical Report 61, IEEE Scalable Coherence Interface Working Group (March 1989) 9. Ahamad, M., Bazzi, R.A., John, R., Kohli, P., Neiger, G.: The power of processor consistency (extended abstract). In: Proc. of the 5th ACM Annual Symp. on Parallel Algorithms and Architectures (SPAA 1993), June 1993, pp. 251–260 (1993) 10. Hutto, P.W., Ahamad, M.: Slow memory: Weakening consistency to enhance concurrency in distributed shared memories. In: Proc. of the 10th Int. Conf. on Distributed Computing Systems (ICDCS-10), May 1990, pp. 302–311 (1990) 11. Shasha, D., Snir, M.: Efficient and correct execution of parallel programs that share memory. ACM Trans. Program. Lang. Syst. 10(2), 282–312 (1988) 12. Ahamad, M., Neiger, G., Kohli, P., Burns, J.E., Hutto, P.W.: Casual memory: Definitions, implementation and programming. Distributed Computing 9, 37–49 (1995) 13. Cheung, R.Y.M.: From Grapevine to Trader: the evolution of distributed directory technology. In: CASCON 1992: Proceedings of the 1992 Conference of the Centre for Advanced Studies on Collaborative Research, pp. 375–389. IBM Press (1992) 14. Saito, Y., Shapiro, M.: Optimistic replication. ACM Computing Surveys 37(1), 42–81 (2005)
Software Security in the Model for Service Oriented Architecture Quality Grzegorz Kolaczek and Adam Wasilewski Institute of Informatics, Wroclaw University of Technology, Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland {grzegorz.kolaczek,adam.wasilewski}@pwr.wroc.pl
Abstract. The paper presents an original approach to locating security aspects in the Service Lifecycle and Service Oriented Architecture (SOA) quality model. The first part of the paper focus on the quality of SOA and security measures and investigate some functional and non-functional requirements for security measurement. The general discussion about SOA quality and security measures have been summarized by the proposition of the novel multi-agent architecture for SOA systems security level evaluation in the second part of the paper. Keywords: Service Oriented Architecture, Software Quality, Software Security.
1
Introduction
Quality of SOA (Service Oriented Architecture) means usually more then only reducing defects. It has to be connected with the requirements of its users, not only for today but into the future as well. When business and IT expectations are mixed, defect-free product is a necessary, but not sufficient. The real challenge with SOA software is in guaranteeing that the application meets the all (business and technological) requirements set out for it. One of such important thing is security assurance. SOA is an approach to designing, implementing, and deploying product (service) as a puzzle it is created from the set of components implementing discrete business functions. These components may be distributed across the world but for the user they have to be secure. The problem is how to evaluate and confirm a security level of SOA product. The paper introduces an approach to locating security aspects in the Service Lifecycle and SOA quality model. Then some functional and non-functional requirements for security measurement are presented and connected with security assurance in SOA. Finally, conclusions and future works are stated.
2
Quality in Service Lifecycle Management
The ability to effectively use of available methods of QA (Quality Assurance) in the lifecycle of services is fundamental to achieving success within SOA. The R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 226–235, 2010. c Springer-Verlag Berlin Heidelberg 2010
Software Security in the Model for Service Oriented Architecture Quality
227
Fig. 1. Typical service lifecycle
simplest model of Service Lifecycle may include: governance, delivery, execution and measure (fig. 1). Nevertheless the main aim for each stage it is important to think about quality analysis and improvement as well (tab. 1).
Table 1. Quality assurance in the service lifecycle
phase
key achievement [16] – committing to a strategy for SOA within the overall IT strategy Governance – explicitly determining the level of IT and SOA capabilities – articulating and refining the vision and strategy for SOA – reviewing current governance capabilities and arrangements – developing a governance plan – establishing or refining a SOA Center of Excellence (COE) Delivery – defining additional capabilities required, such as upgrades to the IT infrastructure – agreeing on policies for service reuse across lines of business – putting funding mechanisms in place to encourage this reuse – establishing mechanisms to guarantee service levels – deploying new and enhanced governance arrangements Execution – deploying technology to discover and manage assets
quality assurance – reviewing of quality aspects – setting the quality expectations (levels) – developing a quality assurance plan
– defining quality characteristics – developing quality metrics
– deploying rules of result’s interpretation – founding procedures for testing – establishing system of work’s documentation – deploying quality assurance model – using external tools and application to service quality analysis
228
G. Kolaczek and A. Wasilewski Table 1. (Continued)
phase
Measure
3
key achievement [16] – communicating and educating expected behaviors and practices within both the business and IT decisionmaking communities – enabling the policy infrastructure – executing the service – monitoring compliance with policies and governance arrangements, such as service level agreements (SLAs), reuse levels, and change policies – analyzing IT effectiveness metrics
quality assurance
– monitoring and analyzing values of quality metrics
– communicating team to improve quality
within service
SOA Quality Model and Security Metrics
SOA quality model should address multiple aspects of service quality across SOA service implementations. In fact, two aspects seem to be the most important software product and business process quality (fig. 2). Both of them determining final quality of the service, which depends on final user expectations and feelings. Companies to ensure top SOA quality of service have to meet customers business requirements, improve their satisfaction and profitability as well as ensure the highest level of software reliability.
Fig. 2. Typical service lifecycle
Software Security in the Model for Service Oriented Architecture Quality
3.1
229
Quality of Business Process
One of the model for business process quality analysis was defined by A.Selcuk Guceglioglu and Onur Demirirs (fig. 3). It presents a complementary processbased approach and focusing on the quality aspect of the process [12]. The structure of the model is based on ISO/IEC 9126, so includes: – – – –
categories (aspects) of quality; characteristics (functionality, reliability, usability and maintainability); subcharacteristics; metrics (to analyze quality attributes).
Fig. 3. Model for business process quality [4]
In this model security (of the business process) is a part of its functionality and may be measured using access auditability metrics. 3.2
Quality of Software Product
One of the well-known model of standard for description the software product quality is ISO/IEC9126 Software engineering Product quality. It defines six quality characteristics (subdivided into subcharacteristics) for internal and external quality and four characteristics for quality-in-use (fig. 4). ISO/IEC 9126 may be used to specify and evaluate software product quality from different perspectives. It is dedicated for users linked with analysis of requirements, development, evaluation, maintenance, use or audit software and typical examples of its use are to: – – – – – –
validate the completeness of a requirements definition; identify software requirements; identify software design objectives; identify software testing objectives; identify quality assurance criteria; identify acceptance criteria for a completed software product. [5]
230
G. Kolaczek and A. Wasilewski
Fig. 4. ISO 9126 model for software product quality [5]
Security according to ISO/IEC 9126 means the capability of the software product to protect information and data so that unauthorised persons or systems cannot read or modify them and authorised persons or systems are not denied access to them [5]. It has several metrics, e.g. access auditability, access controlability, data corruption prevention, data encryption.
4
Security Level Evaluation for Software Quality Measurement
The IEEE standard for a Software Quality Metrics Methodology describes software quality as the degree to which software possesses a desired combination of quality attributes [2]. In this context the crucial element for the quality measurement are quality attributes which are also called quality characteristics. Quality attributes can be classified into two main categories: execution qualities - such as security and usability, which are observable at run time, and evolution qualities - such as testability, maintainability, extensibility and scalability, which are embodied in the static structure of the software system [8]. According to ISO software quality model [3], security is a component of functionality category which is one of the six categories of quality characteristics that has been defined within this model. Definition of security proposed in this document is the capability of the software product to protect information and data so that unauthorized persons or systems cannot read or modify them and authorized persons or systems are not denied access to them. These two documents devoted to software quality measurement are the main set of guidelines to elaborate the security level evaluation framework for SOA systems. 4.1
Security Requirements and Security Measures for Software Development
There are several well known problems and controversies while defining the exact meaning of the security metrics [1,6,10]. There is no one metrics that is
Software Security in the Model for Service Oriented Architecture Quality
231
acceptable and applicable in context of all possible systems and situations. The security metrics are highly context dependant, so the final shape of the metrics is related to a situation and target depend on security goals, technical, organizational, and operational needs, available resources, etc. At the other hand, metrics are essential in measuring the goodness of target system and it is also true in the context of security quality evaluation, so there is continuous need for security metrics definition. As a system activity cannot be managed well if it cannot be measured, metrics provide the manager with instruments which enable to characterize, to evaluate, to predict and to improve process execution. Security metrics and measurements can be used for decision support, especially in assessment and prediction. Exemplary security metrics for security assessment include [11]: – Risk management activities in order to mitigate security risks, – Comparison of different security controls or solutions, – Obtaining information about the security posture of an organization, a process or a product, – Security assurance of a product, an organization, or a process, – Security testing (functional, red team and penetration testing) of a system, – Certification and evaluation (e.g. based on Common Criteria) of a product or an organization, and – Intrusion detection in a system, – Other reactive security solutions such as antivirus software. For example, to predict the security behavior of an organization, a process or a product in the future some metrics using mathematical models and algorithms can be applied to collect and analyze measured data (e.g. regression analysis). A security metric can be qualified as objective or subjective, quantitative or qualitative, dynamic or static, relative or absolute, and direct or indirect [13]. The ISO/IEC 9126 propose external, internal and indirect metrics categories. The internal metrics are understood as static measure products. This metrics measure quality only indirectly. It has also been assumed that at the early stages of development only resources and process can be measured [9]. The external metrics of this standard measure code when during execution. The indirect metrics is a metric of quality in use. According to SOA systems characteristics there are some specific requirements that should be taken into account while considering security measurement. 4.2
Security Assessment in SOA
The security evaluation process should be based on some formal prerequisites. This means that the security evaluation must be objective to guarantee the repeatability and universality of the evaluation results. So, there must be defined notion of the security measure. There are some confusion about this notion. The first problem is that the security measure does not have any specific unit. The other difficulties are: security level has no objective grounding but it only in
232
G. Kolaczek and A. Wasilewski
Fig. 5. The layered model of Service Oriented Architecture
some way reflects the degree in which our expectation about security agree with reality, security level evaluation is not fully empirical process, etc. [7,6] As the SOA system can be defined by its five functional layers (fig.5) the correspondent definition of SOA security requirements for security evaluation process should address the specific security problems within each layer. Some elements from a set defining security requirements for the SOA layers has been presented in Table 1. describing functional and non-functional security evaluation requirements for each of the SOA functional layers (selection). The complete list can be found in [7] The most important functionality related to the SOA security level evaluation architecture is description of the all components, mechanisms and relations that are necessary to precisely evaluate the security level
Fig. 6. The layered model of Service Oriented Architecture
Software Security in the Model for Service Oriented Architecture Quality
233
Fig. 7. The architecture of the multi-agent system for SOA security level evaluation
of the particular SOA system. As it was described in this section the problem of security evaluation is very complex and there exist more than one solution that could be acceptable within a context of a particular system and its environment. We propose some general idea about SOA security level evaluation in a relation to requirements listed in the table presented in the figure 6. The central part of the proposition is a multi-agent architecture presented in the fig. 7. The multi-agent architecture is composed of three types of agents: monitoring agents that tests the various security parameters related to particular SOA layer, superior agents that manage the activity of monitoring agents, managing agents that are responsible for all superior agents and for communication with service consumer agents. This type of architecture was selected according to its correspondence to SOA characteristic. Within the architecture monitoring agents are responsible for performing the task related to security assessment using the selected metrics while the managing agent should provide the final results. The next step of the research will be devoted to the problem of selection or elaboration of appropriate for SOA systems security measures for monitoring agents and data fusion methods for managing agent. Where: • • • •
5
AMOL – SOA functional layer monitoring agents ASL – SOA functional layer superior agents AM – SOA managing agents AC – the agents of consumers of SOA system services.
Conclusions
Improved interoperability is one of the most prominent benefits of SOA. This type of the systems allows service users to transparently call services implemented in
234
G. Kolaczek and A. Wasilewski
disparate platforms using different languages. However, one of the challenges of eliciting quality requirements for a system is that it may not be possible to know all the collaborating parts. This is especially true in SOA-based systems that provide public services and/or search for services at runtime. A quality measures allows to judge quality of the systems. Quality requirements, such as those for performance, security, modifiability, reliability, and usability, have a significant influence on the software architecture of a system. The use of a service-oriented approach positively impacts some quality attributes, while introducing challenges for others. This paper presented the impact of SOA characteristic on different quality measures. The first part of the paper discussed quality of SOA and security measures and some functional and non-functional requirements for security measurement and then the proposition of the novel multi-agent architecture for SOA systems security level evaluation has been presented in the second part. The future work will concentrate on the evaluation of the proposed architecture for security assessment and refinement of the security measures. Acknowledgements. The research presented in this paper has been partially supported by the European Union within the European Regional Development Fund program no. POIG.01.03.01-00-008/08.
References 1. Hecker, A.: On Security Metrics. ENST, DESEREC project document, http://www.deserec.eu/ 2. IEEE1061, IEEE Standard for a Software Quality Metrics Methodology, The Institute of Electrical and Electronics Engineers: New York, USA (1998) 3. ISO/IEC 9126-1, Software engineering — Product quality — Part 1: Quality model (2001) 4. ISO/IEC 9126-2, Software engineering — Product quality — Part 2: External metrics (2003) 5. ISO/IEC 9126-3, Software engineering — Product quality — Part 3: Internal metrics (2003) 6. Kaner, C., Bond, W.P.: 10th International Software Engineering Metrics: What Do They Measure and How Do We Know? In: Software Metrics Symposium, METRICS 2004 (2004) 7. Kolaczek, G.: Multi-agent Security Evaluation Framework for Service Oriented Architecture Systems. LNCS (LNAI). Springer, Heidelberg (2009) (to appear) 8. Matinlassi, M., Niemel¨ a, E.: The impact of maintainability on component-based software systems. In: Proceedings of 29th Euromicro Conference: New Waves in System Architecture. IEEE Computer Society, Belek-Antalya (2003) 9. Swanson, M., Nadya, B., Sabato, J., Hash, J., Graffo, L.: Security Metrics Guide for Information Technology Systems, NIST 800-55, National Institute of Standards and Technology Special Publication #800-26 (2003) 10. Savola, R.: Towards a Security Metrics Taxonomy for the Information and Communication Technology Industry. In: Proceedings of the International Conference on Software Engineering Advances, ICSEA, August 25-31. IEEE Computer Society, Washington (2007)
Software Security in the Model for Service Oriented Architecture Quality
235
11. Savolainen, P., Niemela, E., Savola, R.: A Taxonomy of Information Security for Service-Centric Systems. In: Proc. of the 33rd EUROMICRO Conference on Software Engineering and Advanced Applications EUROMICRO, pp. 5–12. IEEE Computer Society, Washington (2007) 12. Vaughn, R.B.J., Henning, R., Siraj, A.: Information assurance measures and metrics - state of practice and proposed taxonomy. In: Proceedings of 36th Hawaii International Conference on System Sciences, HICSS 2003 (2003) 13. Zeiß, B., Vega, D., Schieferdecker, I., Neukirchen, H., Grabowski, J.: Applying the ISO 9126 Quality Model to Test Specifications — Exemplified for TTCN-3 Test Specifications, Software Engineering (2007) 14. ISO/IEC 9126 Software engineering – Product quality, First edition 2001-06-15 15. Guceglioglu, A.: Selcuk, Demirors Onur, Using Software Quality Characteristics to Measure Business Process Quality, Middle East Technical University, Turkey 16. http://www-01.ibm.com/software/solutions/soa/gov/lifecycle
Automatic Program Parallelization for Multicore Processors Jan Kwiatkowski and Radoslaw Iwaszyn Wroclaw University of Technology, Institute of Informatics Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland {jan.kwiatkowski,radoslaw.iwaszyn}@pwr.wroc.pl
Abstract. With the advent of multi-core processors the problem of designing application that efficiently can utilize it performance become more and more important. Moreover developing programs for these processors requires from the programmers some additional, specific knowledge about the processor architecture. In multi-core systems efficient program execution is the main issue. It can even happen that switching from sequential to parallel computation can lead to decreasing of performance. The paper deals with the short description of SliCer, the hardware independent tool that parallelizes serial programs in automatic way depending on the number of available processing units by creating the proper number of threads that can be later execute in parallel. Keywords: parallel processing, automatic program transformation.
1
Introduction
Nowadays, it becomes harder and harder to enhance single processor speed, therefore multiplying processing units seems to be the best way to achieve better performance. Because of that multi-core processors become more and more popular for both commercial and home usage. Developing programs for these processors requires from the programmers some additional specific knowledge about the processor architecture and parallel programming. Problems such as memory requirements, synchronization and so on have to be taken into consideration. However, the most important problem that should be solved is efficient program execution. For example, when the number of program ”blocks” that can be executed in parallel is larger than the number of available processing units, it can lead even to performance decreasing due to frequently context switching of the working processes. Therefore the most suitable situation is when the number of program ”blocks” that can be executed in parallel is equal to the number of available processing units. Unfortunately it can causes that program will not be portable by means that it slows down during it execution on the processor with the different number of processing units. Different operating systems provide mechanisms which help programmers and allow to use more than one processing unit efficiently but still the knowledge related to the processor architecture and parallel programming is needed. On R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 236–245, 2010. c Springer-Verlag Berlin Heidelberg 2010
Automatic Program Parallelization for Multicore Processors
237
the other hand, there are available different tools which help programmers in program developing. For example: SWARM [2] that is a portable open source parallel programming framework for designing algorithms for multicore systems, RapidMind [9], an execution environment, which takes serial program as the input and then execute it in parallel using the virtual machine. Using both of above tools still requires from programmers to have some knowledge about the processor architecture and parallel programming. As an alternative solution the automatic program parallelization can be used. The paper deals with the short description of SliCer, the hardware independent tool that in automatic way parallelizes serial programs written in C by creating the proper number of threads that can be later executed in parallel. The proposed solution is similar to Octopiler [5] which enables automatic SIMDization of code and automatic user-directed exploitation of shared memory parallelism, however it needs some transformations that are manually managed by the programmer. Currently SliCer offers only some of functionalities normally supported by the parallelizing compilers. It parallelizes loops [3,4] and can determine parallelism between program blocks. It can be used under Windows or Unix Posix standard, but now only Windows is fully supported. The paper is organized as follow. Section 2 briefly presents design overview. The internal SliCer structure is presented in Section 3. Section 4 describes the parallelization process implemented in the SliCer and presents some implementation details. Section 5 illustrates the results of experiments obtained during SliCer evaluation. Finally, section 6 outlines the work and discusses the further works.
2
Design Overview
Multicore processors gives the opportunity of parallel program execution using the number of available processing units. In common during programming multicore processors, the shared memory paradigm is used. To take advantage of this, it is necessary to develop programs in such way, that there are exactly the same number of tasks (e.g. threads) and available processing units. Typical serial programs can utilize only one processing unit, then during it execution other units are idle. From technical point of view it is easy to change such program behaviour, however when we take into consideration the ”program architecture” it becomes very tough. Moreover, each situation when threads are created and synchronized can leads to many problems. On the other hand to take advantage of using multicore processors it is necessary to modify program code in such way, that the most time consuming program parts, for example loops, will be executed in parallel as a separate threads. It is worth to emphasize that dividing program to threads is not the only possibility of parallelization. It is also possible to execute process for each task, but in such case it is also necessary to ensure communication between processes. The way of performing parallelization process depends on the available hardware platform. Therefore regardless of chosen method it is necessary to divide calculation with respect to each thread. For this aim algorithms determining
238
J. Kwiatkowski and R. Iwaszyn
data dependencies and build on this basis dependency graph has to be used. It leads to choosing proper order of instruction execution to ensure that the result will be the same as from the corresponding serial program. In general, automatic parallelization is made in the following steps [7]: performing a dependence test to detect potential parallelism, restructuring the program into some blocks which can be executed in parallel, generating parallel code for a particular architecture by scheduling selected program blocks on available processing units and then synthesizing a convenient mechanism for achieving parallelism depending on the used system. It is clear that the last step of above procedure depends on the used system architecture when the first two are hardware independent with respect to performed program transformations that can be different for different platforms. According parallelization process two relations in the program code can be determined: the first that express the dataflow (data dependence) and the second, which express the control flow (control dependence). Therefore during program parallelization process, in the first step all control dependences have to be identified and handled. In the second step data dependencies are used for determining which program instructions can be executed in parallel. To avoid the fine grain parallelism which is not suitable for multicore processors with the limited number of processing units initially the program is divided into blocks. Program blocks are disjoint and have exactly one entry and one or more exit points, of course they may have many predecessors and successors and may even be its own successors [6]. For example such program blocks as ”loop” block, ”if-then” block, etc. can be distinguished. It is obvious that program blocks constitute a new program and determining parallelism between program blocks is much more suitable than between instructions. On the basis of data and control dependences SliCer creates the dependency graph. Then the possibility of loop parallelization is checked by determining if flow, output, and antidependences exist and if they caused loop-carried dependency. If loop parallelization is possible the tool calculates dependency distance to determine how many treads can be created. The number of treads used during program execution are determined depending on the number of available processing units as well. To optimize loop parallelization process different loop transformation such as loop splitting, loop distribution and others are considered. Then the parallelism between program blocks is determined. It is achieved on the basis of dependency graph for both data and control. Depending on the control relation different program classes have to be distinguished and different ways of control dependency handling has to be utilized. For example when control relation creates a linear sequence of program blocks and all blocks have only one exit point, there are no control dependencies and data dependencies specified at the dependence graph are used for determining, which blocks can be executed in parallel. In the case when not all block have only one exit point, due to the possible branches in the program additional conditions should be fulfilled during dependence analysis. In the most general case when the control relation is only transitive and reflexive, the possibility of existing the local symmetry of the control relation should be taken into consideration [6].
Automatic Program Parallelization for Multicore Processors
3
239
The Internal Structure of SliCer
Slicer gives an opportunity of serial programs parallel execution on the multicore processors. The tool is implemented in C, as input it takes the source code written in C and as an output it generates the source code of parallel program in C/C++ language. Then the new source is compiled by any available C/C++ compiler. It means that comparing presented approach with standard program developing process only one additional step, parallelization is needed. It is not necessary to introduce any other modification in the program source code. Even if some instructions used in the program cannot be parallelized they will be leaved without changes. If hardware platform used for program execution has to be changed it is not necessary to modify source code. The only thing to do is to parallelize program and compile it. Even when operating system is changed it is not necessary to modify source code (assuming there are no direct system commands in the code). Parallelizing is made by executing proper number of threads. The number depends on the available processing units. Threads share memory available for the process, so no extra communication between them is needed. Threads synchronization is made in two different ways: by defining barrier synchronization or when only two threads should be synchronized, simply by waiting for some event.
Fig. 1. The internal structure of SliCer
The internal structure of SliCer is presented in figure 1.The first SliCer’s unit is Code Analyzer which generates program parse tree. The second one is Optimizer which provides two main SliCer functionalities: it performs loop analysis [3,4] and looks for internal parallelism within the program instructions (blocks) [6,7]. It checks the number of available processing units and depending on its number decides how many program blocks can be generated and executed in parallel. These two units are hardware independent. The last unit is Code Generator which is equipped with built-in library used during code generation. These solution gives the opportunity to used the tool for different hardware as well as software platforms by simple changing the built-in library which consists of functions that depends on the hardware and software. Figures 2 and 3 presents two
240
J. Kwiatkowski and R. Iwaszyn
Fig. 2. The main window of SliCer
main SliCer windows that are used for choosing different configuration settings. Obviously, the number of available processors can be defined. On that basis the tool knows how many threads can be used at most. It is important when transformations are applied to code. Another section allows to set path to compiler and when using Widows to editbin.exe. It is necessary if not only parallel code has to be generated, but compiled version of serial program as well. Last section gives a possibility to set time for each elementary instruction. These settings are used during code execution time estimations. It is also possible to choose different loop transformations. It is important that each simple operation time can be set in configuration menu, because when on the target platform any operation takes more time than on the other, it can be simple expressed by changing time for this operation. All configuration settings are very important during parallelization process. Configuration depends on the target platform to which code is compiled and on current system (e.g. compiler path). All configuration data are stored in XML file, so it is possible to change only this file to ensure compiling for a different platform.
4
Program Parallelization with SliCer
SliCer generates as its output a code written in C that can be after compilation executed in parallel on the multi-core processor. The first SliCer unit is Code Analyzer (Fig.1.), which accepts as its input any correct serial program written in C and generates program parse tree as its output. Parser, which is used, was
Automatic Program Parallelization for Multicore Processors
241
Fig. 3. SliCer configuration window
generated with ANTLR [1], which transforms each grammar rule to the function. Program blocks are basic elements of generated parse tree. All language construction such as loop, conditional statement, and others are derived from blocks (”for” loop block, ”while” loop block, ”if” block, etc). Each block contains variable declarations, assignments, function calls and can have subordinate blocks. Assignment can be placed in any type of block and can have unlimited number of arguments and operators which are stored in tree structure as well. This representation is very useful for code generating because operators priority is handled in the most natural way by simple tree traversing. Argument can be of simple or complex (e.g. function call) types. There are two types of assignments: variable initialization and non-initial for already declared variables. Function call represents calling of any function within the program code. It can has unlimited number of arguments of any type. Arguments can be as complex as the right side of assignments (other function calls are acceptable as well). Variables declarations are divided into three groups: simple variables (e.g. int x), arrays (e.g. float[] x) and pointers (e.g. char * x). Such representation is useful for code generating. It is not necessary to have any fields in class which describes its type, all variables can be derived from the same class. When program parse tree is created the Optimizer begins optimization process. It traverses the tree and tries to parallelize each block. First of all it creates data dependency graph for the block. Then it checks the possibility of loop parallelization (currently only for ”for” loop). Based on the graph Optimizer checks which of available transformations are possible. While transforming loop into parallel version it is often possible to use more than one transformation and
242
J. Kwiatkowski and R. Iwaszyn
even if it is chosen it doesnt mean that another one cannot be used. That’s why during parallelization process one of the most important decision is to choose the most suitable transformation set which allows to achieve maximum efficiency. In order to make such choice possible SliCer applies a few transformation sets to each parallelized block and estimates execution time. For every set estimated execution time is stored. This process is achieved virtually which in this case means that parallelized block can be restored to original version and transformations don’t have to be applied to the output code. It is impossible in real programs to check each possible transformation set for each parallelized block, because of large amount of possibilities. To simplify this process SliCer assumes that the best way to choose appropriate transformation set is to check modifications which are ”the best” from the statistical point of view. When we consider not-nested ”for” loops, the most efficient transformations include each type of loop partitioning, currently ”Loop splitting” and ”Loop distribution” [8]. The first one breaks the loop into a parts which has the same body but iterate over different index range. When applying this transformation it is necessary to modify starting index of the loop and its step. Loop body remains unchanged. After that loop parts can be executed simultaneously on multiple processing units (cores). The second transformation ”Loop distribution” works in the different way; loop is divided into multiple loops which iterates over the same index range, but each of them takes only the part of original loop body. This transformation is not so convenient as the first one. The reason is that the execution time of instructions are different and it is hard to divide loop equally into parts with the same execution time. When using the same body it can be considered that each of them is executed in the same time (in most cases). Therefore, the time of execution of each thread should be similar. It is harder to achieve this using ”Loop distribution”. But obviously usage of this modification is beneficial and SliCer tries to apply it, if it is impossible to use ”Loop splitting”. SliCer assumes that if it is possible to partition loop it is always the best choice (excluding of course some special cases e.g. very short loops). It leads to the assumption that sets which include loop partitioning has to be considered. If none of them is possible then the tool tries different possibilities. Although it is not always possible to use loop partitioning as first transformation it can be possible later (e.g. after anti-dependency elimination process). Because of that at each parallelization step SliCer checks the possibility of applying loop partitioning. Algorithm of choosing the most suitable transformation ends when there is no more possible transformation to apply. The only exception is when the tool detects that it generated as many threads that it is impossible to run them in parallel and one or more processing unit will be idle. In such case it is necessary to divide extra work for each unit to achieve its maximal utilization. This issue will be considered in details in our further work. Currently five different loop transformations are implemented: loop splitting, loop distribution, loop interchange, loop unrolling and loop reversal. In the next SliCer version more transformation will be taken into account and parallelization of other loops will be included. In the second step the parallelism between program blocks is
Automatic Program Parallelization for Multicore Processors
243
determined. Then for each possible transformation Optimizer estimates its execution time and chooses the fastest one. Firstly it estimates serial execution time, then parallel execution time for different transformations and finally calculate speedup that can be received for each of them. In that way SliCer knows whether it brings benefits to made the transformation, the best transformation can be chosen. To determine estimated execution time of the block SliCer adds running time of each elementary instruction. To achieve this each operation (e.g. assignment, function call, mathematical operators) has assigned its execution time. Obviously calculated time can be used only to determine speedup and calculations are made in virtual units. Real execution time is different than time estimated by the SliCer. Thats why before program modification is saved SliCer knows whether they brings benefits or not. If so transformations are saved, otherwise they can be withdrawn. To determine the number of generated threads very simple algorithm is used. After transformation process the number of threads, which can be executed in parallel is known, similarly the number of processing units is known, too (specified in configuration). Then if the number of threads that can be executed in parallel is lower than the number of processing units, the number of generated threads are equal to the number of threads defined by the parallelization process. In other case the number of generates threads are equal to the number of available processing units. For this purpose using the estimated time of threads execution, different threads are joined together. In general, in this way the work during program execution is equally distributed among available processing units. When the code is optimized, modified program parse tree moves to Code generator. It traverses the tree and put code for every instruction, block, etc. into output stream. While generating the code it is formatted. Generator uses special module for system calls, because there are different for each operating system (especially thread create functions). It is possible to change this module by implementing special interface to generate code for different platforms. Below the skeleton of implemented algorithm for loop parallelization is presented. For each ”for” loop block apply the following procedure. Firstly create data dependency graph and estimate sequential program
Fig. 4. Speedup when using loop splitting transformation
244
J. Kwiatkowski and R. Iwaszyn
Fig. 5. The cost of parallelization process
execution time, then check possibility of applying ”Loop splitting” transformation. If it is possible apply it, otherwise check the possibility of applying ”Loop distribution” transformation. ”Loop splitting” transformation can be applied when dependency distance for loop is lower than loop step. Otherwise it is possible as well but in more specific cases. It strictly depends on the data dependency graph. Possibility of using this transformation is checked first because it gives the best speedup in program execution. Modification is made in such manner that maximum number of existing cores is used. It is not always possible to use all cores, because of different loop indexes ranges. Then apply ”Loop unrolling” transformation. ”Loop distribution” transformation can be applied when data dependency graph is divided into independent part (subgraphs). Each of them can be represented as a new loop with the same index range as original one but with different loop body. The number of used processing units depends on the number of existing units and independent parts of data dependence graph. Then apply ”Loop unrolling” transformation to all loops created by previous steps if there are only a few iterations to be made. Finally estimate parallel execution time and calculate possible speedup. If there are no speedup, rollback performed transformation, otherwise apply modifications.
5
Results of Experiments
Different experiments have been performed to confirm the usefulness of designed and implemented tool. During the experiments two different hardware platforms have been used Intel Xeon X3040 1,86 GHz and Intel Core Quad Q6600 2.4GHz. The presented results are the averages from the series of 100 experiments performed in various conditions. Obtained speedup has been measured for different loop transformations as well as for some example programs. Analysed loops consists of different number of iterations, from 10 to 100. The average one loop run execution time has been about 1 sec. Real measured speedup has been compared with estimated speedup calculate during parallelization process. For all
Automatic Program Parallelization for Multicore Processors
245
performed experiments the results have been very similar, for two most important transformation, loop splitting and loop distribution and for example programs with loops the speedup has been close to the number of used processing units. Figure 6 presents the results obtained for loop splitting transformation. The left diagram shows results when the program execution time has been correctly estimated by the SliCer and right diagram when estimation has been incorrect. What is very important even when estimation process has not been correct it doesn’t slow down program execution time. Figure 7 shows the cost of parallelization process. It can be determine that overhead related to parallelization process is not very large, the time of parallelisation time is close to the compilation time.
6
Conclusions and Future Work
The project is in current study, therefore presently the tool covers only a part of the functionalities that normally are supported by the parallelizing compilers. The prototype still misses a lot of features and not all types of instructions can be parallelized, they will be implemented in the further versions of the tool. However experiments performed using the first prototype indicates that the presented tool will be very useful for standard multi-core processors. The speedup received for programs parallelized by the tool has been close to the number of available processing units. Additionally using the tool, no specific knowledge about the used processors and experience in the parallel programming are required to build the parallel application. Acknowledgements. The research presented in this paper has been partially supported by the European Union within the European Regional Development Fund program no. POIG.01.03.01-00-008/08.
References 1. ANTLR v.3 Documentation (2007), http://www.antlr.org/wiki/display/ANTLR3/ANTLR+v3+documentation 2. Bader, D.: SWARM Framework, http://www.cc.gatech.edu/~ bader/papers/SWARM.html 3. Banerjee, U.: Loop Transformations for Restructuring Compilers: The Foundation. Kluwer Academic Publishers, Dordrecht (1993) 4. Banerjee, U.: Loop Parallelization. Kluwer Academic Publishers, Dordrecht (1994) 5. Eichenberger, A.J., et al.: Using advanced compiler technology to exploit the performance of the Cell Broadband Engine architecture. IBM Sys. J. 45(1), 59–84 (2006) 6. Kwiatkowski, J.: Automatic program Restructuring. In: Melecon 1991 International Conference, IEEE Catalog No 91CH2964-5, pp. 1041–1044 (1991) 7. Polychronopoulos, C.D.: Parallel Programming and Compilers. Kluwer Academic Publishers, Dordrecht (1988) 8. Randy, A., Kennedy, K.: Optimizing Compilers for Modern Architectures: A Dependence-Base Approach. Morgan Kaufmann Pub., Academic Press (2002) 9. RapidMind., Development Platform, http://www.rapidmind.com
Request Distribution in Hybrid Processing Environments Jan Kwiatkowski, Mariusz Fras, Marcin Pawlik, and Dariusz Konieczny Institute of Informatics, Wroclaw University of Technology, Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland {jan.kwiatkowski,mariusz.fras,marcin.pawlik, dariusz.konieczny}@pwr.wroc.pl
Abstract. In recent years the evolution of software architectures led to the rising prominence of the Service Oriented Architecture (SOA) concept. This architecture paradigm facilitates building flexible service systems. The services can be deployed in distributed environments, executed on different hardware and software platforms, reused and composed into complex services. In the paper a SOA request analysis and distribution architecture for hybrid environments is proposed. The requests are examined in accordance to the SOA request description model. The functional and non-functional request requirements in conjunction with monitoring of execution and communication links performance data are used to distribute requests and allocate the services and resources. Keywords: SOA architecture, SOA request distribution.
1
Introduction
The increasing significance of information technology in contemporary business is determined by two factors: new concepts of modelling processes and software architectures and development of technical means especially in the area of computing resources. New hybrid environments are designed to utilize the power of classical processing architectures together with new architectural developments. Classical processors, used individually or grouped in clusters and grids, are cooperating with novel architectures: general-purpose graphics processing units (GPGPU), filed-programmable gate arrays (FPGA) or general-purpose multiprocessor hybrid architectures [4,3]. The versatility of classical processors is combined with the computational power offered by novel architectures to meet the requirements of contemporary scientific and business applications. In recent years the evolution of software architectures led to the rising prominence of the Service Oriented Architecture (SOA) concept. This architecture paradigm facilitates building flexible service systems. The services can be deployed in distributed environments, executed on different hardware and software platforms, reused and composed into complex services. In the paper [6] the following SOA definition can be found: ”SOA is an architecture approach R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 246–255, 2010. c Springer-Verlag Berlin Heidelberg 2010
Request Distribution in Hybrid Processing Environments
247
for defining, linking, and integrating reusable business services that have clear boundaries and are self-contained with their own functionalities. Adopting the concept of services SOA takes IT to another level, one that’s more suited for interoperability and heterogeneous environments. A service is a function that is well-defined, self-contained, and does not depend on the context or state of other services”. Taken the above requirements into account, the new software architecture has to be developed. In the paper a SOA request analysis and distribution architecture for hybrid environments is proposed. The requests are examined in accordance to the SOA request description model. To ensure efficient resource utilization, incoming SOA requests are attributed to execution classes. The functional and non-functional request requirements in conjunction with monitoring of execution and communication links performance data are used to distribute requests and allocate the services and resources. The paper is organized as follow. Section 2 briefly presents design overview. The general architecture of the SOA request distribution system is presented in the Section 3. Sections 4 and 5 describe two parts of the proposed architecture: the execution analyser and matching module. Finally, section 6 outlines the work and discusses the further works.
2
Design Overview
SOA services are characterized by mobility, reusability and a large degree of execution independence from the underlying hardware infrastructure. The analysis of historical and ongoing SOA services execution data, together with the system monitoring results and the information about the service requirements can be utilized to distribute the incoming SOA service execution requests. The goal of the project is to design and implement the SOA system infrastructure architecture. In the proposed architecture (Fig.1) the service requirements combined with the system monitoring data are utilized to ensure efficient SOA service execution, performed in accordance with the requirements imposed. To achieve this goal, the methods of semantic description of services, resources and requirements was implemented. Ontology-based dependence models of the infrastructure, services and execution requirements were designed. Algorithms were developed to transform execution and requirements information into the execution flow knowledge. A recommendation and description method was proposed for selected SOA/SOKU services utilization in SOA applications. Different aspects related to systems quality assurance, reliability, availability, safety and security were taken into consideration during analysis process. In general system architecture the three main modules can be distinguished Execution Virtualization module, Request and Execution Knowledge Aggregator module and Request Distribution module. The Execution Virtualization module implements an SOA service execution interface, used to hide the underlying hardware-specific details. The virtualization makes it possible to utilize a variety of hardware resources dynamically selected to ensure efficient SOA service execution performed in accordance to the requirements. The Request and Execution
248
J. Kwiatkowski et al.
Fig. 1. The general view of the system
Knowledge Aggregator module aggregates, interprets and utilizes the monitoring data to define the SOA service execution location and execution details using built-in prediction system. The Request Distribution module analyzes the incoming SOA requests, compares them to the historical data acquired from the Request and Execution Knowledge Aggregator module and decides which resources will be used for its execution by the Execution Virtualization module.
3
SOA Request Distribution System
At the highest level of abstraction any service can be seen as a complex service. The complex service has to be decomposed onto atomic services. Basically a complex service can be considered as a graph of atomic services. An atomic service cannot be decomposed into smaller parts. During decomposition it is taken into account only the functional parameters of services. Because of that in the moment it is needed to choose a versions of atomic services which preserve
Request Distribution in Hybrid Processing Environments
249
the expected nonfunctional properties. For every version of atomic service it is needed to choose an instance of such version, from a set of instances which can be registered in different localization. But the registration only inform, that a chosen system can deliver an atomic service with specified functional and not functional properties. But the system itself can run the service on different realizations. This means that during servicing a request the system has to choose a realization of instance of service. Our system starts its work from receiving an atomic service request. The information systems designed with SOA paradigm are working both in local and wide area networks, particularly built with Internet and GRID recourses. For SOA service client, there is no need to carry about the localization and execution meanings of the service it want to use. However, the services are well described in the system and their localizations are known. It can dynamically change, however this process is not frequent, and usually the set of localizations of given service is constant for some period [6]. The paradigm of SOA says that given kind of services can be achievable from different service providers. Because the effectiveness of different SOA servers as well as distance to them (and thus communication costs) can be different, the quality of service delivery (especially the time of delivery) can be significantly different too [9]. To deliver SOA services more effectively we propose the architecture of SOA request distribution system. The assumptions for proposed distribution system for SOA services are following: – the SOA services are considered as the same if they offer the same functionality, – the given SOA service is available in different localizations (execution systems) distributed in the network, – the set of execution systems for given SOA service is known and constant, – the set of SOA services (that are to distribute) is limited and known. The general model of SOA request distribution system assumes that there is special node in the edge of client area the SOA request broker (Fig. 2) which manages distribution of requests for SOA services. For given service there is known constant set of M execution servers I = I1 , I2 , , IM , where Im is identifier of server m, and M communication links sm . The client can be an end user or a SOA service that composes its services with use of other services e.g. building complex services from atomic ones. We assume that the distance between the client and the broker is negligible. The service delivery time consists of two parts: the data transfer time tT of the client request and server reply, and service request execution time tO (preparation of the reply data at the execution server - all back-end processing of the request). In general, even for given class of request, these two values are not constant. They depend on dynamic change of communication link parameters as well as current state of execution server (e.g. its load). Our aim is to distribute client’s request to such server that service delivery time will be the smallest one. Hence, the broker has to support the following functionality:
250
J. Kwiatkowski et al.
I1 s1 C
sm
B
Im
sM IM Fig. 2. General distribution system model
– monitoring of servicing client requests, – prediction of selected communication link parameters, – prediction or registration (in case of cooperation model) of selected remote server’s parameters, – estimation of request transfer times and request execution times for all known servers, on the basis of adopted model of communication links and execution servers, – if necessary, adaptation of adopted model. In the Fig. 3 we propose the general architecture of the broker that fulfils all functionality necessary to determine for every request xi arriving at the moment n destination server which should serve this request in order to satisfy same time based distribution criteria. The decision un of the choice of one of M servers is performed on the basis of i the vector TˆT,n i.e. the vector of estimated transfer times tˆm,i T,n of request xi to i ˆ each server m, and vector TO,n i.e. the vector of estimated execution times tˆm,i O,n of request xi at each server m, m ∈ 1, ..., M . These times are estimated with the aid of models of execution servers and communication links built for every possible class ci of requests. If the client of the broker is SOA service the class of request may be provided by the client. In the other case some classification procedure ought to be applied. Monitoring of Request Transfer Module
xi
SOA Request Classification Module
TiT,n Network Measurement Module
i Comunication TˆT ,n Link Estimation Module
>tˆ
1,i ˆ m,i ˆ M ,i T , n , ..., t T , n , ..., t T , n
Vn Decision Making Module
ci
Request Execution Analyzer
@
Ln TiO,n
SOA Server Estimation Module
TˆOi ,n
>tˆ
un
1,i ˆ m,i ˆ M ,i O ,n , ..., t O ,n , ..., t O ,n
Fig. 3. General architecture of the SOA request broker
@
Request Distribution in Hybrid Processing Environments
251
The key parts of the broker are estimation modules of the communication links and execution servers. They estimate corresponding times according to fuzzy-neural models of links and servers similar as in [1]. Each module is built of M models of servers (or links, each for one destination server or communication link) that works as a controller. The input of the model (Fig. 4) is class ci of the request xi that arrives at the moment n, the server state parameters vector Lm n of the server m, known at the moment n, and completion times of requests executions till moment n. The vector Lm n may consist of different parameters characterizing the current state of the server. We take into account two: the load of the server, and its efficiency. The output of each model block of the module is estimated execution time of the request at each server.
Request Execution Analyzer
SOA Server 1 Model
Lmn ci
tˆO1,,in
t Om,,ni
SOA Server m Model
SOA Server M Model
tˆOm,,ni
tˆOM,n,i
Fig. 4. SOA Server Estimation Module
The communication link estimation is very similar. The difference is input Vnm (instead of Lm n ) which is the vector of appropriate link parameters, and measured m real transfer times tˆm,i T,n of prior requests (before the moment n). Vector Vn consist of two parameters that are taken under consideration: forecasted link latency (namely TCP Connect Time) and forecasted link throughput. These two parameters are derived from periodic measurements of latency and link throughput, and using time series analysis based prediction algorithms. Each model (server and link) is built for each destination and for each class of the request. It is designed as fuzzy-neural controller (Fig. 5) accomplished with use of 3-layered neural network [1,7,2]. The first layer of the network constitute the fuzzyfication block. For link model, the inputs are forecasted link latency tˆT CP C,n at the moment n, and forecasted link throughput tˆT h,n . The outputs of the fuzzyfication block are the grades of membership of latency µtT CP C in defined fuzzy sets for latency, and grades of membership of throughput µtT h in
252
J. Kwiatkowski et al.
defined fuzzy sets for throughput. The logic block inferences output. It computes grades of membership µR to the rules that are of the format proposed in [1]: IF tT h = ZT H,a AND tT CP C = ZT CP C,b THEN y = tT,c where: Z fuzzy sets for throughput (Th ) and latency (T CP C), tT estimated transfer time (output y), a, b, c indexes of appropriate fuzzy sets and output.
tˆTCPC ,n
tˆTh, n
PtTCPC Fuzzyfication block
PtTh
Logic block
PR
Defuzzyfication block
tˆTm, n,i
Fig. 5. Communication link model as a fuzzy-neural controller
After the defuzzyfication (using weight method) estimated transfer times of the request are derived, and along with measured actual value of transfer the network parameters are tuned using back propagation method [5].
4
Parallel Execution Performance Measurements
The request distribution algorithm described in the previous sections requires the information about the resource load and utilization. A monitoring system was designed to provide this information [8]. The system, called rsus is a distributed environment dedicated to perform low-overhead, external measurements of necessary data. The measurements are performed in a non-invasive manner, not requiring any modifications to the service being measured. The rsus environment is dedicated for linux operating system - the most popular OS in the HPC applications. The modularity of rsus architecture makes it relatively easy to port it to other operating systems. The environment offers two execution modes. In the basic mode it utilizes the data gathered by the operating system in the /proc virtual directory hierarchy to ensure low-overhead measuring method. In the extended mode rsus utilizes the communication performance data gathered by the mpiP profiler [10]. For every execution environment, the wall-clock time is measured together with the CPU time communication time and the execution environment load. The rsus environment is built from a set of applications. The main application, rsus is used to perform the measurements and has to be present in every execution environment monitored. It can be used to start and monitor a SOA service or, alternatively, attach to a working service and perform the measurements. Rsus application can send the monitoring results in a continuous, on-line fashion or in an off-line manner after the SOA request is processed by the service. The monitoring results are kept in the RAM memory of the environment being monitored. To avoid the memory exhaustion, the data can be periodically offloaded to the hard drive.
Request Distribution in Hybrid Processing Environments
253
Fig. 6. Rsus system architecture
The rsus application can be utilized in an autonomous fashion but in the proposed system it constitutes a part of the fully assembled monitoring installation. The data gathered is sent to the control application rsus-mom. The data transfer path is established by the rsus application sending an UDP broadcast request consisting of its execution environment address and receiving in response the rsus-mom address. Rsus-mom controls all the rsus applications in the system it is thus possible to analyze the information coming from all the services that participate in the request execution. To distinguish information coming from different requests, the data sent from rsus application, each service execution data has an unique identifier attached. The identifier can be automatically determined or set manually as an rsus application execution parameter. The data acquired by the rsus-mom application is saved on the hard drive for further analysis. It can be later processed by a dedicated analyzer application rsusrp. To integrate the rsus monitoring system with the existing execution environment the set of execution scripts can be used. Currently the execution scripts are available for PBS-compatible parallel execution environments. The rsus monitoring system architecture is depicted on fig. 6.
5
Static Matching Module
The aim of static matching module is to find the best execution environment for a request. Because we have heterogeneous hybrid hardware, with difference load balance we need to check on which ones we can serve the request, and decide on which one we finally execute the code. We assume that we can use a virtualizers with which we can simulate hardware or operation systems. This means that to really run a service we need: hardware, virtualizer, operating system (with libraries), service (executable code). To prepare this all elements we have to find all possible such couples and that decide which one to use. To generate the couples the module first check what hardware we have. Then it have to be connected every unit of hardware with proper virtualizers which
254
J. Kwiatkowski et al. Hardware description Rule-based Vitrualizer description
Expert System
Virtual hardware description
Execution environment
Virtual operating
Matching
system description
Algoritm
description
Requirements
Set of atomic service instances
matching algorithm
Description of atomic service instances
Complex
Selection algorithm
Set of selected atomic service instances
service
Fig. 7. Matching module architecture
can be executed on that hardware. In a description of registered virtualizers there are some rules which define the requirements of it (among of memory, kind of processor, etc.) Such a pair constitutes a virtual hardware (VH). On any of such virtual hardware it can be run an operating systems (it is called a virtual operating systems - VOS). In description of a VOS there are rules about requirement of virtual hardware needed for running the VOS. A pair of VOS and VH constitutes possible execution environment (EE). Then we have to connect service realization with desirable non-functional properties with adequate execution environments. In description of service realization there are rules which define the non-functional properties depending on properties of virtual operating system. At this moment we have a set of possible concrete realization (hardware, virtualizer, virtual operating system, service realization). Depending on load balance and on execution analysis of previous execution of the service realization the module choose one of the concrete realization and propagate it to execution virtualization module with all information needed to execute. Because not all hardware have their virtualizers, there will be also a dummy virtualizer. The virtualizer will vitrualize nothing and it will be in special way interpreted by execution module. This means that the rules for virtualization will change nothing, but this virtualizers are also special marked to inform the execution module to not run a virtualizer. It can also mean that the operating system is still run on such processor and it is only needed to execute a service realization. The system of information about hardware, virtualizers, virtual systems and service realizations is very flexible. Depending on registering the information of aforementioned elements the whole system can be viewed as a one instance of service or not. The information can be registered in such way that an executor
Request Distribution in Hybrid Processing Environments
255
can distinguish between different types of hardware but without distinguishing computation units, or in other way that service will be interchangeably assigned to exactly one unit of hardware. It allow to use different policy of hardware facilitate for execution of service requests.
6
Conclusions and Future Work
In the paper the architecture of a newly designed system of requests distribution in SOA environments is described. A prototype implementation of the system was created, comprised of the Request Distribution module, Stating Matching module, Dynamic Execution Monitoring module and a preliminary implementation of the Execution Virtualization module. Initial tests were performed, confirming the correct functioning of the system. Currently the set of performance tests is conducted to ensure efficient, low overhead operations of the final system implementation. Acknowledgements. The research presented in this paper has been partially supported by the European Union within the European Regional Development Fund program no. POIG.01.03.01-00-008/08.
References 1. Borzemski, L., Zatwarnicka, A., Zatwarnicki, K.: Global distribution of HTTP requests using the fuzzy-neural decision-making mechanism. In: Nguyen, N.T., Kowalczyk, R., Chen, S.-M. (eds.) ICCCI 2009. LNCS, vol. 5796, pp. 752–763. Springer, Heidelberg (2009) 2. Jain, L.C., Martin, N.M.: Fusion of neural networks, fuzzy sets, and genetic algorithms industrial applications. CRC Press LLC, London (1999) 3. Halfhill, T.: Parallel Processing With CUDA, Microprocessor Report (2008) 4. Kahle, J., Day, M., Hofstee, J.C., Maeurer, T., Shippy, D.: Introduction to the cell multiprocessor. IBM Journal of Research and Development (2005) 5. Keong-Myung, L., Dong-Hoon, K., Hyung, L.: Tuning of fuzzy models by fuzzy neural networks. Fuzzy Sets and System 76(1), 47–61 (1995) 6. Mabrouk, M.: SOA fundamentals in a nutshell, IBM Corp. (2008) 7. Mamdani, E.H.: Application of Fuzzy Logic to Approximate Reasoning Using Linguistic Synthesis. IEEE Trans. on Comp. C-26(12), 1182–1191 (1977) 8. Pawlik, M.: Execution Time Decomposition Method in Performance Evaluation, Science and supercomputing in Europe - report 2007. In: CINECA (2007) 9. Williams, W., Herman, R., Lopez, L.A., Ebbers, M.: Implementing CICS Web Services, IBM Readbook (2007) 10. Vetter, J., McCracken, M.: Statistical Scalability Analysis of Communication Operations in Distributed Applications. In: Proc. ACM SIGPLAN Symp. (2001)
Vine Toolkit - Grid-Enabled Portal Solution for Community Driven Computing Workflows with Meta-scheduling Capabilities Dawid Szejnfeld1 , Piotr Domagalski2 , Piotr Dziubecki1 , Piotr Kopta1 , Michal Krysinski1 , Tomasz Kuczynski1 , Krzysztof Kurowski1 , Bogdan Ludwiczak1 , Jaroslaw Nabrzyski2 , Tomasz Piontek1 , Dominik Tarnawczyk1, Krzysztof Witkowski2 , and Malgorzata Wolniewicz1 1
Poznan Supercomputer and Networking Center Tel.: (+48 61) 8582174, Fax: (+48 61) 8582151 {dejw,piotr.dziubecki,pkopta,mich,tomasz.kuczynski, krzysztof.kurowski,bogdanl,piontek,dominikt,gosiaw}@man.poznan.pl 2 FedStage Systems Tel.: (+48 61) 6427197, Fax: (+48 61) 6427104 {piotr.domagalski,krzysztof.witkowski,jarek.nabrzyski}@fedstage.com
Abstract. In large scale production environments, the information sets to perform calculations on come from various sources. In particular, some computations may require the information obtained as a result of previous computations. Workflow description offers an attractive approach to formally deal with such complex processes. Vine Toolkit [1] solution addresses some major challenges here such as the synchronization of distributed workflows, establishing a community driven Grid environment for the seamless results sharing and collaboration. In order to accomplish these goals Vine Toolkit offers integration on different layers starting from rich user interface web components, integration with workflow engine and Grid security and ending up with a built-in meta-scheduling mechanisms, that allow IT administrators to perform load balancing automatically among computing clusters and data centers to meet peak demands. As a result of this particular project a complete solution has been developed and delivered. Keywords: Vine Toolkit, Vine, GRMS, Flowify, wow2green, GRIA, middleware.
1
Introduction
Vine Toolkit (in short Vine) is a modular, extensible Java library that offers developers an easy-to-use, high-level Application Programmer Interface (API) for Grid-enabling applications and even more as it will be described in the given article. It was developed within EU funded projects like OMII Europe [2] or R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 256–265, 2010. c Springer-Verlag Berlin Heidelberg 2010
Vine Toolkit
257
BEinGRID [3] and polish infrastructural project PL-Grid [4]. It was used to build an advanced portal solution integrating many different grid technologies within BEinGRID business experiment called wow2green (Workflows On Web2.0 for Grid Enabled infrastructures in complex Enterprises) [5]. The solution will be described to illustrate the great scope of Vine usage and its advanced features. 1.1
Problem Description
The general problem solved within wow2green business experiment is presented in the given paragraph. This use case is just an illustration of how and where the Vine Toolkit can be used. This article is focused mainly on Vine itself and only on the business logic layer without going into end user interface and functionality description. Big companies consist of several units generating a variety of information sets and performing advanced computations. Workflow description offers an attractive approach to formally deal with the complex processes. Unfortunately, existing workflow solutions are mostly legacy systems which are difficult to integrate in dynamically changing business and IT environments. Grid computing offers flexible mechanisms to provide resources in the on-demand fashion. Wow2green system - called officially Flowify [6] which is based on the Vine Toolkit - is a Grid solution managing computationally intensive workflows used in advanced simulations. Each department within big enterprise or a small company in a supply chain: – performs unsynchronized computations among departments, – exchanges computing results independently with other project participants, so input data is re-entered in different formats, output data is lost or overwritten, – overloads computing resources during deadlines. Flowify helps users synchronize the data processing for execution and collection of results in an automated fashion. Easy-to-use Flowify workflow interfaces are provided as intuitive Web applications or integrated with commonly used tools. Thus many users can simultaneously form dynamically workspaces, share not only data for their computing experiments but also manage the whole process of advanced calculations. Built-in GRMS meta-scheduling mechanisms provided by Flowify allow IT administrators to perform load balancing automatically among computing clusters and data centers to meet peak demands. 1.2
Main Requirements and Demands
The following main requirements have been identified within wow2green experiment: – enable workflow executions via the wow2green portal (portal based solution should provide a suitable, user-friendly interface for workflow executions using Web 2.0 style easy-to-use web interfaces. Users should have a fully transparent access to underlying computing resources so they can take advantage of high level workflow management)
258
D. Szejnfeld et al.
– provide detailed information and feedback for users about their computing workflow simulations (every workflow is a kind of template for creating a computing simulation which could be later executed on different computing resources. External workflow engine was selected - Kepler [7] which supports provenance mechanisms required to search and discover templates in created workflows. These features are also available via the wow2green portal) – enable more efficient platform usage through applying brokering of jobs defined in workflow nodes (GRMS [8] broker is used as a metascheduling and brokering component for load balancing among computing resources. Thus, end users will not have to select computing resources manually and the overall utilization of computing resources will be improved) – easy-to-use stage in and stage out simulation data (users are able to upload and load simulation data and workflows to and from our portal in an intuitive way) – easy-to-use, Kepler-like graphical user interfaces for workflow management and control (to provide user friendly web interfaces we selected a new web technology called Adobe Flex. It will enable us to implement various web applications for easy management and control of workflows via portal) – support for new workflow definitions (additional web tools for workflows will be developed to enable users to create, share and modify workflows. Users (workflow creators) will be able to share created workflows with other users interested only in performing simulations of existing workflows). – provide a repository of workflow definitions.
2
Flowify Architecture
The overall Flowify architecture is presented below, all components are described to show its function and reveal the complexity of the prepared environment (Fig.1). Identified components: – Gridsphere portal framework - portal framework that provides an opensource portlet based Web portal. GridSphere enables developers to quickly develop and package third-party portlet web applications that can be run and administered within the GridSphere portlet container. The wow2green portal is based on this framework. – Vine Toolkit - used as business model layer within the portal, applications created for the scenario uses the Vine mechanisms to interact with grid services and workflow engine, integrated Adobe Flex with BlazeDS technology allows creating advanced web applications. Vine Toolkit components used in portal: o Workflow Manager - general component used to interact with different workflow management solutions through common API; in case of the wow2green solution kepler plugin was introduced
Vine Toolkit
259
Fig. 1. Wow2greem system (Flowify) overall architecture
–
–
–
–
o Role Manager - role management component which allows to manage role based access within the portal regarding GUI elements and accessible actions and also to define role based access policies regarding data managed through logical file manager component o File Manager - File Manager component which allows to interact with different data management systems by means of common API o Portal Security Component - used for portal user certificates and proxies management, together with the Certificates generation module allows to automatic certificate generation for registered portal users o Portal User Management - used for portal users management, user registration in the portal; together with the MS Active Directory plugin it allows for using existing accounts in the given organisation to make the integration easier with the existing IT environment (it is also possible to use some existing LDAP service if required). Vine web service - web service which exposes a subset of functionality of the Vine Toolkit component to external systems/services - regarding Job and File Manager, the web service is used by Kepler actors to interact with grid enabled resources Kepler workflow engine - core part of the Kepler/Ptolemy standalone application/framework for workflow management, it is used in the wow2green solution as primary workflow management engine, kepler workflow description called MOML is used within the system Kepler engine web service - the web service exposes the workflow engine functionality to external systems/services, portal workflow manager component uses it as the workflow management service Set of kepler’s vine actors - base components of the workflows’ definitions to submit jobs and manage data through the Vine Toolkit service layer
260
D. Szejnfeld et al.
– GRMS broker with web service interface - job broker component with advanced scheduling technology, scheduling policies could be easily exchanged dependently on the requirements – GRIA middleware [9] - grid middleware - service-oriented infrastructure (SOI) designed to support B2B collaborations through service provision across organizational boundaries. Following services are used in the wow2green system: Job Service - job management, Data Service - data management, Membership Service - to manage groups of users relevant to the roles defined in the portal – Griddle Information Service - web service which exposes the information about available GRIA resources and batch queues statistics.
3
Vine Toolkit Portal Based Solution
Vine Toolkit (Fig. 2) is a modular, extensible Java library that offers developers an easy-to-use, high-level Application Programmer Interface (API) for Gridenabling applications. Vine can be deployed for use in desktop, Java Web Start, Java Servlet 2.3 and Java Portlet 1.0 environments with ease. Additionally, Vine Toolkit supports a wide array of middleware and third-party services. Using the Vine Toolkit, one composes applications as collections of resources and services for utilizing those resources (basic idea in Vine - generic resource model - any service and data source can me modeled as an abstract entity called resource and can be integrated with web applications using high-level APIs). The Vine Toolkit makes it possible to organize resources into a hierarchy of domains to represent one or more virtual organizations (VOs). Vine offers security mechanisms for authenticating end-users and authorizing their use of resources within a given domain. Other core features include an extensible model for executing tasks (every action is persisted as Task) and transparent support
Fig. 2. Vine Toolkit high level architecture in portal deploy scenario
Vine Toolkit
261
for persisting information about resources and tasks with in-memory or external relational databases. Adobe Flex [10] and BlazeDS [11] technology allows creating advanced and sophisticated web applications similar to many stand-alone GUIs. 3.1
Workflow Engine Support
Vine Toolkit API has been enriched with the generic Workflow Engine API. It was created as a separate sub project which could be added as a feature by the portal installation. API reflects identified requirements regarding functionality used within end user web applications to manage users’ workflows through the independent API. It allows quite simply add support for different workflow engines if desired or required by some applications constraints. For the wow2green experiment Kepler engine was used as the primary workflow engine. The base API allows define so called WorkflowDefinition which represents the workflow definition prepared as an XML string. In case of Kepler engine MoML [12] based workflow description is used. Through the use of generic WorkflowDefinitionParameter a set of input parameters could be passed along with the workflow definition to the target workflow engine plugin. WorkflowManager component plays role of the generic workflow manager while the concrete instance is created during workflow submission to the service deployed on the target host. Resources defined in the Vine domain description (it is a resource registry in the given domain). Vine contains Kepler plugin project which encapsulates the Kepler engine client to implement the generic Vine Workflow API. Figure 1 clearly shows the connection from the portal to the Kepler Web Service it is another component which exposes the functionality of the Kepler workflow engine to the external clients acting as a workflow management service. The key point here are so called the Vine actors small java classes which are in fact Vine Web Service clients. These ”bricks” are used by the workflow designer to prepare workflow description to execute applications using grid resources and to acquire results of the computations. User is not aware where the computations take place - Vine Web Service by using grid services like GRMS which meta-scheduler and job broker dispatches the jobs on the accessible grid environment. Vine is used here in different separate role in servlet mode to be a backed of the Vine Web Service and provide ease and unified access to different grid services like in this case GRMS and GRIA. Such construction allows to easy switch to different grid technology and reconfiguration of the whole environment to different middleware stack like Unicore 6 [13] for example. 3.2
Uniform Interface to HPC Infrastructures
In order to provide end-users with transparent access to resources, we developed a mechanism responsible for the management of uniform interfaces to diverse Grid middleware. Advanced classloader management mechanism enables dynamic loading of components that provide access to the functionality of specific Grid middleware through a single uniform interface. These components are
262
D. Szejnfeld et al.
called plug-ins in this context (and are realized as Vine subprojects with separate set of client jars). These uniform interfaces are provided for such functionality like job submission, control, monitoring, data management, role management, user registration and authentication. They are based on standards where possible (e.g. JSDL for job submission) and take into account both the gathered requirements and the functionality provided by Grid middleware deployed in HPC centers. Job Submission, Control and Monitoring Interface. One of the interfaces which makes use of the mechanism described above is the uniform interface for job submission functionality. It was based, first, on an analysis of the functionality of middleware systems and, second, on the Job Specification Description Language (JSDL) [14] created by the JSDL working group at the Open Grid Forum [15]. The following main functionality was included in the job submission interface: startJob, stopJob, getJobOutputs, getJobStatusById, createJobSpec and beside this a bunch of support methods like getJobByTaskId to retrieve job instance from Vine using its task id. Currently Vine supports such middleware like gLite 3.x [16] WmProxy and CREAM, Unicore 6 UnicoreX job submission, Globus GT4 [17] WsGram, GRIA Job Service and recently QosCosGrid [18] SMOA Computing and GRMS. Vine’s plugin management allows to simultaneous usage of different middleware technologies what enables to uniform access to the different HPC resources. In case of wow2grenn use case GRIA and GRMS Plugins were used to access grid services. Another motivation beside seamless usage of different grid technologies was ease of configuration and possibility to switch on to different mieddleware stack without modifications to the developed web applications (Vine plays role of the abstract access layer with well defined interface). Data Management Interface. Another interface is the uniform interface for data management functionality. It was based, first, on an analysis of the functionality of storage services of middleware systems trying to find common API. The following main functionality was included in the data management interface: getHomeDirectory, createInputStream, createOutputStream, info, exists, list, goUpDirectory, changeDirectory, makeDirectory, copy, rename, move, delte, upload, download, createNativeURLString and beside this a bunch of support methods like getDefaultFileManager to retrieve default data manager instance from Vine. Currently Vine supports such middleware like gLite 3.x SRM and LFC, Globus GridFTP, Unicore 6 Storage Management System (SMS), WebDAV, Storage Resource Broker (SRB), GRIA Data Service, HTTP. Vine’s plugin management allows to simultaneous usage of different storage technologies and copying data between them. In case of wow2grenn use case GRIA Data Service and HTTP Data Service plugins were used to access grid services. By using Vine abstract layer the web applications could be written without taking into account different technological nuances. Vine allows also to use different storage servers as uniform services making data management quite easy for the application developer.
Vine Toolkit
3.3
263
Security Issues
Vine can solve different security issues while accessing different grid services. A significant interoperability problem results from the differences in the security models of systems such as UNICORE or GRIA (where authentication is based on standard X.509 certificates) and Globus-based Grids (authentication based on GSI [19]). The main difference between these systems is that basic versions of UNICORE and GRIA do not support dynamic delegation while GSI-compliant systems do. Currently, to overcome this problem, we allow a portal to authenticate on behalf of end-users. To do this, the portal signs a job with its own certificate and adds the distinguished name of the end-user before submitting the job to, for example, a GRIA site. In case of Unicore Vine supports GSIenabled extension for Unicore gateway what enables to authenticate users by using proxy certificates. In case of gLite3 middleware voms proxy certificate are created and used - Vine can be configured to support different voms servers and allow users interactions from different VO without any problem. Vine support MyProxy [20] server to provide true single sign-on - during the user logging in to the portal proxy certificate is retrieved and used while accessing different middleware services. In case of the wow2gren solution the GRIA middleware was used. So as mentioned before portal common certificate was used to submit jobs (GSI is not supported). To distinguish users so called GRIA policy rules are used to set job and data file owners by using users’ public certificates and their distinguish names. One of the requirement of the experiment was to simplify the security management issues. So user certificates are maintained by the Vine based portal integrated with Certificate Authority - while user is registered for the first time its certificate is generated and hold in the safe place on the server. Later proxy certificates are used to interact with the grid middleware. Common portal certificate is used to submit jobs to enable GRMS broker to manage them on behalf of users without need to past the private user’s key certificate. 3.4
User Management
Common problem to solve while using different middleware infrastructures is user management and account provision. Vine is able to register users in different middleware infrastructures like gLite3, Unicore 6 or Globus GT4. Vine contains registration modules which allows to register users on different resources. Currently such ones are provided: GridsphereRegistrationResource - allows adding new users to Gridsphere portal, GssCertificateRegistrationResource - creates x509 certificate and key pair for a new user, Gt4RegistrationResource - creates a user account on Globus Toolkit 4 host machine, GriaRegistrationResource - creates a user account in Gria 5.3 Trade Account Service, Unicore6RegistrationResource creates a user account on Unicore 6 XUUDB, DmsRegistrationResource - register a user on Data Management Service, VomsRegistrationResource - register a user to Virtual Organization Membership Service. Modular architecture and plugin management system allows easy to plugin other registration modules if
264
D. Szejnfeld et al.
desired and extend Vine with new functionality. In case of wow2green use case GssCertificateRegistrationResource, GridsphereRegistrationResource and DmsRegistrationResource were used to generate users’ certificates, create portal accounts and logical file system DMS [21]. Vine comes also with bundled administrative web applications regarding user managamnet. One of it is UserRegistrationApp which allows to create requests for new accounts in Vine. Another is UserManagementApp where administrator has got option for deactivate user account and edit user’s profile information. In near future also Liferay [22] portal framework will be supported with suitable registration module. 3.5
Role Management
Vine introduces also role management mechanism to authorize users while taking different actions in the web applications. Roles are used in two scenarios as roles at the portal level and middleware stack level. Roles at the portal level allow to set permissions to access different functionality in the web applications based on Vine. Administrator is able to set permissions for roles to different part of application and distinguish users regarding their possible scope of possible actions. Roles at the middleware stack are used to set access permissions regarding jobs and data files at the storage services. For the wow2green use case GriaRoleResource, DmsRoleResource were implemented. Thus the roles created in the portal are propagated to the middleware stack - in this case GRIA. User by using end user interface is able to share data for the selected roles. Such action is propagated to the middleware resources and could be used later while accessing remote storage by using given role name. Such approach makes permission management much simpler and does not multiply remote policies for every user of the system - it is enough to assign policy for some role and assign an user to the given role at the target system. Of course it is applicable only on middleware system which supports roles management like GRIA for example or DMS logical file system.
4
Conclusions
In this paper we presented Vine Toolkit as universal framework to create software solution to interact with different services and technologies (among others different grid middleware stacks) by means of uniform, easy API. The generic architecture of the Vine Toolkit were described. We also included some requirements and initial ideas for the wow2green business experiment as illustration of Vine application in a real, practical use case. Vine allows to submit jobs to different middleware stacks through a uniform user interface. In similar way uniform API for data management, user authentication, role and user management are provided. Resource based model allows straightforward service support and configuration by uniform way regardless of the target technology. Security issues which always occur while accessing different middleware stacks are solved by using Vine registration modules and proxy certificates support. Role management
Vine Toolkit
265
mechanism allows administrators and end users for easy permission management in the applications and easy grant permissions to data files on the storage server if applicable. Beside simple jobs whole workflows could be managed by the uniform workflow interface. Currently only the Kepler engine is supported but it could be easily extended by adding another workflow engine plugin. Vine Toolkit could be used as framework for building advanced web applications for example with use of Adobe Flex technology as well as business logic provider for advanced web service solutions.
Acknowledgments We are pleased to acknowledge support from the EU funded OMII-Europe and BEinGRID projects and polish infrastructural project PL-Grid.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.
13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
Vine Toolkit web site, http://vinetoolkit.org/ OMII-Europe project, http://www.omii-europe.com/ BEinGRID project, http://www.beingrid.eu/ PL-Grid project, http://www.plgrid.pl/en Wow2green business experiment, http://www.beingrid.eu/wow2green.html, http://www.it-tude.com/wow2green_sol.html Flowify, http://www.flowify.org/ Kepler, https://kepler-project.org/ GRMS, http://www.gridge.org/content/view/18/99/ GRIA, http://www.gria.org/about-gria/overview Adobe Flex, http://www.adobe.com/products/flex/ Adobe BlazeDS, http://opensource.adobe.com/wiki/display/blazeds/BlazeDS/ MoML language specification, http://ptolemy.eecs.berkeley.edu/publications/papers/00/moml/ moml erl memo.pdf Unicore 6, http://www.unicore.eu/ JSDL, http://www.ogf.org/documents/GFD.56.pdf Open Grid Forum (OGF), http://www.gridforum.org/ gLite 3.x, http://glite.web.cern.ch/glite/default.asp Globus GT4, http://www.globus.org/toolkit/ QosCosGrid, http://node2.qoscosgrid.man.poznan.pl/gridsphere/gridsphere Grid Security Infrastructure (GSI), http://www.globus.org/security/overview.html MyProxy, http://grid.ncsa.illinois.edu/myproxy/ Data Management System (DMS), http://dms.progress.psnc.pl/ Liferay, http://www.liferay.com/
GEM – A Platform for Advanced Mathematical Geosimulations Radim Blaheta, Ondˇrej Jakl, Roman Kohut, and Jiˇr´ı Star´ y Institute of Geonics AS CR, Studentsk´ a 1768, 708 00 Ostrava-Poruba, Czech Republic {blaheta,jakl,kohut,stary}@ugn.cas.cz http://www.ugn.cas.cz
Abstract. The contribution deals with the development of a 3-D finiteelement package called GEM and its aspirations in demanding mathematical modelling and simulations arising in geosciences. On the background of two complex applications from the presently running projects, formulated as linear elasticity and thermo-elasticity problems, the most important characteristics, especially those of the solvers, are presented. Features related to high performance computing, including parallel processing, are focused on. Keywords: Finite element modelling, parallel solvers, applications in geosciences.
1
Introduction
GEM can be characterized as an in-house non-commercial 3-dimensional finiteelement package oriented on the solution of problems arising in geosciences. It serves both research purposes and practical modelling and its development is mainly problem-driven, reflecting the needs of the current research and applications. This contribution presents GEM in its development towards a code for advanced and demanding modelling. In this sense, it reviews its interesting features and points to relevant references for more detailed treatment. GEM was originally designed for the solution of linear elasticity models, which were appropriate for most problems arising from the local mining industry to facilitate coal extraction from deep underground deposits. As a rule, the goal of the modelling procedure was to assist the expert evaluation of the stressstrain fields induced by mining activities in complex geomechanical conditions, to increase the safety of the extraction process both in the project and production phase.
2
Elasticity Problems
From the mathematical point of view, the considered elasticity problems are formulated in terms of displacement u(x) of points x of the body Ω ⊂ R3 . In the weak formulation, we want to find u : Ω → R3 , u − u0 ∈ V
and a(u, v) = b(v) ∀v ∈ V ,
R. Wyrzykowski et al. (Eds.): PPAM 2009, Part I, LNCS 6067, pp. 266–275, 2010. c Springer-Verlag Berlin Heidelberg 2010
(1)
GEM – A Platform for Advanced Mathematical Geosimulations
267
where (i)
V = {v = (v1 , v2 , v3 ) : vi ∈ H 1 (Ω), vi = 0 on ΓD ⊂ ∂Ω for i = 1, 2, 3}, a(u, v) = cijkl εij (u) εkl (v) dx Ω ∂uj ∂ui ∂vl 1 ∂vk εij (u) = 12 ∂x + , ε (v) = + kl ∂xi 2 ∂xl ∂xk . j Above H 1 (Ω) is the Sobolev space, u0 ∈ [H 1 (Ω)]3 is the function, which can (i) prescribe nonzero Dirichlet boundary conditions, cijkl are elastic moduli, ΓD are parts of the boundary ∂Ω and b is a linear functional determined by the applied forces. For more details see e.g. [1]. 2.1
GEM’s Building Blocks
The core of every modelling software is the solver for numerical solution of the differential equations describing the model. In the GEM’s case, the solver can be highlighted as follows: Finite elements. For the solution of the problem of elastic deformation (1), the finite element (FE) method is employed. The FE discretization of this boundary value problem, based on linear tetrahedral finite elements in GEM, leads to the linear system of the type Au = b ,
u, b ∈ Rn ,
(2)
where A is a symmetric positive definite n × n stiffness matrix, b ∈ Rn is the right-hand side given by the loading and u ∈ Rn is the n-dimensional vector of unknown nodal displacements. The dimension n of the system can be very high, typically in the range from 105 to 107 for the 3D analysis of elasticity problems of stress changes induced by mining. Structured meshes. GEM makes use of structured meshes for the discretization of the modelled domain Ω, which can be viewed as adaptation (deformation) of a regular cubical (reference) grid of nodes — see e.g. [11]. The cubical cells are divided into six tetrahedral elements in a way known as Kuhn’s triangulation. We have developed procedures that allow structured meshes to fit the geometry of geomechanical models quite easily. Moreover, the resulting stiffness matrix provides possibility for efficient storage and fast matrix handling routines, essential for demanding modelling. Note that the use of structural grids is quite common in geomechanical applications, although in some other application areas this approach may not be flexible enough. Conjugate gradients and preconditioning. The symmetry and positive definiteness of the stiffness matrix A permit to solve the linear system (2) by the (iterative) preconditioned conjugate gradient (PCG) method. In the PCG algorithm,
268
R. Blaheta et al.
the preconditioning operation is a very important for its efficiency — roughly speaking, it improves the search directions for the solution. GEM employs preconditioning based on the incomplete factorization and the so-called displacement decomposition, see e.g. [2]. For parallelization, we combine displacement decomposition with overlapping domain decomposition approach. We shall tackle this topic in the next section. 2.2
Parallelization
Displacement decomposition (DiD). DiD separates the three components of the nodal displacement vector by rearranging and decomposing the vector u (and correspondingly the matrix A of the system (2)) to u = (u1 , u2 , u3 ), where the blocks u1 , u2 , u3 respectively include all nodal displacements in x, y, z directions. With this partitioning, parallelization is relatively straightforward, since the displacement directions can be processed by three concurrent processes. Thanks to the block diagonal structure of the preconditioner in the DiD case, the preconditioning step does not involve any communication. However, beside a few small-sized communications in each iteration of the parallel PCG algorithm, the most demanding data exchange is involved in the repeated matrix by vector multiplication using the same displacement decomposition. Aiming at the reduction of iterations, we realized an alternative to the this fixed preconditioning, namely the so-called variable preconditioning, in which the pseudoresiduals are computed by an inner CG method, or more precisely, by three independent (communication-free) PCG procedures.1 In this way we can reduce the number of outer iterations (and their overhead) at the price of increasing the computational work involved in the preconditioning. This is appropriate for parallel systems with high computation/communication performance ratio, e.g. with slow interprocess communication. Note, that in general the variable preconditioner is not linear, symmetric and positive definite, and the standard CG method can fail or converge too slowly. Therefore, we consider also the generalized preconditioned CG method (GPCG) [7,5], with an additional orthogonalization of the search directions. See [9] for a more detailed treatment of DiD and parallel PCG based on it. The parallelization based on DiD made possible “comfortable” solution of demanding models of several millions degrees of freedom (DOF) more than ten years ago already. It perfectly fits small parallel environments, at that time consisting of 3 – 4 workstations, usually with low-speed interconnecting network. But also nowadays parallel computations based on DiD can match e.g. personal computers with four-core processors. Domain decomposition (DD). For HPC however, more scalability of the parallel computations is definitely required. That is why another space decomposition method [7] is implemented in parallel GEM. With DD, GEM partitions the domain Ω “horizontally” in the Z dimension into m subdomains with a two-layer 1
Each inner CG iteration can be again preconditioned by the incomplete factorization. Moreover, it is sufficient to compute the pseudoresiduals with a low inner accuracy.
GEM – A Platform for Advanced Mathematical Geosimulations
269
minimal overlap, perform corresponding decomposition of vectors and matrices and assigns them to m concurrent processes, which pursue the CG algorithm on the corresponding blocks of data. During the matrix-by-vector multiplication just local communication with neighbours is needed and the amount of communicated data is fairly small and proportional to the overlapped region. Moreover, efficient Schwarz type preconditioners can be constructed, see [8] as classical reference. Our version of these preconditioners can be characterized by the following features: – use of subproblems corresponding to the slightly overlapping subdomains, – inexact solution of the subproblems by incomplete factorization, – algebraic creation of an auxiliary global approximation by aggregation of DOF and its use in two-level version of the preconditioner, – inexact solution of the global subproblem by inner CG iterations, – use of both additive and multiplicative (hybrid) combination of corrections from local and global subproblems. These topics, too extensive to be covered in detail at this place, are discussed in [6]. Note that algebraic construction of both local subproblems and the global approximation provides a black box approach. Realization aspects. From the technical point of view, the developed parallel realizations of the PCG algorithm in GEM follow the message passing paradigm, and can be implemented by means of any message passing system. We have made use both of PVM and MPI message passing, called from Fortran codes. The design gives rise to one master process and a number of slave processes, one per partition. As a computationally inexpensive task with just controlling functions, the master can share its processor with one of the workers. See [7] for more realization details. 2.3
Applications – Analysis of Geocomposites
The number of practical elasticity computations related to mining was very large during GEM’s existence. Maybe the most challenging mathematical model of this kind, with great impact on GEM’s development, was the simulation of extraction of a uranium ore deposit in the Bohemian-Moravian Highlands, see [11] for details. It aimed at the assessment of geomechanical effects of mining, e.g. the comparison of different mining methods from the point of view of stress changes and possibility of dangerous rockbursts. The current elasticity computations performed in GEM concern another research theme: microstructure 3D finite element modelling. In Fig. 1 we can see a FE model of a geocomposite specimen from the coal environment reinforced by injection of polyurethane resin. Such technology can be used e.g. to strengthen coal pillars during mining. The mechanical properties as well as permeability depend on degree of filling the fractures and properties of the used resin. The numerical upscaling and evaluation of properties of the homogenized geocomposite material enable to assess the effect of grouting and can be also used for optimization of the grouting process.
270
R. Blaheta et al.
Fig. 1. CT scan of a geocomposite specimen (left) and timings of its GEM processing using various solvers (right – see text for a legend)
For the numerical upscaling, the geocomposite samples (cubes with edge 75 mm) are scanned by X-ray computer tomography, which provides their microstructure and properties of different voxels. In the case above, the sample was discretized with an uniform grid of 231 × 231 × 37 voxels, giving 2 045 312 nodes and more than 6 million DOF for the investigation of elastic properties. The homogenized (upscaled) properties are obtained by strain and stress driven tests implemented numerically by means of the FE analysis. To give an idea on performance of GEM’s solvers, Fig. 1 right provides wall-clock solution times for the corresponding FE system on a NUMA system with 8 AMD Opteron quad-core processors and 128 GB of RAM. The following parallel solution methods apply: displacement decomposition with fixed (DiD fix.) and variable (DiD var.) preconditioning and two-level Schwarz domain decomposition without/with an algebraic coarse space created by aggregation of DOF (DD/DD+CG). The sequential computing time (not shown) was 304 seconds. Note that the grid size small in the Z dimension limited the scalability to 16 subdomains.
3
Thermo-Mechanical Modelling
The investigation of performance of deep geological repositories (DGRs) of the spent nuclear fuel, a pressing issue associated with nuclear power utilization, motivates advancement of GEM’s modelling repertoire towards multiphysics. Modelling of DGRs in full extend is a great challenge since it has to take into account many processes like heat transfer, water flows, mechanical behaviour and also transport of chemical agents and chemical reactions. Moreover, mutual interactions of the processes and very long DGR operation make the modelling even more complex. This research orientation underlines the necessity of efficient parallel solution methods.
GEM – A Platform for Advanced Mathematical Geosimulations
271
In an initial step we focused on the thermal and mechanical (T-M) phenomena with the fairly natural assumption that temperature changes produce additional deformations but the mechanical deformations are only slow and do not influence the temperature fields. More precisely, we consider only thermo-elasticity modelling with an one-directional coupling via a thermal expansion term in the constitutive relations. In this case, the model can be mathematically formulated as follows: Find the temperature τ : Ω × (0, T ) → R and the displacement u : Ω × (0, T ) → R3 that fulfill the equations κρ −
∂2τ ∂τ =k + Q(t) ∂t ∂xi 2 i
∂σij = fi (i = 1, . . . , 3) ∂xj j σij = cijkl [εkl (u) − αkl (τ − τ0 )] kl
εkl (u) =
1 2
∂uk ∂ul + ∂xl ∂xk
in Ω × (0, T ) ,
(3)
in Ω × (0, T ) ,
(4)
in Ω × (0, T ) ,
(5)
in Ω × (0, T ),
(6)
together with the corresponding boundary and initial conditions. The four expressions represent the heat conduction equation, equations of equilibrium, Hook’s law and tensor of small deformations, respectively, with symbols having the following meaning: κ is the specific heat, ρ is the density of material, k are the coefficients of the heat conductivity, Q is the density of the heat source, σij is the Cauchy stress tensor, εkl is the tensor of small deformations, f is the density of the volume (gravitational) forces, cijkl are the elastic moduli, αkl are the coefficients of the heat expansion and τ0 is the reference (initial) temperature. The considered time interval is denoted (0, T ). For the heat conduction, the boundary conditions prescribe the temperature, the heat flow through the surface heat flux and the heat transfer to the surrounding medium. For the elasticity, the boundary conditions set the displacement, stresses and surface loading. The initial conditions are represented only by the initial temperature. See e.g. [3] for this topic. 3.1
Numerical Processing
GEM solves the initial-boundary value problem of thermo-elasticity (3) – (6) in two steps: First, the temperature distribution is determined by the solution of the nonstationary heat conduction equation for a set of time points. Second, the linear elasticity problem for a chosen subset of those time points is solved in a post-processing procedure. Time stepping. After the variational formulation, the problem is discretized by finite elements in space and finite differences in time. GEM, employing linear
272
R. Blaheta et al.
tetrahedral finite elements and the simplest finite differences, computes the vectors τ j and uj of the nodal temperatures and displacements respectively at the time points tj (j = 1, . . . , N ), defining the time steps Δtj = tj − tj−1 . GEM follows the time-stepping algorithm, which involves repeated solution of linear systems with millions of DOF: For each time point, we solve the thermal linear system (Mh + ϑΔtj Kh )τ j = (Mh − (1−ϑ)Δtj Kh )τ j−1 + ϑΔtj qhj + (1−ϑ)Δtj qhj−1 and in a chosen “interesting” time levels also the elasticity linear system Ah uj = bj In the expressions above, Mh is the capacitance matrix, Kh is the conductivity matrix, Ah is the stiffness matrix, qh represents the heat sources and b = bh (τ ) comes from volume and surface forces including a thermal expansion term. The parameter ϑ ∈ 0, 1 sets the so-called time scheme of the computation: the explicit Euler scheme for ϑ = 0, the Crank-Nicholson (CN) scheme for ϑ = 12 and the backward Euler (BE) scheme for ϑ = 1, which is used by default. Adaptive time stepping. To ensure accuracy and economize the computational effort, it is worth adapting the time steps according to the behaviour of the solution. For this, we use the procedure based on a local comparison of the BE and CN time steps. Evaluating some easily computable ratio, we can control the subsequent time step Δtj and adjust its size if the variation is too small or large. For more details on this point see [10] and the references therein. Parallel solution. The parallel solution of the heat conduction problem is conceptually analogous to the DD elasticity solution with one exception: For reasonable sizes of time steps, one-level Schwarz method is efficient enough and construction and use of a global auxiliary subproblem is not necessary, see e.g. [12]. 3.2
The APSE Problem
As an example of a thermo-mechanical model solved in GEM, let us introduce the APSE problem, w