Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page i 2008-10-24 #1
PROCESS
ALGE...
46 downloads
1935 Views
4MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page i 2008-10-24 #1
PROCESS
ALGEBRA FOR PARALLEL AND DISTRIBUTED PROCESSING
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page ii 2008-10-24 #2
Chapman & Hall/CRC Computational Science Series SERIES EDITOR Horst Simon Associate Laboratory Director, Computing Sciences Lawrence Berkeley National Laboratory Berkeley, California, U.S.A.
AIMS AND SCOPE This series aims to capture new developments and applications in the field of computational science through the publication of a broad range of textbooks, reference works, and handbooks. Books in this series will provide introductory as well as advanced material on mathematical, statistical, and computational methods and techniques, and will present researchers with the latest theories and experimentation. The scope of the series includes, but is not limited to, titles in the areas of scientific computing, parallel and distributed computing, high performance computing, grid computing, cluster computing, heterogeneous computing, quantum computing, and their applications in scientific disciplines such as astrophysics, aeronautics, biology, chemistry, climate modeling, combustion, cosmology, earthquake prediction, imaging, materials, neuroscience, oil exploration, and weather forecasting.
PUBLISHED TITLES PETASCALE COMPUTING: Algorithms and Applications Edited by David A. Bader PROCESS ALGEBRA FOR PARALLEL AND DISTRIBUTED PROCESSING Edited by Michael Alexander and William Gardner
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page iii 2008-10-24 #3
PROCESS
ALGEBRA FOR PARALLEL AND DISTRIBUTED PROCESSING
EDITED BY
MICHAEL ALEXANDER WILLIAM GARDNER
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page iv 2008-10-24 #4
Cover Image Credit: Intel Teraflops Research Chip Wafer from Intel, used with permission.
Chapman & Hall/CRC Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2009 by Taylor & Francis Group, LLC Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 6 5 4 3 2 1 International Standard Book Number-13: 978-1-4200-6486-5 (Hardcover) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Process algebra for parallel and distrubuted processing / editors, Michael Alexander and William Gardner. p. cm. -- (Chapman & Hall/CRC computational science series) Includes bibliographical references and index. ISBN 978-1-4200-6486-5 (alk. paper) 1. Parallel processing (Electronic computers) 2. Electronic data processing--Distributed processing. 3. Formal methods (Computer science) I. Alexander, Michael, 1970 Sept. 25- II. Gardner, William, 1952QA76.58.P7664 2009 004.01’51--dc22 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com
2008029295
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page v 2008-10-24 #5
Contents
Foreword
vii
Acknowledgments
ix
Introduction
xi
Editors
xix
Contributors
xxi
Part I 1
2
3
Parallel Programming
Synthesizing and Verifying Multicore Parallelism in Categories of Nested Code Graphs Christopher Kumar Anand and Wolfram Kahl Semi-Explicit Parallel Programming in a Purely Functional Style: GpH Hans-Wolfgang Loidl, Phil Trinder, Kevin Hammond, Abdallah Al Zain, and Clem Baker-Finch Refinement of Parallel Algorithms Fredrik Degerlund and Kaisa Sere
1 3
47
77
Part II Distributed Systems
97
4
Analysis of Distributed Systems with mCRL2 Jan Friso Groote, Aad Mathijssen, Michel A. Reniers, Yaroslav S. Usenko, and Muck van Weerdenburg
99
5
Business Process Specification and Analysis Uwe Nestmann and Frank Puhlmann
129
6
Behavioral Specification of Middleware Systems Nelson Souto Rosa
161
v
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page vi 2008-10-24 #6
vi
Contents
7
Abstract Machine for Service-Oriented Mobility Herv´e Paulino
199
8
Specifying and Implementing Secure Mobile Applications Andrew Phillips
235
Part III Embedded Systems 9
Calculating Concurrency Using Circus Alistair A. McEwan
10 PARS: A Process Algebraic Approach to Resources and Schedulers MohammadReza Mousavi, Michel A. Reniers, Twan Basten, and Michel Chaudron 11 Formal Approach to Derivation of Concurrent Implementations in Software Product Lines Sergio Yovine, Ismail Assayad, Francois-Xavier Defaut, Marcelo Zanconi, and Ananda Basu Index
285 287 331
359
403
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page vii 2008-10-24 #7
Foreword
This book brings together the state of the art in research on applications of process algebras to parallel and distributed processing. Process algebras constitute a successful field of computer science. This field has existed for some 30 years and stands nowadays for an extensive body of theory of which much has been deeply absorbed by the researchers in computer science. Moreover, the theoretical achievements of the field are to a great extent justified by applications. The applications, in turn, strongly influence how the field evolves; some of the field’s success may be attributed to frequently addressing needs that arose in practice. Meanwhile, an explosion of complex systems of interacting components has been going on since the emergence of parallel and distributed processing. The complexity of the systems in question arises to a great extent from the many ways in which their components can interact. In developing a complex system of interacting components, it is important to be able to describe the behavior of the system in a precise way at various levels of detail, and to analyze it on the basis of the descriptions. Process algebras were and are developed for that purpose. Roughly speaking, a process algebra provides a collection of operators, a collection of equational laws for these operators, and a mathematical model of these laws. The latter allows for the behavior of a system to be described as composed of the emergent behaviors of several interacting components and for the described behavior to be analyzed by mere algebraic calculations. The advent of process algebras was marked by the introduction of CCS in the seminal monograph A Calculus of Communicating Systems by Milner, published as volume 92 of Springer’s Lecture Notes in Computer Science in 1980; the elaboration of CSP in the influential paper “A theory of communicating sequential processes” by Brookes, Hoare, and Roscoe, published in the Journal of the ACM in 1984; and the presentation of ACP as a strict algebraic theory in the paper “Process algebra for synchronous communication” by Bergstra and Klop, published in Information and Control in 1984. The very first applications of process algebras were mostly concerned with the description and analysis of communication protocols. Later on, the applications became more and more advanced. Often, they were concerned with the description and analysis of embedded systems, and more recently with Internet-based distributed systems. The applications of process algebras led to a number of developments. The very first applications brought about the development of basic algebraic verification techniques, i.e., basic techniques to establish—on the basis of algebraic
vii
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page viii 2008-10-24 #8
viii
Foreword
calculations—whether the actual behavior of a system is in agreement with its expected behavior. The construction of basic tools to facilitate description and analysis followed. Later applications led to extensions of existing process algebras and the development of more advanced algebraic verification techniques. Both make it easier to describe and analyze the behaviors of the systems often encountered in practice nowadays: systems that may change their communication topology dynamically, systems that must react within a certain amount of time under certain circumstances, systems that exhibit at certain stages behavior that is stochastic in nature, systems that in their behavior depend on continuously changing variables other than time, etc. Experience with the existing process algebras led also to the development of special process algebras for the definition of the semantics of programming languages that support parallel programming or the design of microprocessors that utilize parallelism to speed up instruction processing. It is difficult to foresee future developments and applications, yet some tendencies are noticeable. One area that is gaining momentum is specialized process algebras that are tailored to a certain paradigm for parallel or distributed computing or even to a certain technology for parallel or distributed computing. This fits in with the tendency to apply process algebras to describe and analyze a prototype of a certain class of systems. Such applications can be useful in understanding certain aspects of the systems of an emerging class. However, the adequacy of the prototypes may be a major issue, because often drastic simplifications are needed to keep the description manageable. There is also a tendency to apply process algebras outside the realm of computing, which shows promise. Many theoretical developments in the field of process algebras were collected by Bergstra, Ponse, and Smolka in the Handbook of Process Algebra, published in 2001. In 2005, a workshop was organized in Bertinoro, Italy, to celebrate the first 25 years of research in the field. Several special issues of the Journal of Logic and Algebraic Programming are devoted to this workshop. Despite the importance of applications of process algebras for the success of the field, both the handbook and these special issues concentrate strongly on the theoretical achievements. This shortcoming is compensated for in a splendid way by this book, which brings together the state of the art in research on applications of process algebras. Kees Middelburg Programming Research Group University of Amsterdam, the Netherlands
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page ix 2008-10-24 #9
Acknowledgments
The editors are very grateful to those listed below who served as peer reviewers for contributions to this book. Their diligent and conscientious efforts not only helped us to make the final selection of chapters but also provided numerous valuable suggestions to the authors, resulting in a high-quality collection. Luca Aceto School of Computer Science Reykjavik University Reykjavik, Iceland Lorenzo Bettini Dipartimento di Informatica Universita’ di Torino Torino, Italy Tommaso Bolognesi CNR—Istituto di Scienza e Tecnologie dell’Informazione “A. Faedo” Pisa, Italy Gerhard Chroust Systems Engineering and Automation Institute of Systems Sciences Johannes Kepler University of Linz Linz, Austria Philippe Clauss Scientific Parallel Computing and Imaging Universit´e Louis Pasteur Strasbourg, France Pedro R. D’Argenio Department of Computer Science Facultad de Matem´atica, Astronomia y Fisica Universidad Nacional de C´ordoba—CONICET C´ordoba, Argentina
John Derrick Department of Computer Science University of Sheffield Sheffield, South Yorkshire, United Kingdom Ga´etan Hains Laboratoire d’Algorithmique, Complexit´e et Logique University of Paris-Est Cr´eteil, France and SAP Labs France Mougins, France Michael G. Hinchey Lero—The Irish Software Engineering Research Centre University of Limerick Limerick, Ireland Thomas John Department of Information Systems Vienna University of Economics and Business Administration Vienna, Austria Kenneth B. Kent Faculty of Computer Science University of New Brunswick Fredericton, New Brunswick, Canada
ix
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page x 2008-10-24 #10
x
Acknowledgments
Felix M¨odritscher Institute for Information Systems and New Media Vienna University of Economics and Business Administration Vienna, Austria Raymond Nickson School of Mathematics, Statistics, and Computer Science Victoria University of Wellington Wellington, New Zealand
Gwen Salaun ¨ Department of Computer Science University of M´alaga M´alaga, Spain Emil Sekerinski Department of Computing and Software McMaster University Hamilton, Ontario, Canada
Frederic Peschanski Laboratoire d’Informatique de Paris 6 UPMC Paris Universitas Paris, France
Kenneth J. Turner Department of Computing Science and Mathematics University of Stirling Scotland, United Kingdom
Luigi Romano Department of Technology University of Naples Parthenope Naples, Italy
Huibiao Zhu Software Engineering Institute East China Normal University Shanghai, China
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xi 2008-10-24 #11
Introduction
Parallel processing is a rich and rapidly growing field of interest in computer science. Its spectrum ranges from small-scale, fine-grained multithreaded parallelism on single- and multicore processors to coarse-grained parallel execution on large, geographically dispersed distributed systems. With the rapid commoditization of multicore computers, developers are increasingly eager to exploit those multiple cores. Yet, this requires them to adopt parallel execution models that in turn need to explicitly address the underlying concurrency issues. At the other end of the spectrum are applications that are spread out over a network of computers. They may be tightly coupled, such as for multiple computers cooperating on a single application in a high-performance cluster. Or, they may be loosely organized, as for mobile agents or service-oriented architectures (SOA). Regardless of the scale, a common requirement in parallel execution models is for carrying out interprocess communication and synchronization. Programmers know—or soon learn—that when this requirement is handled carelessly, a host of unpleasant failure modes tend to manifest: deadlocks, unrepeatable errors from race conditions, along with corruption of shared data. One root cause is that the usual tools that programmers know best—mutexes, semaphores, condition variables, monitors, and message-passing protocols—are difficult to use correctly. Ad hoc design is common, and too often, success seems like a matter of good luck. In reality, many programmers new to parallel programming have an insufficient theoretical basis for what they are trying to do. Meanwhile, designers have learned that modeling is extremely helpful for all phases of system development—from understanding the requirements to expressing the specifications, and for systematically, even automatically, deriving an implementation “correct by construction” from the model. Yet, most popular modeling tools do not seem to give sufficient guidance for programming in concurrent environments. Help is coming forth from computer scientists, who are fond of developing rigorous and logical ways of thinking and reasoning about computing. The purpose of this book is to show how one formal method of reasoning—that of process algebra— has become a powerful tool for solving design and implementation challenges of concurrent systems. Its power stems from providing a sound theoretical basis for concurrency, along with a formal notation that—in contrast to popular modeling techniques—is unambiguous. The price for obtaining the benefits is the necessity of coping with mathematical notation, symbols, and equations. Admittedly, this prospect may appear daunting to conventionally trained developers, who likely feel more comfortable drawing UML diagrams than puzzling out obscure equations. Fortunately, practitioners of process algebraic techniques are building domain-specific languages
xi
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xii 2008-10-24 #12
xii
Introduction
and application-specific tools that can be utilized by developers who then need to know less about the underlying theoretical elements. In such tools, the process algebra may be largely kept under the hood. This book is not intended as a tutorial in process algebras, but there is space here for a brief orientation: Just as with the “algebra” we all learned in grade school, its ingredients include symbols standing for constant values and for variables, and operators that act on the symbols. While elementary algebra is concerned with manipulating numbers, a process algebra—or the synonymous term process calculus—is concerned with the creation, life, and death of processes that carry out computations. The emphasis is on interactions of processes among each other and with their environment. Symbols are used to stand for individual actions or events that a process may engage in; the processes themselves; channels, an abstraction used for interprocess communication; and data transmitted over the channels. Operators specify a sequential ordering of events; a choice between several events; if/then decisions; looping or recursive invocation of processes; composition of processes so that they execute concurrently, either synchronizing cooperatively on specified events, or running independently; and syntax for parameterizing and renaming so that process definitions can be reused in a modular fashion. Practitioners can use one of the classical process algebras mentioned in the Foreword—CCS, CSP, and ACP—or a more recent one such as the pi-calculus. Established process algebras have the advantage of the availability of automated tools that can be used for exploring a specification’s state space and proving properties such as absence of deadlock (see Section 9.1 for a good explanation of these provable properties). Alternatively, practitioners can extend any of the base algebras by adding new symbols and operators, or even invent a new process algebra from scratch. A key advantage of formal, mathematical notations with rigorous semantics is that one can prove that particular conditions such as deadlock states do or do not exist, not just hope for the best, or discover them at runtime. While the notion of process algebras goes back over two decades, what is new is the rapid proliferation of parallel computing environments that need their help in transforming sequential programming models to new ones better suited to parallelism. The urgent necessity for reliably exploiting concurrent computing resources has caused researchers to press classical process algebras into practical service, along with their invention of new ones. This book is intended as a showcase for recent applications of process algebras by current researchers from diverse parts of the international computer science community. While these contributions may appear largely theoretical due to the quantities of symbols and formulas, they are, in reality, process algebras applied to specific problems. The formalism is needed to establish the soundness of the theoretical basis, and to prove that the resulting tools are properly derived. This book will be of special interest to students of process algebras, to practitioners who are applying process algebras, and to developers who are looking for fresh approaches to software engineering in the face of concurrency. The chapters are worth studying from two perspectives: first, those who identify with the problem
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xiii 2008-10-24 #13
Introduction
xiii
domains (e.g., middleware systems or multicore programming) may ask, “Are the authors doing something that I could use, or can I adapt their approach?” and second, those who are interested in process algebras as a tool can ask, “How did the authors use a process algebra in their solution? What role did it play? How did they formally define it, and what did they prove in order to give it a sound basis?” A common pattern is to first create a process algebra whose symbols and operators are tailored to the problem domain (e.g., mobile agents). Next, a virtual (or abstract) machine (VM) is defined that executes specifications in the process algebra. Because both are formally defined, it is possible to prove that the VM is correct with respect to the algebra. The last step is to implement the VM, and this may be done by transferring it into a language that is semantically close, e.g., a functional programming language. Since the target language already possesses a compiler and/or runtime framework, the job of implementation is done, at least for prototyping and demonstration purposes. The applied nature of the contributions is emphasized by organizing them into three target areas: Part I Parallel Programming, Part II Distributed Systems, and Part III Embedded Systems. We will now introduce each section and the chapters within it. Part I Parallel Programming The specific problem in view here is how to parallelize an algorithm, so as to take advantage of, say, multiple processor cores. In an ideal world, parallelization would be accomplished automatically, perhaps by compilers that are able to detect implicit parallelism in source code and generate instructions for concurrent threads on their own. Bearing in mind that modern processors already do this, in effect, with instruction streams—selecting independent instructions for out-of-order execution by multiple logic units—it may seem surprising that compilers have largely failed to match this at the source code level. But processors are detecting and exploiting implicit fine-grained parallelism. In contrast, fine-grained parallelism in software algorithms is often not worth exploiting, since there are significant overheads in spawning, managing, and collecting results from concurrent threads, plus potential bottlenecks for access to shared data. Furthermore, automatically identifying higher-level, coarsegrained parallelism implicit in source code, while desirable from an efficiency perspective, has proven to be a very challenging problem. Therefore, in the real world, parallelization is still done on a best effort basis by hand, and then it becomes a question of ensuring that a parallel version is truly equivalent to the serial version. Our contributors have developed formal methodologies for achieving precisely that result. In Chapter 1, Anand and Kahl address programming the Cell BE (Broadband Engine) processor from Sony, Toshiba, and IBM. Its heterogeneous multicore architecture features eight Synergistic Processor Units (SPUs) with their own local storage on a token ring under the control of a general-purpose Power Processor Element (PPE) core. The SPUs are intended to act as coprocessors for the PowerPC, being loaded on-the-fly with instructions and data, and coordinating via signals. The authors’
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xiv 2008-10-24 #14
xiv
Introduction
Coconut tool set provides the means to take an algorithm written in a domain-specific language (DSL) embedded in Haskell, and parallelize it for the Cell BE. The key to their approach is utilizing a graph-based internal representation of the program’s data and control flows, which are then targeted to a VM that deals strictly with concurrency issues, e.g., data transfers and interprocessor signaling. The programmer manipulates the graph to create a high-performance schedule on the eight SPUs, with the authors’ tool being used to verify that the scheduled version is correct, i.e., independent of a parallel execution order. The role of process algebra in this approach is to define the VM language, and then carry out correctness verification. While currently targeted to the Cell BE, their approach can potentially be ported to other multicore platforms. In Chapter 2, Loidl et al. take an approach similar to that in Chapter 1 in that they also develop a runtime environment, Glasgow parallel Haskell (GpH), that can be ported to different parallel platforms, and they also focus on a functional programming language. Rather than attempting to automatically extract parallelism from GpH source code, they allow programmers to insert “par” and “seq” constructs into a program to give “semi-explicit” direction while carrying out successive steps of refinement to a parallel version. Their runtime environment, GUM, has been ported to a number of different parallel platforms. These two chapters describe tools that are currently being used to program parallel systems. In comparison, Chapter 3 is more theoretical. Degerlund and Sere present an approach to taking algorithms described using another formal model called action systems, and developing an equivalent parallel version useful for scientific computing. An action system is specified using a process algebra called refinement calculus. The steps of refinement are used to introduce parallelism into the action system, with execution on a parallel target platform in view. The formal semantics of the refinement calculus ensure that the transformations are correct. Part II Distributed Systems Process algebras find their natural application in terms of formally modeling and verifying the behavior of distributed systems. Distributed systems are quite diverse, and this section also has the largest number of chapters. We start with the work of Groote et al. in Chapter 4, who have developed a process algebra, mCRL2, which is specifically targeted at distributed applications. Its provision for local communication scope (i.e., restricted to a hierarchy of processes), as opposed to purely global scope, makes it useful for describing component-based architectures. mCRL2 also accommodates true concurrency “multiactions” distinct from interprocess communication, and supports the specification of abstract data types and action times. The authors have built tools capable of analyzing properties and simulating applications specified using mCRL2. This chapter has examples of visualizing the state space of a specification by means of a generated graphic. One category of distributed systems that is currently gaining attention is the SOA that enables a business process to be automated by invoking software components located across a network. However, business processes and services described in words are subject to misinterpretations, leading to errors in integrating SOAs. A key
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xv 2008-10-24 #15
Introduction
xv
area for formalization is turning prose descriptions into unambiguous specifications. Chapters 5 and 6 make contributions in this area. Chapter 5 shows Nestmann and Puhlmann using an existing process algebra, the pi-calculus, to formally specify business processes. Their approach allows business processes to be captured in the form of Business Process Modeling Notation (BPMN) process graphs—whose nodes specify interactions such as parallel split, synchronizing merge, exclusive choice, and others—and then converted to pi-calculus agents enhanced by the authors’ “trio” construct. Interactions of business processes and services can be formally modeled, and the models analyzed for various soundness properties. The authors have created a tool to automate the property analysis. In Chapter 6, Rosa uses an ISO-standard process algebra, LOTOS, to formalize the construction of middleware systems. Each architectural component is specified as a single LOTOS process, which defines the component’s structure—the ports available to connect to other components—and behavior in process-algebraic terms. The use of a formal notation as an architecture description language, in contrast to ambiguous prose descriptions, aids both would-be service providers and service integrators, and makes it possible to prove temporal properties of an architecture. The author further employs this technique to create a library of abstract message-oriented components for use in defining middleware. A third demonstration formalizes middleware for wireless sensor networks. Another manifestation of distributed systems is based on the notion of mobile agents moving around a network. The purpose is to send software to the data rather than pulling the data down to a computation node. As argued in Chapter 8, mobility helps to minimize the impact of two problems common to distributed systems: network latency and network failure. Systems based on mobile agents will also benefit from formal descriptions, especially when it comes to ensuring security. Chapters 7 and 8 deal with mobility. As in Chapters 1 and 2, the authors of Chapters 7 and 8 take the approach of formally defining a VM. In Chapter 7, similar to Chapters 5 and 6, Paulino targets SOA and middleware. His VM, service-oriented mobility abstract machine, is based on an extension of the pi-calculus. By programming for the VM, programmers can be abstracted from the details of the network while still utilizing mobility. The author’s strategy for deployment is to execute the machine on network nodes running an existing framework called DiTyCO. In Chapter 8, Phillips presents his Channel Ambient (CA) calculus for specifying mobile applications. The textual notation also has a helpful graphical counterpart. Based on the abstract notion of an “ambient”—which may stand for a machine, a mobile agent, or a software module—the CA calculus provides operations whereby ambients may interact, and migrate in and out of each other, via channels. Security properties can be verified for a given specification. The execution target is called the Channel Ambient Machine (CAM), and its correctness with respect to the CA calculus is proven by the author. His implementation strategy is to map the CAM to a functional programming language, OCaml, that can then be executed on network nodes.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xvi 2008-10-24 #16
xvi
Introduction
Part III Embedded Systems While an exact definition of an embedded system is debatable, it is generally described as a product that contains within it some combination of software running on a general- or special-purpose processor, plus associated custom digital logic. The latter is commonly used to accelerate specific portions of calculations, which would otherwise require a more expensive processor in order to meet timing constraints. The term “embedded” refers to the fact that the end user is not necessarily aware that the product contains a computer, and, in any case, cannot utilize the computer for some other purpose. Embedded systems have special design constraints because, unlike software for desktop or server computers, the products have a significant recurring cost, such as parts, assembly, packaging, labor, etc., in addition to the nonrecurring engineering cost that goes into developing and interfacing the software and hardware components. Furthermore, they may have to meet rigorous requirements of power consumption (battery life), size, and weight. Cell phones and digital cameras are common examples. Robotic devices, such as autonomous vacuum cleaners, are embedded systems. Some are safety-critical, such as an antilock brake controller. The aim of choosing the best combination of hardware and software is to meet all the performance requirements at the lowest manufacturing cost, i.e., offering a sufficient set of features at a competitive price. Parallelism in embedded systems comes in several forms: embedded devices are often designed in terms of concurrent threads, some monitoring sensor inputs, others computing outputs, or actuating control outputs. Some have multiple processors, which may be heterogeneous, such as a 16-bit microcontroller plus a DSP, and some have hardware/software concurrency. They are often designed and marketed in a family of related products with more/fewer features, or adding features over time, or that utilize differing HW technology more favorable to different production quantities. Formal methods are very attractive for ensuring reliability, especially for safetycritical products, and those in hard-to-access locations. In Chapter 9, McEwan leads off with a contribution targeted at formally deriving a hardware implementation, although his technique is also applicable to software. As with Chapter 3, which combines the state-based formalism of action systems with the event-based refinement calculus, McEwan combines state-based Z with the event-based process algebra CSP, in a formal methodology called Circus. Z is used to provide a formal model for the data. Refinement toward an implementation proceeds by applying laws that safely inject parallelism into a sequential specification. The target language, Handel-C, used to synthesize digital logic, is close to CSP. Circus is flexible enough to allow engineering choices to guide the refinement, while still ensuring that the resulting implementation is correct. Typical parallel systems leave the scheduling of multiple processes to the operating system under the assumption of adequate CPU time and memory. Accordingly, many process algebras used to specify parallel systems do not have visibility into process scheduling, yet real-time embedded systems must guarantee responses within certain time constraints, and do so in the context of limited resources (chiefly CPU time and memory). Formal notations to specify resource requirements such as timing constraints are therefore of great utility in the embedded domain.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xvii 2008-10-24 #17
Introduction
xvii
In Chapter 10, Mousavi et al. address this problem by developing a pair of process algebras, together called PARS: one to specify processes, and another to specify scheduler behavior. The two, combined, produce a scheduled system. The final chapter, Chapter 11, considers embedded systems through the lens of product lines, in which reuse of concurrent artifacts across different hardware platforms is emphasized. Yovine’s solution is to use an algebraic language, FXML, to specify concurrent behavior, associated control and data dependencies, and timing constraints. FXML specifications are then processed by a software synthesis tool, Jahuel, to yield an implementation customized for a given platform. Platforms may differ in the means by which concurrency is supported and interprocess synchronization and communication are carried out. FXML can also be used in a design automation toolset as an intermediate specification with formal semantics. Moreover, one can define a translation of a nonformal language into FXML, thus giving the language a formal semantics and opening up the possibility of verification. We would be remiss not to acknowledge a “dark cloud” in the picture: the challenge of scaling up some of these techniques for industrial-sized applications. Such specifications may have so many states that automated verification tools cannot reach them all in reasonable time. This drawback may not be evident from these chapters themselves, since the authors were forced to use small examples, both for the sake of clarity and due to space limitations. Yet some claim to have applied their techniques to larger-scale problems. For more details, consult the references to the authors’ own related work, and they will be pleased to answer queries. The process of collecting these chapters has highlighted for us, as educators, that colleges and universities have further to go in training undergraduates to be comfortable with concurrency and skilled in reliable methods of parallel programming. More exposure to formal methods is also needed, so that these approaches do not appear so foreign. As it is, in most software development curriculums, particularly in North America, formal methods are more “honored in the breach” than by systematic instruction. At the same time, researchers will do well to embed their process algebraic techniques into tools suitable for users without special training, in order to widen their prospects for adoption. To conclude, we offer this book as an early collection of research fruits in what will undoubtedly become a burgeoning growth industry in the coming years, and to which the editors are eager to contribute their own research.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xviii 2008-10-24 #18
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xix 2008-10-24 #19
Editors Michael Alexander holds degrees in electrical engineering (Technologisches Gewerbemuseum [TGM]), business administration (University of Southern California), and economics (University of Vienna). He is a supervisor in the Vienna, Austria KPMG IT Advisory practice. His experience includes teaching and research at Wirtschaftsuniversit¨at Wien; as a lecturer and product management at IBM, Siemens, Nortel Networks, and Alcatel; as a product line manager for ADSL and Optical Access Networks. He has authored a textbook on networks and network security [1] published by H¨uthig/Verlagsgruppe S¨uddeutsche, and has edited a special issue on mathematical methods in network management of the Wiley International Journal of Network Management. For the last four years, Dr. Alexander has served as the program committee chair for the Workshop on Virtualization in HighPerformance Cluster and Grid Computing (VHPC). His current research interests include computer networking, formal methods, network management, concurrency, payment systems, machine learning methods in operations research, and e-learning. William Gardner received his bachelor’s degree in computer science from the Massachusetts Institute of Technology. Subsequently, he pursued his career in software engineering at Litton Systems (Canada) Ltd., Toronto, working primarily on embedded systems for defense customers. Returning to graduate school later in life, he did his doctoral research in the VLSI Design and Test Group in the Department of Computer Science at the University of Victoria, British Columbia, Canada. His interests are chiefly in design automation starting from formal specifications, particularly targeting embedded systems via hardware/software codesign. His software synthesis tool CSP++ [2] translates formal CSP specifications into executable C++. Its design flow features “selective formalism,” where a system’s control backbone is specified and verified in CSP, and then the principal functionality is provided by plugging C++ user-coded functions into the backbone, to be actuated by CSP events and channel communications. He was formerly on the faculty of Trinity Western University, British Columbia, Canada, and moved in 2002 to the University of Guelph, Ontario, Canada, where he is currently an associate professor in the Department of Computing and Information Science. With his graduate students, he has been participating in the R2D2C project at the NASA Goddard Space Flight Center, which involves automatic code generation from scenarios via under-the-hood formal specifications [3]. Dr. Gardner teaches courses on software development, operating systems, embedded systems, and hardware/software codesign.
xix
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xx 2008-10-24 #20
xx
Editors
References
[1] M. Alexander, Netzwerke und Netzwerksicherheit—Das Lehrbuch, H¨uthig Telekommunikation, Heidelberg, 2006. [2] W.B. Gardner, Converging CSP specifications and C++ programming via selective formalism, ACM Transactions on Embedded Computing Systems (TECS), Special Issue on Models and Methodologies for Co-Design of Embedded Systems, 4(2), 302–330, May 2005. [3] J. Carter and W.B. Gardner, Converting scenarios to CSP traces with Mise en Scene for requirements-based programming, Innovations in Systems and Software Engineering, 4(1), April 2008.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xxi 2008-10-24 #21
Contributors
Abdallah Al Zain School of Mathematical and Computer Sciences Heriot-Watt University Edinburgh, United Kingdom
Fredrik Degerlund Turku Centre for Computer Science ˚ Akademi University Abo Turku, Finland
Christopher Kumar Anand Department of Computing and Software McMaster University Hamilton, Ontario, Canada
and
Ismail Assayad VERIMAG Grenoble, France Clem Baker-Finch Department of Computer Science Australian National University Canberra, Australia Twan Basten Department of Electrical Engineering Eindhoven University of Technology Eindhoven, the Netherlands Ananda Basu VERIMAG Grenoble, France Michel Chaudron Leiden Institute of Advanced Computer Science Leiden University Eindhoven, the Netherlands Francois-Xavier Defaut VERIMAG Grenoble, France
Department of Information Technologies ˚ Akademi University Abo Turku, Finland Jan Friso Groote Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands Kevin Hammond School of Computer Science University of St Andrews St Andrews, Scotland United Kingdom Wolfram Kahl Department of Computing and Software McMaster University Hamilton, Ontario, Canada Hans-Wolfgang Loidl Institut f¨ur Informatic Ludwig-Maximilians-Universit¨at Munich, Germany
xxi
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C000 Finals Page xxii 2008-10-24 #22
xxii Aad Mathijssen Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands Alistair A. McEwan Department of Engineering University of Leicester Leicester, United Kingdom MohammadReza Mousavi Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands Uwe Nestmann School of Electrical Engineering and Computer Science Berlin Institute of Technology Berlin, Germany Herv´e Paulino Computer Science Department New University of Lisbon Lisbon, Portugal Andrew Phillips Microsoft Research Cambridge, United Kingdom Frank Puhlmann Hasso-Plattner-Institut Potsdam, Germany Michel A. Reniers Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands
Contributors Nelson Souto Rosa Centre of Informatics Federal University of Pernambuco Recife, Brazil Kaisa Sere Department of Information Technologies ˚ Akademi University Abo Turku, Finland Phil Trinder School of Mathematical and Computer Sciences Heriot-Watt University Edinburgh, Scotland, United Kingdom Yaroslav S. Usenko Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands Muck van Weerdenburg Department of Mathematics and Computer Science Eindhoven University of Technology Eindhoven, the Netherlands Sergio Yovine VERIMAG Grenoble, France Marcelo Zanconi VERIMAG Grenoble, France
Alexander/Process Algebra for Parallel and Distributed Processing C6486 S001 Finals Page 1 2008-10-15 #2
Part I
Parallel Programming
Alexander/Process Algebra for Parallel and Distributed Processing C6486 S001 Finals Page 2 2008-10-15 #3
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 3 2008-10-20 #3
Chapter 1 Synthesizing and Verifying Multicore Parallelism in Categories of Nested Code Graphs Christopher Kumar Anand and Wolfram Kahl Contents 1.1
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Cell BE Architecture and the Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . 1.1.3 Background on Coconut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Nested Code Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Data-Flow Graphs and Parallelism. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Nonconcurrent Control-Flow Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Control-Flow Graphs and Concurrency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Concurrent Interpretation of Data-Flow Graphs . . . . . . . . . . . . . . . . . . . . . . 1.3 Efficient Multicore Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Motivation for Multicore Virtual-Machine Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Virtual-Machine Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.1 Virtual-Machine Instruction List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Scheduling Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.1 Payload Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.2 Payload Buffer Allocation + Communication Generation . . . . . . . . . . . . 1.6.3 Communication Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Concurrency Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.1 Motivating Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.2 Strictly Forward Inspection of Partial Order . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.3 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7.4 Proof of Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 5 6 7 8 8 12 12 14 14 15 17 18 21 24 27 27 27 28 30 31 32 39 41 42 42
3
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 4 2008-10-20 #4
4
Process Algebra for Parallel and Distributed Processing
1.1 Introduction Current trends in high-performance processor technology imply that the programming interface is becoming more complicated. The pronounced shift toward pervasive Single Instruction Multiple Data (SIMD) parallelism, and the introduction of heterogeneous computing elements, including hierarchically distributed memory (encompassing conventional distributed memory, streaming memory, software-directed caches, and private memories), will require changes in software development tools. Among the first generation of such processors, the Cell Broadband Engine (BE) is the most widely deployed. Using its resources efficiently requires explicit synchronisation of both code and data, and packaging of both to fit in small memory spaces and constrained communication infrastructures. The COde CONstructing User Tool (Coconut) project has the aim of capturing the entire design and development process for safety-critical, high-performance scientific software, e.g., medical imaging, in a sequence of formal specifications. The hardware-specific parts of Coconut currently target a single Cell BE. See Section 1.1, for background on the Cell BE and Coconut. This chapter describes the support Coconut provides for multicore parallelism, including the way that concurrent control flow is represented in nested hypergraphs, the way multicore issues are abstracted into a virtual machine, and the support Coconut provides for developers of parallelization patterns. This support includes a linear-time algorithm to verify the soundness of a parallelization. For the developer, verification eliminates the need for time-consuming parallel debugging, and the analogy between the virtual machine and a superscalar CPU provides a familiar way of thinking about scheduling. The key to being able to provide an efficient verification tool is the abstraction provided by modeling multicore communication as a simple process algebra. Simplicity is justified because parallelism is not an inherent part of algorithm specifications in the domain of scientific computation, it is only added to implementations to meet performance targets, so the process algebra only needs to model the patterns of communication which support this. Adding additional features to support general control flow would significantly complicate verification, and could be used by developers to implement program logic, making programs much harder to understand, and thereby reducing programmer productivity. The Coconut tool set uses as intermediate representations of programs and program fragments, a variant of term hypergraphs we call “code graphs.” Term graphs have a long history as an efficient implementation of symbolic expressions. A certain kind of term graph also provides a natural model of a resource-conscious kind of categories (gs-monoidal categories, [8]); this provides a useful justification of many kinds of algebraic data-flow graph transformations. To combine control flow and data flow, we nest the simple data-flow code graphs inside an outer-level code graph with control-flow semantics, similar to the two different semantics of flownomials [19]. Software pipelining, i.e., instruction reordering to make better use of instruction pipelines and multiple-issue queues, is an algebraic transformation involving both levels of such nested code graphs.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 5 2008-10-20 #5
Synthesizing and Verifying Multicore Parallelism
5
This approach naturally extends to modeling (restricted) concurrency: We introduce a third, outer layer for explicit concurrency, with a semantics more on the dataflow side (this is related with the implicit concurrency in pure functional programs), but again with some control-flow characteristics. Algebraic transformations crossing the outer two levels in such three-level nested hypergraphs are used to map potential concurrency to explicit concurrency, and to hardware parallelism. On the level of multicore processors, this will involve the automatic introduction of appropriate synchronization instructions or, in the opposite direction, the elimination of such concurrency primitives for the purpose of verifying against a concurrency-independent specification. This chapter presents the control flow at the middle multicore level. In conventional programs, this level of control flow is usually encoded in library calls (e.g., MPI [33,38]) in the program text. Our approach is to make this disjoint in a separate level of the graph. This allows for a simple verification of correctness for deeply pipelined data-flow graphs through a network of cores (processors). As a result, whether multicore parallelism is directed by user code, user-code generators, or automatic parallelization, the programmer/compiler writer can focus on tuning for performance, and check correctness in a computation which grows linearly with program size, and quadratically in the complexity of the hardware. The distinguishing feature of code graphs at this level is that they are presented as ordered lists of instructions. Proofs of sound parallelization work by going through the list in order, maintaining a limited set of state information (status of signaling and data transfers, contents of buffers). Any ambiguity (from a race condition) or deadlock can be flagged in the presented program.
1.1.1 Related Work This work is the first step in bringing together disconnected research efforts in (1) verification of concurrent programs, (2) scheduling for distributed processors, and (3) decomposition of large numerical problems. Practical parallel computation is a special subset of the set of concurrent programs. We have defined a precise subset of the vaguely defined set “parallel computation.” Verification of correctness in this subset should be simpler than in the larger set, but many important ideas come from efforts to verify general concurrent programs. We make fundamental use of partial orders in analyzing concurrency within our programs. Godefroid and several collaborators have demonstrated that partial order methods can be much more efficient than other methods for general concurrent problems [16,20]. In the language of Ref. [16], we construct a partial order in such a way as to show that there are no backtracking points, i.e., no points where a different execution sequence would have led to a different result. Our implementation is much simpler, with tight complexity and memory bounds, because it is tailored to the restricted set of programs we consider. The trade-off is that the programs we are able to consider are orders of magnitude larger than typical concurrency benchmarks. We target the Cell BE processor, using matrix multiplication as our first benchmark, as do [5]. Our approach differs in that we present the programmer with an
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 6 2008-10-20 #6
6
Process Algebra for Parallel and Distributed Processing
unconventional interface via domain-specific languages (DSLs) embedded in Haskell. DSLs [14] can be a useful tool for making special-purpose software development accessible to domain experts; Ref. [13] and, in particular, Hudak [25,26] showed that embedding DSLs in Haskell has significant advantages both for implementability and for usability. There is a lot of literature related to loop decomposition, and specifically the decomposition of large operations in linear algebra. The generation of consumers used in our scheduling algorithm implicitly encodes a loop tiling (see Refs. [21,22]). Research in this direction is currently considering much more complex computational examples than we are able to at this time, and will provide a rich set of benchmarks for the future. That said, some of the work on tiling will have to be reconsidered in light of the shift from computational to main-memory bottlenecks. We hope that our tool will accelerate the development of patterns of communication involving many point-to-point pipelined transfers.
1.1.2 Cell BE Architecture and the Virtual Machine The Cell BE architecture exposes both computational and data-flow resources to user-level programming. In Figure 1.1, one sees the main computational resources. The synergistic processing units (SPUs) do not have direct access to the multilayer token ring, but must go through the attached memory flow controller (MFC), which operates most efficiently when executing reorderable lists of direct memory access (DMA) instructions. The PowerPC Unit (PPU) is capable of general computation, but has much lower performance than the SPUs. The initial implementations of the Cell BE include interfaces to extend the ring architecture to multichip rings or switched networks [7]. In implementing the approach to concurrent control flow presented in this chapter, the PowerPC (PPC) core manages run-time start up (address resolution, code loading, etc.). Our virtual machine model was designed with the Cell BE in mind, but could easily be adapted to other architectures. If an architecture does not support signals,
SPU LS
SPU LS
SPU LS
SPU LS
MFC
MFC
MFC
MFC
PPU
Ring extension or I/O
2-Way arbitrated ring Memory interface
MFC
MFC
MFC
MFC
SPU LS
SPU LS
SPU LS
SPU LS
FIGURE 1.1: Bus diagram for Cell BE.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 7 2008-10-20 #7
Synthesizing and Verifying Multicore Parallelism
7
they could be emulated using data transfers. Just as object code optimized for one generation of a processor family may run poorly on the next generation, the performance of a generated program will depend on it being optimized for the right implementation of the virtual machine. If signals are not supported, some patterns of communication may become more expensive relative to other patterns. The current Cell BE is a tightly coupled single-chip device. For this reason, fault tolerance is not designed into the virtual machine, and efficiently handling partial equipment failure would require extensions to the virtual machine model.
1.1.3 Background on Coconut Coconut is architected with both long- and short-term correctness requirements. In the long-term, we believe that safety-critical applications like medical imaging should come with meaningful correctness guarantees. Meaningful, to the patient, means that the image which comes out is proven to be the result of applying algorithms with known specifications and understandable physical inputs. Algorithms can be incorrect, but they should be specified by experts in the field. No low-level optimizations should be taken without floating up their consequences to the algorithm level for approval. Specifications at this level must include concrete anchors in the physical universe, including not just units, but oriented coordinate systems, etc. High level User Specification of Constraints (HUSC) [40] is a prototype including special implementations of type inferencing and alternatives suited to these applications. Current practice starts with an informal specification of the algorithm, possibly supplemented with a partial implementation in a mathematical scripting language. This specification is translated into different types of low-level specifications and finally implemented in an imperative language, possibly including assembly language. The Coconut workflow involves no unverifiable human translation of specifications or manual editing of assembly code. Code optimization experts contribute by formally specifying patterns of efficient implementation as code transformations or code generators. At this time, all transformations and leaf functions are specified in Haskell. For the lowest-level patterns, which are implemented without control flow (although they may contain control flow synthesized using logical and permutation instructions) we have defined a DSL embedded in Haskell, including a number of support tools such as simulation and symbolic execution. Such tools are important not just for verifying of correctness, but for pinpointing errors in code under development. Much of the technical difficulty in optimizing code (whether by human or by compiler) is the recognition of patterns of control flow expressed using conventional patterns. The Coconut strategy for avoiding this is to provide more expressive control constructions at the language level. One of these is the MultiLoop [3]. Novel control structures require novel instruction scheduling. Explicitly Staged Software Pipelining [39] is a novel approach to scheduling loops which supports the MultiLoop. This work defines a language for concurrent control flow, and the principal tool for both debugging patterns and verifying correctness.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 8 2008-10-20 #8
8
Process Algebra for Parallel and Distributed Processing
Coconut is being implemented from the bottom up to insure that intermediate representations do not interfere with the identification of optimizable patterns of execution. One advantage of this approach is that we are able to produce high-performance C-callable single-core functions, and using the infrastructure defined here, we will be able to provide libraries which encapsulate optimal parallel execution. At this point, Coconut will be able to generate the most difficult aspects to verify in our target applications. For many applications, verification of this level of computation is enough. There is no plan to allow Coconut-generated code to call functions written in other languages, since this would make verification much more difficult. This strategy allows us to earn early dividends from the considerable investment required to build verifiably correct software at this level of performance. Coconut is implemented in Haskell, except for type inferencing in HUSC. Haskell is used for developer-productivity reasons, mostly strong and flexible typing. The current DSL embedding takes advantage of Haskell language features, and the existence of support for literate code, and these features are appreciated by novice users, but another pure functional language could have been used.
1.1.4 Overview We explain in Section 1.2 how directed hypergraph syntax naturally integrates data flow and concurrent control flow, and summarize how we use these code graphs in nested arrangements for different aspects of the generation of high-performance code. We then turn our attention to implementation aspects of multicore parallelism (Section 1.3), contrasting it with other kinds of parallelism, to provide the background for the decision (Section 1.4) to define a virtual machine for encapsulating certain low-level aspects of targeting multicore architectures. The details of this virtual machine are then presented in Section 1.5, and in Section 1.6 we show how we produce, partially guided by graph transformations, a multicore program scheduled to hide communication latencies as far as possible. To independently verify and certify the concurrency aspects of the result, we introduce, in Section 1.7, an efficient algorithm that checks satisfaction of the relevant safety and liveness properties in a single pass over the fuzzy multicore schedule.
1.2 Nested Code Graphs Graphs are frequently used as internal presentations for code generation and analysis purposes. We use code graphs as uniform graph syntax. One use of our code graphs is as data-flow graphs generalizing term graphs, and another use is as controlflow graphs generalizing finite-state machines and Petri nets. To express concurrent processes and parallel execution, we use the same code-graph syntax at different levels of nesting to combine data-flow aspects, control-flow aspects, and concurrency aspects in nested code graphs.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 9 2008-10-20 #9
Synthesizing and Verifying Multicore Parallelism + 1 0
+ ∗
2
2
∗
∗ 0 1 1 0 7 ∗ 0 1
7
+
+ ∗
5
+
+
1 0 30 ∗
3
9
1 1 3 ∗0 1 0 + 1 0 7 ∗ 1 0 + 10 2 ∗ 10 +
1 (a) x
5
(b)
5
(c)
1
FIGURE 1.2: (a) Horner term graph, (b) jungle, and (c) code graph.
Term graphs, the abstract version of data-flow graphs, are usually represented by graphs whose nodes are labeled with function symbols and edges connect function calls with their arguments [37]. We use the dual approach of directed hypergraphs where nodes are labeled with type information (if applicable), function names are hyperedge labels, and each hyperedge has a sequence of input tentacles and a sequence of output tentacles (each incident with a node). For example, Figure 1.2 shows first a conventional term graph (as presented by HOPS [27]) for calculating the polynomial 5x 3 + 3x 2 + 7x + 2 using Horner’s rule, i.e., as x ∗ (x ∗ (x ∗ 5 + 3) + 7) + 2, and second a conventional jungle with nodes drawn as black discs, and edges drawn in boxes with a result tentacle from the result node to the edge and numbered∗ argument tentacles from the edge to the argument nodes. Variables are represented by nodes that are not the result of any edge. In contrast, the tentacles of our code graphs follow the direction of the data flow, and we draw the data flow downward, as shown in the right-most drawing of Figure 1.2. In our code graphs, we also explicitly flag input nodes (corresponding to jungle variables) by attaching a numbered triangle with an arrow to the node, and output (result) nodes by attaching a numbered triangle with an arrow from the node. Such data-flow code graphs are the central data structure of the SPU scheduler of Coconut. The assembly-level data-flow code graph for the same polynomial is shown to the left in Figure 1.3, with edges containing in this case either the SPU SIMD opcode fma (fused floating-point multiply-add), or SIMD constants replicating (splatting) the same 32-bit floating-point number four times into 128-bit SPU registers. In this drawing, we show node numbers (normally for debugging purposes), ∗ Since all these code-graph drawings have been automatically generated via dot [29], these numbers
are necessary to indicate argument sequence.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 10 2008-10-20 #10
10
Process Algebra for Parallel and Distributed Processing f 32Splat 5.0
f 32Splat 3.0
2:REG 1
1
1:REG
3:REG 2
1 0
fma
f 32Splat 7.0
1qa $11, const0 5:REG
4:REG 1 2
1qa $10, const1 0
fma
f 32Splat 2.0
7:REG 2
6:REG 1
1qa $9, const2 1qa $8, const3 0
fma $10, $7, $11, $10
fma fma $9, $7, $10, $9 8:REG fma $8, $7, $9, $8 1 (a)
(b)
1
FIGURE 1.3: (a) Horner assembly-level data-flow and (b) control-flow code graphs. and indicate the node type “REG” standing for 128-bit SPU register values. Since the SPU has a unified register file, there are no further type distinctions at this level, although a refinement makes sense for the purpose of analyzing and verifying the data flow of individual register components, see Ref. [4]. To the right in Figure 1.3, we show the assembly instructions that the left graph is compiled to, presented as a control-flow code graph. Assembly instructions are opcodes together with argument and result registers, so the data flow can be reconstructed from this.∗ For this chapter, it is central that code graphs can be used for both data-flow and control-flow graphs. In the light of the above, a hypergraph in the sense of this paper is therefore directed, with sequences of input (argument) and output (result) tentacles at each edge:
∗ The constants are transformed into data segments for the assembler, and referenced by the labels
“const0”, . . . , “const3”. This is necessary since there are no SPU instructions with immediate arguments of 32 bits.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 11 2008-10-20 #11
Synthesizing and Verifying Multicore Parallelism
11
DEFINITION 1.1 A hypergraph H = (N , E, src, trg, nLab, eLab) over a node label set NLab and an edge label set ELab consists of • A set N of nodes and a set E of hyperedges (or edges) • Two functions src, trg : E → N ∗ assigning each hyperedge the sequence of its source nodes and target nodes respectively • Two functions nLab : N → NLab and eLab : E → ELab assigning labels to nodes and hyperedges Hypergraph homomorphisms are pairs consisting of a total node mapping and a total edge mapping, preserving src, trg, nLab, and eLab, and induce a category. If a hypergraph in some sense represents a program, it has an input/output interface: DEFINITION 1.2 A code graph G = (H, In, Out) over a node label set NLab and an edge label set ELab consists of • Hypergraph H = (N , E, src, trg, nLab, eLab) over NLab and ELab • Two node sequences In, Out : N ∗ containing the input nodes and output nodes of the code graph A code-graph homomorphism is a hypergraph homomorphism that additionally preserves In and Out. Typing aspects are dealt with in the usual way: DEFINITION 1.3 A hypergraph signature is a hypergraph where the labeling functions nLab and eLab are the identities on the respective label sets. A typed hypergraph is a hypergraph together with a homomorphism into a hypergraph signature. The resulting categories of (typed) hypergraphs have pushouts, and can therefore be used as basis for the double-pushout approach to graph transformation [11], where rules are induced by spans in the category of code graphs [4]. Code graphs also give rise to a gs-monoidal category with sequences of node labels as objects and code graphs as morphisms; Ref. [8] uses this to establish functorial semantics for code graphs, and Refs. [9,28] show examples where this semantics uses locally ordered categories, such that code-graph homomorphisms F : G1 → G2 document ordering relations G2 G1 on the semantics.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 12 2008-10-20 #12
12
Process Algebra for Parallel and Distributed Processing
1.2.1 Data-Flow Graphs and Parallelism Using code graphs as term graphs follows the jungle view of term graphs, initially put forward by Hoffmann and Plump [24,34], and equipped with functorial semantics by Corradini and Gadducci [8]. In such code graphs, nodes are labeled with types and in a certain sense represent values of the respective types; hyperedges are labeled with operations taking input values as arguments and producing results as outputs. Data-flow semantics of hypergraphs implies the following: • If an edge has multiple input tentacles, the computation associated with it has multiple argument positions, one for each input tentacle. This is the only kind of branching present in term trees. This kind of branching is also the source of potential parallelism, particularly in functional programming, since it represents a split of the computation of the outputs of this edge into potentially independent computation of its inputs. • If a node is attached to multiple input tentacles, the value represented by this node is used by several consumers. This sharing marks the transition from term trees to term graphs, and is frequently only a tool for efficiency (shared effort) without semantic relevance, for example, in most implementations of functional programming. (Sharing does have semantic relevance for example in functional-logic programming [1] and in multialgebras [9,10].) • If an edge has multiple output tentacles, the computation associated with it produces multiple results, one for each output tentacle. This presents a relatively unproblematic generalisation of term trees or term graphs. • Possibility that a node is attached to multiple output tentacles was used in Ref. [28] to represent joins in data flow, i.e., different equivalent ways to produce a value. Cycles in term graphs are frequently used to encode recursions; this normally uses a least fixed-point semantics and relies on the presence of nonstrict constructs, in particular conditionals, or, in lazy functional programming, data-type constructors. In the central assembly-level components of Coconut, assembly op-codes (which have strict semantics) serve as edge labels, with inputs corresponding to argument registers and outputs corresponding to result registers. We use acyclic data-flow code graphs with these op-codes as edge labels to represent loop bodies of scientific computation applications, and use graph transformation and analysis to exploit instruction-level parallelism (ILP) and achieve optimal schedules [2,4].
1.2.2 Nonconcurrent Control-Flow Graphs The archetypal control-flow graphs are finite-state machines; nodes are labeled with state types and represent states, and edges labeled with actions, sometimes
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 13 2008-10-20 #13
Synthesizing and Verifying Multicore Parallelism
13
1 1
(r,k) := (1,n)
(r,k): = (r ∗ k, k −1)
if k == 0
(a)
1
if k ! = 0
1 (b)
FIGURE 1.4: (a) Factorial control-flow graph and (b) underlying Petri net.
perceived as active (operations), sometimes as passive (recognition of input symbols), and sometimes as both (for example, in Mealy machines). For conventional edges, finite-state machines serve as the standard control-flow model, and represent nonconcurrent control flow. Cycles in control-flow graphs express iteration, and correspond to the asterate operator of Kleene algebras. In Figure 1.4, we show the control-flow graph of an imperative implementation of the factorial function presented as a code graph. Even though from a flow-chart point of view, it may seem strange to have two separate edges for the conditional, this is in fact the way such programs are handled in relational semantics (see Refs. [17,36]), and fits well with the concurrent semantics below—indeed, abstracting all edges to transitions and considering the nodes to be places produces the place-transition Petri net (drawn to the right in Figure 1.4) representing the control flow of this program. At the lowest (output) level in Coconut, assembly instructions (except conditional branches), i.e., op-codes together with register references, serve as control-flow edge labels, with the state before as single input, and the state after as single output. The conversion of op-code-labeled hyperedges to instruction-labeled control-flow edges happens as part of scheduling of the data-flow graphs; since scheduling essentially means adding of control dependencies, and scheduling of a data-flow graph together with a register allocation enforces the data-flow dependencies via the stronger control-flow dependencies. At a higher level, we represent computational kernels as sequential control-flow graphs where most of the edges are labeled with larger data-flow graphs, and implement software pipelining essentially as graph transformation on these control-flow graphs [3].
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 14 2008-10-20 #14
14
Process Algebra for Parallel and Distributed Processing
1.2.3 Control-Flow Graphs and Concurrency The standard model for concurrent control flow are Petri nets, which are, in more theoretical work (e.g., [18]), formalized as directed hypergraphs, with places represented by hypergraph nodes and transitions by hyperedges (but with multisets of input and output places for each transition; Sassone introduced the concept of prenet which has sequences instead of multisets [6,35]). Putting everything together, control-flow semantics of directed hypergraphs therefore implies the following: • If a node is attached to multiple input tentacles, it is normally understood that control propagates along any applicable outgoing edge. This kind of branching implements decisions by making different edges applicable depending on the circumstances. In finite-state automata recognizing regular languages, those edges whose labels match the next available input are applicable; more generally, conditions depending on the current state can be used, as for example “k == 0” in Figure 1.4. If several outgoing edges of the same node are applicable, then this kind of branching introduces nondeterminism. If this is the only kind of branching, the resulting graphs can be used as decision trees, or as tree structures for branching-time temporal logics [15]. • If a node is attached to multiple output tentacles, the corresponding different operations pass control to the same state. This corresponds to moving from tree structures for branching-time temporal logics to graph structures, and similarly from infinite regular languages to finite automata; in imperative programming, this is typically induced by jumps or higher-level control structures such as loops. • If an edge has multiple output tentacles, we understand this as forking the current thread of control into multiple threads of control. This corresponds to the fact that Petri net transitions with multiple outgoing edges place tokens into each successor place upon firing. • If an edge has multiple input tentacles, we understand this as introducing synchronization of concurrent threads. With this background, it becomes clear that, in the nonconcurrent case, conditionally branching assembly instructions should not be represented by branching hyperedges, but instead as two conventional edges of which in each state only one is applicable, as can be seen in the example shown in Figure 1.4.
1.2.4 Concurrent Interpretation of Data-Flow Graphs From the theoretical literature [6,18,35] one notices that Petri net computations, which are finite acyclic versions of Petri nets equipped with a start and end interface,
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 15 2008-10-20 #15
Synthesizing and Verifying Multicore Parallelism
15
give rise to essentially the same kinds of categories as term graphs, namely variants of gs-monoidal categories [8]. The description in the previous section was accordingly tuned to emphasize the compatibility between the concurrent control-flow view of a hypergraph with the data-flow view, as long as no cycles are present. Indeed, Sassone’s prenets [6], designed to reflect the “individual-token philosophy” to composition of Petri net computation, can be understood as making individual threads of control explicit. The two views combine easily into data-flow graphs with concurrency aspects, where the distribution aspects are most easily made explicit by adding host identifiers to nodes and hyperedges. For an example, see Figure 1.7. Combining this representation of concurrency aspects of distribution with the nested hypergraph representation of sequential computational kernels, we obtain a nesting of hypergraphs with at least three levels. Implementing iteration of concurrent behavior will even require wrapping the concurrent control-flow code graph at the third level into a cyclic sequential control-flow graph, which produces a four-level nesting.
1.3 Efficient Multicore Parallelism Different levels of parallelism raise different issues with respect to efficiency and correctness. It is useful to put multicore parallelism into a context categorized in order to contrast current issues with parallelism issues in the past. Distributed parallelism: It involves execution on multiple hosts, also called nodes, each running an operating system. Long and variable latencies, and network and node failures during computation may be significant issues in this context. The message-passing library MPI [33,38] is most commonly used for distributed scientific processing, typically implementing a Single Program Multiple Data (SPMD) model. The most feature-rich tools for distributed parallelism are marketed as grid computing. The SPMD approach allows programmers to write a single program which runs on multiple nodes. Each instance therefore contains code for all instances and knowledge of the data distribution across nodes. This model is not suited to heterogeneous multicore architectures with small private memories. ILP: CPUs need multiple execution units for different types of computation to reduce design complexity. ILP takes advantage of this hardware design imperative by dispatching instructions to multiple units in parallel. In the vast majority of cases, the CPU has circuitry to introduce this parallelism into a sequential stream of instructions, and may make other modifications (e.g., register renaming) to improve throughput. Using this mechanism, sequential programmers and basic compilers get some degree
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 16 2008-10-20 #16
16
Process Algebra for Parallel and Distributed Processing
of parallel execution for free, while advanced compilers can produce significant levels of parallelism (e.g., four-way on recent Power architectures). Very Long Instruction Word architectures expose this level of parallelism explicitly to the compiler. SIMD: It properly refers to architectures with hardware support for large-vector arithmetic operations. The Cray-1 was a successful computer using this model. Vector-in-register parallelism (VIRP): In current architectures, SIMD most often refers to short-vector-in-a-register architectures designed to accelerate the manipulation of digital media. Typical instructions treat a 128-bit instruction as four 32-bit integers and perform operations on them in parallel. Although some early implementations could accurately be described as SIMD, current instruction sets have added rich horizontal instructions which operate across elements in a single register values, for example, by shuffling bytes within a single register value [7]. Multicore architectures: They are evolving rapidly, nevertheless, we can assume the following characteristics: computational resources are organized into units similar to conventional CPUs, with asynchronous execution in the multiple cores, and weakly coherent access to common memory resources. The programming model for these resources differs greatly: private memory, user-directed caching of global memory, or hardware-directed caching of global memory. The models range from highest potential performance to simplest programming model. ILP has been so successful that almost no programmer considers it when writing code. Large-vector SIMD is too restricted to apply to a wide enough range of applications to make it viable. Distributed parallelism always requires significant programmer intervention, and a whole industry has grown up around providing tools to write and debug distributed parallel programs. We have previously shown that taking advantage of VIRP requires the development of new patterns of efficient execution, and that an approach marrying a DSL programmer interface to a system based on graph transformation can produce highly efficient, highly readable, and high-assurance code [2]. Our goal is to duplicate the success of ILP and VIRP at the multicore level, avoiding as many of the problems associated with distributed parallelism as possible. Current SPMD models for distributed parallelism raise the cost of developing software above what most applications can support. Run-time requirements for library and operating-system support do not scale well to large numbers of cores, and exceed the capabilities of some lightweight cores. Why is ILP universally exploited while SPMD is considered an arcaneart? Accepting that it makes sense to compare such different levels of parallelism, we believe that the answer lies in the way the two are understood (both formally and informally). SPMD is implemented as a library, while ILP is implemented as a language (machine language). SPMD can be used to express any programming construct and implemented at any scale, and most support libraries try to abstract away the scale and architecture of the underlying hardware. The level of parallelism supported is unbounded. Any conceivable program can be implemented. Unfortunately, most conceivable programs are not correct, and it is difficult in practice to tell which ones they are.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 17 2008-10-20 #17
Synthesizing and Verifying Multicore Parallelism
17
ILP, on the other hand, is defined in terms of a language (machine or assembly language). Both resources and the amount of parallelism are bounded. All resources whose efficient use determines performance are accessible to the programmer through instruction (re)scheduling. For the most common architectures, the programming model is strictly sequential within the CPU. Some architectures allow external observers to see limited nonsequential execution (by reordering external memory accesses for performance reasons). Obviously, presenting a sequential model to the programmer makes ILP easier to use. Note, however, that programmers (and more commonly compiler writers) aiming for peak efficiency must be aware of nonsequential (out-of-order) program execution. The key point is that correctness can be determined using the sequential programming model or the data-flow-graph-based model, even if significant changes in scheduling are required to efficiently use resources. Furthermore, the limited resources in support of parallel execution make it possible to characterize the set of schedules and design effective heuristics for searching within it.
1.4 Motivation for Multicore Virtual-Machine Model Our aim is to encapsulate the important features of multicore parallelism in a virtual machine (language with semantics) with the hope that techniques developed for ILP will apply to parallelism at this level. This is our attempt to find a balance, exposing enough details to enable high-performance computation while hiding enough details to make it easy for the programmer to understand program semantics. The most important feature we need to capture is data locality. The hardware required to present the programmer with a flat memory model across multiple execution threads is very expensive, and introduces performance-reducing nondeterminacy. Processors designed for signal processing have for many years provided user-directed cache modes, including directly addressable caches, in which the programmer can fix part of the cache to memory mapping in order to guarantee fast access to important data structures. Because such optimizations are nonportable and require support from the operating system, they have not been used in multiuser servers or desktop processors. The Cell SPUs [23] have no cache, but rather have private local storage. Communication with the rest of the processor must be performed explicitly using DMA transfers. This can be seen as integrating on-chip the architecture of distribution that has existed for decades on the circuit boards of workstations, where microprocessor and graphics processor units execute independently, and communicate via DMAs. Data locality poses a challenge for synchronization for both cached and noncached memory architectures. This is because access to memory which appears to be local may require distant access. In a cached architecture, this occurs when two physical cores access the same physical memory causing the bus hardware to transfer cache lines between the two processors, introducing delays. To avoid this, our language does not contain access to nonlocal memory, even as part of interprocess signaling, which implies that we do not use mutexes.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 18 2008-10-20 #18
18
Process Algebra for Parallel and Distributed Processing
Our virtual machine must contain multiple threads of execution corresponding to individual cores. Each thread has a small list of local resources: data buffer to store blocks of data for computation and signal registers which can be set by remote processes. These are the registers of our virtual CPU. The former usually contain values visible at the high level, like registers containing the results of arithmetic. Signal registers are only visible to the parallelizer, just as registers for floating point exceptions and PowerPC condition registers are only visible to the compiler or lowlevel assembly language coder. The execution of procedures implementing register assignments based on pure,∗ local functions is the arithmetic execution unit of our virtual CPU, and asynchronous communication is the data movement execution unit.† We can synthesize branching control flow inside the pure functions just as we did for VIRP, and we can move it to a higher level in the Coconut code graph. Restricting computation to pure functions, and control flow to thread synchronization, significantly simplifies the process of reasoning about parallelization for the programmer or compiler and the verifier. By separating control and data flow, this allows a single-threaded control process which may be distant from some or all of the computation engines to originate the complex data-flow patterns necessary in scaling up multicore computations, while separating algorithm-intrinsic control flow from control flow required by parallel computation. In this way, the algorithm designer can solve one problem at a time: writing a correct algorithm and writing an efficient parallelization strategy. Unlike ILP and like SPMD, to tune performance, the programmer must explicitly insert parallelization, although, in Coconut, this is done in program generators, not programs, and can be simplified by using higher-level patterns. Like ILP and unlike SPMD, verification ensures that the program as a whole is a pure function, i.e., independent of parallel execution order.
1.5 Virtual-Machine Language A sequence of computations is a list of pairs (c, i) of execution core identifiers and instructions. In the present description, the instructions on each core execute in order, although just as in the ILP case, it would be easy to introduce limited out-of-order execution managed by the run-time system or dedicated hardware.
∗ Pure functions are functions without side effects, which here, in particular, includes no invocations of
communication primitives, and implies referential transparency. The latter means that results depend only on the explicitly provided arguments, and no other circumstances, in particular, not on execution order of subcomputations in cases where this order is not fully determined. † Although a single communication primitive would be sufficient, we separate data flow and signals because machine architectures can (and should) optimize one for throughput and the other for latency. On the Cell BE, signals are implemented as both signals and messages, and data flow as DMA transfers.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 19 2008-10-20 #19
Synthesizing and Verifying Multicore Parallelism
19
Data movement instructions, however, are assumed to run asynchronously from other computations, so the exposed instructions only initiate data movement/signaling or wait for it to complete. DMA engines to handle DMA are almost always implemented as autonomous processors executing transfer lists. We embed the data movement with the purely local computations on a core because this is analogous to the interleaving of load/store instructions and register-based computation on a Reduced Instruction Set Computer (RISC). Unlike load/store instructions in a RISC CPU, we have to separate the request for transfer from the acknowledgment of completion—in RISC processors, issuing the instruction using the target register of a load instruction will stall the execution pipeline until the data have been transferred, while DMA engines do not provide equivalent synchronization mechanisms. We choose to expose this for several reasons. Signaling may compete with data transfers on a common bus, and this allows the instruction scheduler to take this into account. Data transfers themselves introduce ordering, so signaling may not always be necessary. The separation makes it easier to maximize latency hiding for twoway communication within given resource constraints. Each transmission executes asynchronously with a variable latency, so it is useful to separate them enough to hide the expected latency. While computation within a core is in order, computation on different cores is a priori unordered, as are all data transfers. For this to work, the computational threads on the send and receive sides must refrain from writing to the source and target areas, respectively, and the receive side must also refrain from reading from the target area, until completion of the transfer has been signaled. To take advantage of the higher efficiency of large transfers, signaling of DMA completions is handled via additional data—a tag—contained in the transfer. This is the method we use on the Cell BE; it is possible to do this since the sequence of writes to DMA target addresses can be ordered using fences.∗ Detection of a tag (which must be different than the tag associated with the previous buffer contents) indicates to the receiver that the transfer has completed. We will assume that the sender also has a mechanism for detecting completion,† and that the architecture contains a signaling mechanism, with signal status stored locally (in local private memory, or a special hardware register) on each core.‡
∗ DMA engines usually support lists of blocks to transfer. Transfer out of order may be more efficient,
so order is usually ignored, but different constructions (bit flags on the DMA list instructions) may be available to enforce ordering within a particular list of blocks, and within all lists of blocks queued up for transfer. By putting the tags in a separate atomic unit for transfers (aligned 16 bytes), and putting this line in a separate DMA block, separated by a fence, following the list of blocks needed for the data transfer, the tag will not appear to the receiver until after all data have been transferred. † On the Cell BE processor, DMA engines are paired with computational cores, and detecting DMA completion is low cost and does not involve bus traffic. This is the preferred setup from a hardware point of view, because each private local memory needs an interface to the on-chip network fabric, and this requires that most of the functionality of a DMA engine is implemented in any case, ‡ On the Cell BE, we use the SPU signal registers.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 20 2008-10-20 #20
20
Process Algebra for Parallel and Distributed Processing Cores are referenced by CoreId. To each core, we associate the following state:
State Signal registers
Reference SignalId
Buffers
LocalAddress
Data tags
DataTag
Function Enumeration of a small, predetermined number of Boolean values local to each core, but writable from any core Store blocks of data
Values Can be set or unset
Readable, or writable by set of CoreId s, or exclusively for local access Tags associated with data Contain integers buffers
Two examples are helpful at this point. In Figure 1.5, we introduce a graphical representation of the virtual-machine language. Following universal modelling language (UML) convention, communication is asymmetric, so it is shown with half arrowheads. There is both synchronization and data transmission, so thin and thick lines are used to represent the respective low and high bandwidth requirements. Signals are labeled by their SignalId, and data are labeled by a symbol representing the data being transferred, subscripted by the data tag. In a simple transfer, the receiving core sends a preassigned signal to indicate that the receiving buffer is available for writing. It is up to the scheduler to ensure that no local or remote reads or writes to this buffer occur. In the case of local reads and writes, this is easy to verify visually, by checking the inputs and outputs of all pure
WaitSignal SendData . . . Other operations . . .
Receiving core 1
x1
SendSignal . . . Other operations . . .
No reads or writes to buffer until past barrier WaitData
No writes to buffer until DMA completion is confirmed
Sending core
WaitDMA WaitData
FIGURE 1.5: Simple data transfer requires one ready signal before data are transferred.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 21 2008-10-20 #21
Synthesizing and Verifying Multicore Parallelism Core 1
Core 2
Core 3
Core 4
21
Core 5
1 x1 f(x) f (x)1 g( f(x)) g( f (x))1 h( g( f (x))) h(g( f (x)))1 Further computation 2 y2
FIGURE 1.6: Complex transfer, minimizing signal traffic. functions between the SendSignal and WaitData. Ensuring that no DMA activity sources or targets this buffer during this time is more difficult, and requires the establishment of a partial order on the instructions executing on different cores. Similarly, the scheduler must ensure that the source buffer is not written to between the WaitSignal and WaitDMA instructions, although reading is allowed, and is effective in pipelining instructions. In the more complex example shown in Figure 1.6, a single signal is sent from core 5 to core 1 to indicate that the destinations for a whole chain of transfers are available. This amortizes the cost of the signaling over multiple transfers, at the expense of potentially tying up buffers for longer periods of time. Using such a single signal may be easier for the scheduler to software pipeline than a sequence of simple transfers, because there are fewer latencies to worry about. For algorithms requiring numerous transfers of small amounts of data, the reduction in signaling bandwidth may also be significant. In this example, we introduce conventional mathematical notation for pure functions, f , g, and h. Since the data transfers in one cycle have distinct target cores, a single data tag can be used for all transfers. To achieve good performance, the signals and data transfers would be software pipelined within each node, so that, for example, in a simple mapping of h ◦ g ◦ f over an array {xi }, with minimal pipelining, the evaluation of f (xi+2 ) would overlap the evaluation of g(f (xi+1 )) and the calculation of h(g(f (xi ))).
1.5.1 Virtual-Machine Instruction List We present the syntax of the virtual-machine instructions, together with high-level description of their intended effects. We also summarize how we implement these for
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 22 2008-10-20 #22
22
Process Algebra for Parallel and Distributed Processing
the current Cell BE architecture, how they could be implemented on other architectures, and their performance implications. There are no nested instructions. Each instruction takes a sequence of operands from the finite types CoreId, SignalId, DataTag, DMATag, LocalAddress, GlobalAddress. The sizes of these types are determined by the specific hardware/software architecture. Different architectures have different addressing schemes, but we assume in general that each core uses a LocalAddress to refer to the memory it can access with low latency, and a GlobalAddress to refer to other memory. On the Cell BE, local memories can be mapped into the originating process’s memory space, but on other architectures, global addresses may require different representations depending on its location. SendSignal CoreId SignalId Send the specified signal to the specified core. If the signal is already set, it remains set. The receiver must unset it (with a WaitSignal) in order to be able to detect the next signal. Any core can send any signal to any other core. It is up to the programmer/generator to assign signals just as registers are assigned so that different uses do not overlap. One use of this command is as a “clear to send” notice so that a remote SPU is able to initiate a DMA data transfer. This is illustrated in Figure 1.5. When used as a “clear to send” notice, the buffer to which the requested data will be written should have already been marked as free with the FreeBuffer command, unless the programmer/compiler has arranged that the new data tag is different from the current data tag. WaitSignal SignalId Pause execution on this core until the specified signal has been received. Efficiency of the schedule depends on this instruction only occurring in the execution list for a core after the signal has already been received by the underlying hardware. In this case, the latency of the transfer is completely hidden. SendData LocalAddress DataLength GlobalAddress DataTag DMATag Initiate a DMA transfer to send a data block from the memory of this core to the private memory of a second core, using its address in the global address space. In the current implementation, the DataTag is written to a reserved memory word preceding the buffer, and the data transmission is adjusted to include it in the transmission. The DMATag is an identifier used to identify one or more DMA commands, so that the local thread of execution can query the DMA engine about completion. The DMATag must be used on the same core with a WaitDMA instruction to verify the end of the transfer before local writes are allowed. On the Cell BE, it is possible to interrupt on DMA completion, which can be used to reduce the latency between initiation of a send and the next use of the buffer.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 23 2008-10-20 #23
Synthesizing and Verifying Multicore Parallelism
23
This introduces interrupt overhead into the run-time system, and significantly complicates the verification of the run-time system. WaitDMA DMATag Wait for completion of the DMA transfer with the given tag. Only the initiating core can check this tag. Another core, the controller, or off-node I/O must infer that this transfer is complete from other synchronizing instructions. WaitData LocalAddress DataTag Wait for the data block located at the given local address to have fully received an inbound transfer identified by the given data tag. In the current Cell BE implementation, this is done by repeatedly reading the data tag from the local private memory until the expected tag is read. This is convenient because it is possible to put fences in DMAs, ensuring that the data tag is transferred after the block data is transferred. More efficient hardware would support sending a signal after a DMA transfer is complete. To implement this on the current Cell BE, one would have to wait for the local signal for DMA completion and then send a signal, or use DMA list elements separated from the data write by a fence to write to memorymapped IO registers controlling signaling. The first approach would introduce either a lot of extra latency or, if interrupts are used, extra overhead on the sending core for the interrupt handler, and significantly more-complicated verification of the run-time system for the cores. The second approach may be more efficient than the tag scheme, and we plan to benchmark it. FreeBuffer LocalAddress Indicate that the buffer at the given local address is free to be written to. This has to occur before an incoming data-transfer request is initiated when the previous value of the data tag is not known, in order to be able to tell when the data-block transfer has completed. For simplicity of generation or verification, each WaitData may be paired with a FreeBuffer. In the current Cell BE implementation, this instruction causes a byte to be written to a reserved memory location preceding each buffer. If light weight signals are supported for signaling DMA completion to remote locations, this instruction might be required to reset a hardware register. LoadMemory LocalAddress DataLength GlobalAddress DMATag Initiate a DMA transfer of the specified data from main node memory to the core’s private memory. The source buffer can be written to, and the target used after checking the DMATag. StoreMemory LocalAddress DataLength GlobalAddress DMATag Initiate a DMA transfer of the specified data from the core private memory to node main memory The source buffer can be written to, and the target used after checking the DMATag.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 24 2008-10-20 #24
24
Process Algebra for Parallel and Distributed Processing
LoadComputation Computation GlobalAddress DMATag Load new computational kernel from location in main memory specified in the Computation structure. This duplicates the functionality of LoadMemory with a different type. Separating them makes verification a lot easier, and takes advantage of the encapsulation of Computation. In the Cell BE implementation, LoadComputation can be converted to LoadMemory, because the SPU cores do not distinguish between text and data. RunComputation Computation LocalAddress∗ LocalAddress∗ Parameter∗ Run the specified computational kernel, using a list of addresses for input and output buffers, and a list of parameters. The signature of the computation includes information on the sizes of the buffers and which input and output buffers may coincide (for in-place computations). This information is needed to evaluate the instruction stream into a data-flow graph. The computation is a compiled pure function, adhering to a defined binary interface. The binary interface allows a list of input data buffers, a list of output data buffers, and a list of parameters to be passed to the function. In the Cell BE implementation, the pure functions are generated from source code in an existing DSL.
1.6 Scheduling Algorithm The advantage of the approach developed in this chapter is that users can develop their own schedulers for multicore parallelism, taking advantage of the most efficient low-level asymmetric messaging, without having to cope with distributed debugging. In this section we outline the algorithm for the scheduler we are using to test this framework. This scheduling problem is NP-complete, and finding good approximation algorithms is a current research problem, especially for the (harder) case of multiprocessor scheduling [31]. Devising a good heuristic algorithm depends on understanding the properties of a good solution. Just as in the single-pipelined-processor ILP problem, the most important feature of a good schedule is the separation of loads and instructions consuming the results. We use a greedy algorithm, which seeks to move the signal/data transfers as far from the computations as possible. We divide the process into three stages, guided by the code-graph transformation sequence shown in Figure 1.7. These graphs present a multicore implementation of matrix multiplication, specialized, for the sake of readability, for multiplying a matrix A tiled into 1 × 1 blocks with a matrix B tiled into 1 × 2 blocks, splitting the two resulting block multiplications, labeled “∗”, over two cores 0 and 1. Allocation to a core i is denoted by prefixing edge and node labels with “i:”. These edges labeled with “∗” should be considered as nested edges, labeled with a control-flow graph representing the loop structure of a software-pipelined submatrix
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 25 2008-10-20 #25
Synthesizing and Verifying Multicore Parallelism
25
2
1
0: B(0,0) 2
1
0: A(0,0)
1
3
0 0: ∗
B(0,0)
A(0,0) 1
B(0,1) 0
0
1: mv
3
0: C(0,0)
1
0: B(0,1)
1: A(0,0) 1
0: ∗
0
1: ∗
1: ∗
1 C(0,0)
C(0,1) 1: C(0,1)
1
2
2
(a)
(b)
1: SS
1
2
0 0: B(0,0) 1
0: A(0,0) 0
0
1
0: Signal
1: C(0,0)
1: WD
0: <>
1: A(0,0)
1 2
0: SD
1: DMA 0 0: WM
1
0: WS
0 1
1: A(0,0) 0
1
0: ∗
2
1: Tag
3
1: B(0,1) 0
3
1 1: ∗
1: C(0,1)
1 2
(c)
FIGURE 1.7: (a) Pure, (b) distributed, and (c) concurrent data-flow graphs.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 26 2008-10-20 #26
26
Process Algebra for Parallel and Distributed Processing
multiplication program; the loop body in that control-flow graph is labeled with a pure data-flow graph assembled from several stages of an unrolled submatrix multiplication loop body [3]. The three stages, described in more detail in Sections 1.6.1 through 1.6.3, essentially produce a schedule for the final, concurrent data-flow graph by approximating it first with a schedule for the data-flow graph and then with one for the intermediate, distributed data-flow graph. Payload scheduling (see Section 1.6.1) enumerate the data blocks with sources, in the order in which they are used in computations (as inputs or outputs of pure functions, or as sources for transfers to main node memory), and enumerate the computations in the order they will be executed (this can be done once for each core or in a unified time-line.) We consider this as scheduling a pure data-flow graph, which is shown for a trivial 2-core example in Figure 1.7 (left); this drawing shows that the two multiplications are already assigned to different cores. Payload buffer allocation and communication generation (see Section 1.6.2) generate ordered lists of instructions for each core from the initial lists, converting pure functions into RunComputations, and inserting data transfers and signaling as required to get data blocks into local buffers before pure computations. We consider this as refining the previous schedule for the corresponding distributed data-flow graph mainly by also allocating the nodes to cores, and simultaneously performing resource allocation (corresponding to register allocation in simple dataflow graphs). Nodes adjacent to computations that are all on the same core are directly allocated to the same core as those computations; the other nodes are split into node sequences connected by explicit “mv” edges representing data transfer between cores (we consider “mv” to be executed by the transfer target, but do not consider it as a computation). This implies that an input node allocated to core i has its data directly transferred from memory to core i, and analogously for output nodes. The result of this is shown (for the same simple example) as the middle code graph in Figure 1.7. Communication scheduling (see Section 1.6.3) reorder the instruction lists to better hide latency of signals. Since the “mv” edges of the previous graphs are implemented using sequences of instructions with widely varying latencies and limited dependencies, these instructions must be scheduled separately, which amounts to producing a schedule for the concurrent data-flow graph shown (again for the same example) as the right-most drawing of Figure 1.7. For that graph, note that the WaitDMA instruction “WM” does not produce a (data) output, but still needs to be scheduled since it is strict like all other instructions here. It also does not have a control output since nothing directly depends on its completion; its only effect is to keep the buffer allocation for the buffer containing A(0, 0) on core 0 alive until after its completion: the outgoing
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 27 2008-10-20 #27
Synthesizing and Verifying Multicore Parallelism
27
control edge added to WM by scheduling it will, together with a correct allocation, ensure that there are no writes to that buffer before completion of that WM. We now expand on each of these three steps:
1.6.1 Payload Scheduling This first stage is separated out because the key to efficiently utilising resources in high-efficiency multicore architectures is the maximization of data reuse within the cores, and hiding of latency. Access bandwidth to main node memory (and to distant memory on other nodes) is bounded by hardware interfaces which cannot grow at the rate computational performance is growing. The first stage orders computations so that input and intermediate data can be maximally reused. In the future, on multicore processors with more processors and topologically nontrivial on-chip networks, this is the point at which operations can be reordered to distribute intercore communication sufficiently over fabric bottlenecks. On the Cell BE, main node memory traffic is the bottleneck for all of our target computations. We will refer to the list of buffer users as consumers.
1.6.2 Payload Buffer Allocation + Communication Generation The second stage maximizes latency hiding by filling buffers with data blocks required for future computations as soon as buffers become available. This is done by iterating the following six stages: 1. Look for unoccupied buffers on each core and where found assign consumers, generating instructions to send and receive signals and initiate DMA transfers, except for output blocks, which require a buffer reservation, but no action. 2. Age all pending transfers by the estimated time to initiate transfers. 3. Look at the next instructions on each core, and if all of the required blocks have been assigned to local buffers, and inputs are available, generate the RunComputation or StoreData instruction. 4. Adjust use counts on data blocks following computation. 5. Age all pending transfers by the estimated computation time for executed computations. 6. Find the oldest pending data transfer; add a WaitData instruction to the instruction stream for its core, and mark the data block in the target buffer as available.
1.6.3 Communication Scheduling The resulting schedule makes maximal use of the available buffers to hide the latency of data transfers. It does not hide the latency of signals. If signals are
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 28 2008-10-20 #28
28
Process Algebra for Parallel and Distributed Processing
implemented as high-priority, low-bandwidth side-band communication, it would not be necessary to worry about this. If signals share a single data bus, signal latency may be as long as data latency. On the Cell BE, the situation is somewhere in between. Signals are transmitted on the same bus as data, but the bus is implemented as multiple token rings running in different directions. So shorter signals running in the opposite direction to data will tend to have significantly shorter latency. Nevertheless, it is important to be able to hide this latency as well. We do this in a final stage, using what is akin to peephole optimization, by locating the matching WaitSignal and SendData pair and moving the WaitSignal forward across a number of instructions representing the smallest amount of computation above the minimum of a fixed expected latency, and a fraction of the total computation time between the WaitSignal and SendData in the input instruction stream.
1.7 Concurrency Verification Recall that verification serves two customers: the developers and users (possibly represented by certification authorities and government regulators). The user wants to know whether safety-critical software is correct. The developer also needs to understand the source of defects, so that they can be fixed, or used to develop a new design. Although errors in concurrent parallelism are inherently nonlocal, the algorithm in this section identifies the type of error and one of the instructions involved in creating an unsound program. We envisage two verification steps, which make sense in the context of multicore parallelism: Reduction of the parallel program to a pure data-flow graph, and matching the data-flow graph to a specification. For the types of problems we are immediately interested in, like structured linear algebra, it is easy to generate single- and multilevel pure data-flow graphs and convert them to an appropriate normal form. This makes the second step easy, and the first step the more interesting. The first step requires the verification that the introduction of multicore parallelism does not modify the semantics of a program. Even where normal forms for pure dataflow graphs are not available, this means that testing can be as efficient as it is for single-threaded programs. This is similar to the graph-transformation strategy we are applying successfully to the development of code transformations for VIRP [4]. In practice, incorrect programs are as interesting to the programmer whose job it is to fix them as are correct programs. On failure, our verifier will identify a program location and type of error. This is much easier to do with an ordered input program than with a set of communicating processes. This brings us back to the superscalar, out-of-order model. Although compilers know that instructions are executed out of order, they have to be presented in a linear order, and the semantics must match the sequential program semantics. This is the case for verified programs presented in our DSL. The difference from the superscalar CPU case is that some programs cannot be verified, and therefore do not have the same semantics as the equivalent sequential programs.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 29 2008-10-20 #29
Synthesizing and Verifying Multicore Parallelism
29
We will use the following terminology: • A program is a list of pairs (CoreId, Instruction). We assume that the instructions originate in this order and are transmitted to cores in this order. Since each core buffers its instructions and executes them without additional synchronization, the original program order is not necessarily the same as the resulting execution order. However, due to limited buffer sizes there is a limit to the distance within the list between any two instructions whose execution overlaps in time. Therefore, considering a single list is more restrictive than a separate list of instructions for each core. The situation here corresponds to the dispatch queues and instruction-fetch buffers on superscalar CPUs, which result in a maximum number of instructions in-flight.∗ We do not use the maximum in-flight property, but do use the presentation order, and we will add restrictions to the presentation order below. • A program is order independent if, given the same input (in main memory input locations), all possible execution orders terminate and produce the same output (in main memory output locations), although intermediate values computed and temporarily stored in private or global memory may differ. • A program is locally sequential if every (c1 , SendSignal c2 i) is followed by a corresponding (c2 , WaitSignal i), and every (c1 , SendData b c2 t m) is followed by corresponding (c2 , WaitData t) and (c1 , WaitDMA m) instructions. Note that it is easy to construct order-independent programs without this property, simply by ordering the instructions by core. • A program is safely sequential if it is locally sequential and order independent. The programmer wants to know whether the programs are order independent. Checking this is very expensive. The programmer understands sequential programs, and with some difficulty can understand parallel programs close to sequential programs. Weakly sequential programs are our attempt to formalize this concept, and we may restrict it to improve programmer understanding in the future as we gain experience writing and generating a wider range of programs. The key to usability, however, is the fact that we can efficiently identify the safely sequential programs within the class of locally sequential programs. To summarize these definitions using relational language, instructions on a single core are totally ordered; instructions on different cores have a partial order derived ∗ The instructions in-flight at a given time are all of the instructions in the process of being decoded by the
CPU, dispatched to execution units, executing (pipelined or nonpipelined), or being completed (having their results recorded in register files, store queues).
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 30 2008-10-20 #30
30
Process Algebra for Parallel and Distributed Processing
from the use of synchronizing primitives; the partial order is the intersection of all possible execution orders; the presentation order (order in the list) is a total order, and we will give conditions that imply that the partial order is a subrelation thereof, and in particular, that the presentation order is a valid execution order. One reason we can identify the safe subset efficiently is that we can identify what the intended result should be by using the pairing of send and receive instructions, and from there we can verify that these are the only results by keeping track of a bounded set of state information (with the size of the information being O(cores2 ).
1.7.1 Motivating Example Before explaining the efficient method of verification, it is important to understand that locally sequential programs are not necessarily sound. For simplicity, the following locally sequential program uses signals, but the same issues arise with data transmission: Index 1 2 3 4 5 6
Core 1 SendSignal s → c2
Core 2 Long computation WaitSignal s computation
Core 3
SendSignal s → c2
WaitSignal s
Remember that each core executes independently of the other cores, except where explicit wait instructions block execution until some kind of communication (signal, change in data tag, DMA) is confirmed to have completed. Therefore, in this case the most likely instruction completion order has core 3 executing the SendSignal as soon as it is queued, allowing the signal to be sent before core 2 has received the core 1’s signal and cleared the signal hardware: Index 2 5 1 3 4 6
Core 1 SendSignal s → c2
Core 2
Core 3
SendSignal s → c2 Second signal overlaps the first, only one registered Long computation WaitSignal s computation No signal is sent, so the next WaitSignal blocks WaitSignal s
To be precise, completion of the SendSignal means that the signal has been initiated by the sender, and reception may be delayed, so the signal from core 3 could even arrive before the signal from core 1. In either case, neither signal will arrive after
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 31 2008-10-20 #31
Synthesizing and Verifying Multicore Parallelism
31
the first WaitSignal, so the second WaitSignal will wait forever, and this program execution will not terminate. The problem is caused because there are no signals or data transmissions enforcing completion of instruction 5 to follow completion of instruction 3. This example, when considered as part of a longer program, also demonstrates a possible safety violation with the valid completion order: Index 1 5 3
Core 1
Core 3 SendSignal s → c2
WaitSignal s Computation using wrong assumptions
4 2 6
Core 2 Long computation
SendSignal s → c2 WaitSignal s
In this case, the computation 4 might rely on assumptions that are only available once the SendSignal 2 completes—for example, it might initiate a DMA to core 1, which could arrive before some earlier instructions on core 1 are still writing to the DMA’s target area, thus invalidating the DMA transfer.
1.7.2 Strictly Forward Inspection of Partial Order We have seen that locally sequential programs are not always sound. The advantage that locally sequential programs have over general programs is that orderinducing instructions must occur before instructions whose well-definedness depends on them. Using simple inference to maintain a list of known ordering relations between instructions on different cores, we can inspect the program in order (even though it can execute out of order). Referring to Figure 1.8, we see that after instruction 3, we know that instructions on core 2 will complete after instruction 2 on core 1 (and any previous instructions). We know nothing about the relative execution of instructions other than this until instruction 6. At this point we know that instructions after and including 6 on core 1 must execute after any instruction before instruction 5 on core 2. After instruction 8, we have the relation that this and further instructions on core 3 execute after instructions up to instruction 7 on core 1, but since instruction 7 on core 1 executes after instruction 5 on core 2, we also know that instruction 8 on core 3 executes after instruction 5 on core 2. Contrast this with the example in Section 1.7.1, where we never have any relations between instructions on core 3 and other cores. Keeping track of these relations is the key to efficient verification of correctness. There is one other class of (bad) example to keep in mind: the situation where, in presentation order, two cores send the same signal to a third core without a wait
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 32 2008-10-20 #32
32
Process Algebra for Parallel and Distributed Processing Good Program Index 1 2 3 4 5 6 7 8 9 10
Core 1 SendSignal s → c2
Core 2 Long computation
Core 3
WaitSignal s computation SendSignal s → c1 WaitSignal s SendSignal s → c3 WaitSignal s SendSignal s → c2 WaitSignal s
FIGURE 1.8: Weakly sequential program.
on the third core to separate them. This case is visually apparent, because the sends overlap in program order, but we have to safeguard against it nonetheless. Clearly, the same considerations apply to data transfers. With data transfers, we must additionally check the values of the data tags, because if the existing tag matches the tag being transferred in, the WaitData instruction will never block execution, even if the data are not available.
1.7.3 State Recall that a locally sequential program is an sequence of instructions (communication primitives) with a totally ordered presentation. The order does not determine the execution order, but it does define the analysis order for the verification algorithm. Without this order, there would be no linear-time verification algorithm. Given the order, it makes sense to talk about the state maintained by the verification algorithm. This state encodes only the information about the possible execution states required to verify the soundness of the program in this way. For brevity, we will now write I instead of Instruction for the set of instructions, C instead of CoreId, S instead of SignalId, and L instead of LocalAddress. Let D be the set of DMA Tags. Let T be the set of data tags. In the following, we assume a fixed program P . Depending on this program P , let N = {0, . . . , length(P ) − 1} ⊆ N be the set of indices into P considered as a list, so that we can now consider the program as a total function P : N → C × I instead. N is considered to be ordered with the standard ordering of the natural numbers. For each core c : C, define Nc = {n : N|P (n)1 = c} to be the corresponding restriction of the index set to instructions for c. We are now interested in the partial strict-ordering ≺ (depending of course on P ) on N where n1 ≺ n2 means the instruction (of P ) at n1 necessarily completes before the instruction at n2 . The reflexive closure of this is frequently known as “happensbefore relation” [16], or Mazurkiewicz’s trace [20,32].
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 33 2008-10-20 #33
Synthesizing and Verifying Multicore Parallelism
33
For a fixed core, c : C, the instructions in the program assigned to c are completed in order, so we demand that, for each core c : C, the restriction of < to Nc is included in≺: ∀c : C . ∀n1 , n2 : Nc . n1 < n2 ⇒ n1 ≺ n2 There is no a priori order between instructions assigned to distinct cores, and this may lead to unsound programs like the example in Section 1.7.1. Ordering is determined by transition dependencies, which in our case means blocking communication. It would be sufficient to calculate ≺ completely, but only a small portion of this partial order is required at a time. Since instructions on a single core are totally ordered, it is easy to calculate the entire relation ≺ if, for each instruction index n and distant core, the latest instruction known to complete before n, and the earliest instruction known to complete after n are stored. We will see that it is sufficient to store only the latest instruction known to complete before the current instruction, and convenient to repeat information for all cores, in the form of a “ollows” map indexed by n ∈ N: n : C × C → N n (c1 , c2 ) = max≤ {m : Nc1 | ∀n : Nc2 . n ≤ n ⇒ m ≺ n } . Note, in particular, that if n ∈ Nc2 this definition does not say that is the latest instruction on c1 such that all future instructions on c2 follow m. This property could be used instead, but is more expensive to compute since it requires information about future instructions. Figure 1.9 shows that the instruction at index n (c1 , c2 ) is always a send instruction on core c1 . For the verification method, we need to define a function : (C × N × F ) × (C × F ) → F,
Φn(c1, c2)
n
c1
c2
c
FIGURE 1.9: For convenience, n (c1 , c2 ) is defined for all n, even if n ∈ Nc2 , to be the index of the last instruction on c1 known to complete before an instruction on c2 up to and including index n.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 34 2008-10-20 #34
34
Process Algebra for Parallel and Distributed Processing
where F = C × C → N models a slice of . combines the slices of the partial order at a corresponding pair of send/wait instructions that induce an order on the completion of the wait and instructions on other cores. This potentially strengthens what we know about n2 (c, cWait ) for all c, i.e., about the last instructions known to complete before the wait and future instructions on the destination core. ((c1 , n1 , n1 ), (c2 , n2 ))(c , c ) ⎧ ⎪ c = c1 , c = c2 ⎨max{n2 (c , c ), n1 } = max{n2 (c , c ), n1 (c , c1 )} c = c2 ⎪ ⎩ n2 (c , c ) otherwise. Our method of verification is a construction of the following functions inductively in the instruction index, n. ⎞ ⎛ C ×N ×F ⎠ σn : C × S → ⎝ ⊕ N ⎞ ⎛ T ×C×N ×F ⎟ ⎜ ⊕ ⎟ ⎜ ⎟ ⎜ T ×N ⎟ ⎜ ⎟ ⎜ ⊕ n : C × L → ⎜ ⎟ D ⎟ ⎜ T ×2 ⎟ ⎜ ⎠ ⎝ ⊕ T ×D n : C × C → N where ⊕ means disjoint union 2D is the power set of D To calculate Xn+1 , where Xn = (σn , n , n ), it is only necessary to have access to Xn , because we make copies of slices of which are needed in future steps. The disjoint unions can be understood in the following way: σ , which records the last instruction to send the given signal to the given core, has two cases for signal status: the signal has already been received by a WaitSignal or the receiver is still pending. Case I: a signal is expected from a core c : C, at instruction number n : N, with : F being the known partial order at n. Case II: the signal has been received at instruction n : N. To simplify the cases, we assume that all signals have last been received at instruction −1 (before the program starts). For , these are the following cases: 1. An incoming transfer with tag in T , from core in C started at instruction in N with known partial order in F is expected. 2. The data tag is in T , and the data were last used at an instruction in N.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 35 2008-10-20 #35
Synthesizing and Verifying Multicore Parallelism
35
3. The data buffer is the source of one or more DMA transfers whose list of (completion) tags which have not yet been consumed by a WaitDMA are in 2D , and the data tag is in T . 4. An incoming DMA transfer with tag in D was started, and the data tag is in T . Such a transfer can only have come from main memory, and been initiated by a LoadMemory. Initially, σ0 maps every argument to −1 (indicating that the last signal was received before the beginning of the program), and 0 maps to the pair (unused data tag, −1). 0 maps every argument to −1 as well, indicating that nothing is known about ordering. The maps σn+1 , n+1 , and n+1 are constructed from the maps at n and the instruction at n executing on core c using the universal rule n+1 (c, c) = (n), and specific rules for the following cases: SendSignal c s We need to verify that there is no current use of this signal in any program order. Dividing into two cases according to the form of σn (c , s ), the first case excludes programs in which the signal is in use in the presentation order. This is not sufficient because the presentation order is not the only possible execution order, so we need to use the follows relation to check this. If σn (c , s ) = (c , n , ), this is an error, because this signal s may be sent by both cores (from c at the present instruction n and from c at n ) at the same time. If n (c , c) ≥ σn (c , s ), then we know that the last instruction to consume the signal s on c at σn (c , s ) completes before n (c , c) on c which (by definition) occurs before the current instruction on c. (This includes the case that this signal has never been used, in which case σn (c , s ) = −1.) Let n+1 := n , n+1 := n , and σn+1 (c , s ) := (c, n, n ), σn+1 (c , s ) := σn (c , s ) for other values of (c , s ). If the inequality is not satisfied, the program is not sound, because we do not have n ≺ n, meaning that this signal could arrive before the previous WaitSignal s on c , which is the case of the bad example program. WaitSignal s If σn (c, s ) = (c , n , ), then we know that the signal s will come from core c , and we can use the fact that the wait completes after the send, to update the follows map. Let n+1 := ∨((c , n , ), (c, n )), n+1 := n , σn+1 (c, s ) := (n) otherwise σn+1 (c , s ) := σn (c , s ). If σn (c, s ) = (n ), this is an error, or the program is not locally sequential, as required for this analysis.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 36 2008-10-20 #36
36
Process Algebra for Parallel and Distributed Processing
SendData l λ c l t d At the source: We need to ensure that the source data are not about to be rewritten and that the use of t and d does not conflict with another use. If d ˜ then the DMA tag is already in use, and the ˜ for some l, appears in n (c, l) completion of this DMA would be indistinguishable from the completion of another DMA already originated from this core, so this is an error. If n (c, l ) = (t , n ), then no DMAs (SPU to SPU transfers or transfers from main memory) are pending for this buffer, so record the send by letting n+1 (c, l ) := (t , {d}) (with n+1 equal to n at all other values). If n (c, l ) = (t˜, d ), then on the sending core c an incoming DMA is in progress from main memory with data tag t˜ and DMA tag d . Therefore it is not safe to start an outgoing DMA as this data could be overwritten while it is being sent, so this is an error. If n (c, l ) = (t˜, D ) for some subset D ⊂ D, then there are already outgoing DMAs in progress, which is valid if we do not try to reuse DMA tags; or if t˜ = t , then two different tags are being used for outgoing DMAs, which is not supported (because it would result in an unknown data tag being sent for the previously initiated transfer). Otherwise, record its use by n+1 (c, l ) := (t, D ∪ {d}), and copy the other maps from n to n + 1. Otherwise, this buffer is the target of a DMA from another core, which is an error. At the destination: Between cores, execution order does not have to follow presentation order. So we need to check that all use of the previous contents of the target buffer must have completed before the current instruction. If n (c , l ) = (t , n ), t = t , and n ≤ n (c , c), then the destination buffer has no pending DMAs after instruction n , the new tag does not match the old tag (so it will be detectable by the WaitData), and the current instruction is known to complete (n (c , c)) after the last instruction (n ) to have used the target buffer, so it is safe, and we set n (c , l ) := (t , c, n, n ). Otherwise, report the error that multiple incompatible DMAs with source and target (c , l ) were detected, or the data tag was illegally reused. WaitData l t If n (c, l ) = (t , c , n , ), then we are waiting for this data, and have stored the follows map at the send instruction, which we can combine with the map at the current instruction: n+1 := ∨((c , n , ), (c, n )). Let σn+1 := σn , and n+1 := n except n+1 (c, l ) := (t , n), which indicates that this buffer is safe to use after the current instruction. Otherwise, no incoming DMA from another core preceded this instruction, so the program is not locally sequential, or this DMA overlaps DMAs initiated locally, which is unsafe.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 37 2008-10-20 #37
Synthesizing and Verifying Multicore Parallelism
37
LoadMemory l λ g d Check that d does not appear in the image of n ({c} × L), otherwise the uses of this tag overlap, which is an error. If n (c, l ) = (t, n ), then the buffer has no pending IO, and we can initiate an incoming DMA by setting n+1 := n except n+1 (c, l ) := (t, d), σn+1 := σn and n+1 := n . Otherwise, the target buffer is the source or target for transfer, which may not have completed, and this will produce an indeterminate result. StoreMemory l λ g d Check that d does not appear in the image of n ({c} × L), otherwise the uses of this tag overlap, which is an error. If n (c, l ) = (t, n ), then no other IO is pending, and the store is safe. Set n+1 := n except n+1 (c, l ) := (t, {d}) σn+1 := σn and n+1 := n . If n (c, l ) = (t, D ), other outgoing DMAs are pending, which is allowed. So set n+1 := n except n+1 (c, l ) := (t, D ∪{d}) (to record the new DMA tag we are waiting for), σn+1 := σn and n+1 := n . Otherwise, the source buffer is the target for a pending transfer, and this will produce an indeterminate result. WaitDMA d The DMA tag could be associated with incoming or outgoing IO. In either case ∃l such that n (c, l) = (t, d) (incoming), or n (c, l) = (t, D ) and d ∈ D (outgoing), and l is unique. Otherwise, there is an error, because no DMAs are pending using this tag. This either indicates an unsound program or a nonlocally sequential presentation. If n (c, l ) = (t , {d}) then n+1 (c, l ) := (t , n). If n (c, l ) = (t, D ) then n+1 (c, l ) := (t, D \ {d}) If n (c, l ) = (t , d) then n+1 (c, l ) := (t , n). For other values n+1 := n . Also, σn+1 := σn and n+1 := n . RunComputation x (l , l ,...) (l˜ , l˜ ,...) (p , p , ...) Inputs: If for every l ∈ {l , l , ...}, n (c, l) = (t, n ) or (t, D ) then n+1 (c, l) := (t, n) or (t, D ) (respectively), otherwise, the buffer has pending incoming IO, which makes this computation unsound. Outputs: If for every l ∈ {l , l , ...}, n (c, l) = (t, n ) then n+1 (c, l) := (t, n), otherwise, there is pending IO and this modification makes the program unsound. Other instructions have no effect, so n+1 := n , n+1 := n , and σn+1 := σn . THEOREM 1.1 A locally sequential program is order independent iff the inductive verification described in this section terminates without error.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 38 2008-10-20 #38
38
Process Algebra for Parallel and Distributed Processing For the proof of this theorem, we need the following lemmas:
LEMMA 1.1 Each d ∈ D occurs at most once in n ({c} × L). PROOF This property is preserved because the verifier starts with no elements of D in the images of any of the maps, and all rules check for previous use of a tag before constructing the n + 1 maps from the n maps and signal an error if multiple occurrences of some d would result. LEMMA 1.2 In a locally sequential program which passes the verification procedure, the only signal a (n, c, WaitSignal s) can trap is the one in σn (c, s). PROOF Assume that there is another signal which can be trapped. Since the program is locally sequential, the “matched” SendSignal c s must precede the WaitSignal s in presentation order. There are two cases to check: the other send precedes the wait, or follows the wait. 1. If it precedes the wait, we have two different SendSignal c s instructions sending the same signal to the same core, c ∈ C, preceding the WaitSignal, with unrelated instructions potentially interleaved. However, this situation would not pass the verification, because at the second send, (n, SendSignal c s), σn (c, s) = (c , n , ) where n , c are the index and location of the first send, which would cause the verifier to return an error. 2. If the second SendSignal instruction follows the wait in presentation order, but both sends can execute before the wait, then at the second send, the verifier will find n (c , c) < n , where (n, c) are the coordinates of the second send, and (n , c ) of the intermediate wait. This would have caused the verifier to signal an error, precisely because it could not guarantee the safety of the second send. LEMMA 1.3 In a locally sequential program which passes the verification procedure, the only data a (n, c, WaitData l t) can observe arriving is the one in n (c, l). PROOF
Analogous to the proof of Lemma 1.2.
LEMMA 1.4 The presentation order of a locally sequential program is a valid execution order up to n if verification produces no error up to instruction n.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 39 2008-10-20 #39
Synthesizing and Verifying Multicore Parallelism
39
PROOF Assume the contrary, and let m be the first instruction which is not executable after instruction m − 1 and all previous instructions have completed. The only instructions which may not be executable are the wait instructions (for a signal, for a data tag, or for a DMA completion tag). Since m is the first nonexecutable instruction, all previous instructions have completed, which includes the corresponding send instruction (since the program’s presentation is locally sequential). So m is also executable, which is the contradiction. LEMMA 1.5 For all c , c ∈ C, n ∈ Nc , n ∈ Nc , if n > n (c, c ), then there exists an execution order in which n < n. PROOF Construct an execution order as follows. Let Uc = {m ∈ Nc : m ≤ n (c , c )}, and U = c ∈C Uc . A modification of the proof of Lemma 1.4 shows that the presentation order is a valid execution order for the subset U. The only new point is that the instructions not in U are not required to make the instructions of U executable, because the follows maps capture the supremum of the necessary instructions for the subset U. Since n is not in the execution order up to this point, n > n in this order, as required.
1.7.4 Proof of Theorem To prove that a locally sequential program which is order independent will pass verification we consider the converse situation, a locally sequential in which an error is flagged. For each case of such an error, one must construct a possible execution order which results in a deadlock. We do this for the hardest case, the errors raised during the processing of a SendSignal. The other errors can be treated in the same way, or they obviously indicate a failure to be locally sequential (waits preceding sends). (n, c, SendSignal c s) 1. Error caused by σn (c , s) = (c , n , ) where we have a signal s which could be sent by both c and c to c . Presentation order is a valid execution order up to n if it produces no error up to instruction n by Lemma 1.4. Therefore both signals could arrive at c before the next WaitSignal, in which case both would be consumed. This would leave a second WaitSignal without a paired SendSignal according to the 1-1 pairing of SendSignal and WaitSignal required in a locally sequential program, and a deadlock would result. (Note that even if the program is malformed in other ways, as long as the sends and waits are paired, each send can make at most one wait executable, so the pairing is the only required property to insure a deadlock, unless some other problem manifests itself first.)
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 40 2008-10-20 #40
40
Process Algebra for Parallel and Distributed Processing 2. Error caused by σn (c , s ) > n (c , c). This tells us that the last instruction to consume signal s on c at σn (c , s ) does not necessarily complete before n (c , c) on c . By Lemma 1.5, there is an execution order in which the signal sent at n completes before the wait at σn (c , s ), in which case the wait would capture two signals, and there would be an unpaired wait later in the program to cause a deadlock, as above.
To prove the converse, let P be a locally sequential program passing verification. We must first show that the program has no deadlocks, and always produces the same result. Assume that there is an execution order with a blocking instruction. Let n be the first instruction in presentation order which blocks. In analogy with the proof of Lemma 1.4, this instruction must be a wait instruction, which by Lemma 1.2 and Lemma 1.3 is paired with a unique send instruction which precedes it in the presentation order. Since that instruction did not deadlock (because n is the first such instruction), it is executable, making n executable, implying that a verified program can have no deadlocks. To show that the (nondeadlocking) result is order independent, we can symbolically execute it, checking at each step that the results do not depend on execution order. To do this, modify the image of n to include an expression type, E, with different meanings for each case in the disjoint union: T × C × N × F × E—an incoming transfer is expected, and when it arrives, E will be the contents of the buffer; T ×N ×E—the buffer contains the expression E, which was last used at instruction n ∈ N; T ×2D ×E—the buffer contains the expression E, and is the source for outgoing transfers; T × D × E—the buffer is the target of an incoming transfer from memory, and when it completes the buffer will contain E. New expressions are formed by a RunComputation at instruction n from the expressions in the input buffers in n and the new expressions are stored in n+1 . The verification mechanism ensures that the source and destination buffers in are in the definite state T × N × E. Sourcing buffers in other (indefinite) states would have been flagged as an error. Input expressions are formed by memory loads, and an additional map from memory store locations to expression values represents the output of the program. Memory outside the on-core buffers is considered to be static input, or write-once output data. This concludes the proof of the theorem. For the second step in verification, one must compare the expressions generated in this way with specifications for the program. For computations of limited size, this can be done by using a nodes in a directed acyclic graph (DAG) with sharing as the type E. The result is a term forest, which can easily be compared to a specification in the same form. This procedure would be considerably more complicated for
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 41 2008-10-20 #41
Synthesizing and Verifying Multicore Parallelism
41
programs whose corresponding DAGs would not fit in program memory, but it is the nature of this approach that computation, and specifications are nested, so that the size of DAGs at each level are manageable. So far, this has always been the case.
1.8 Conclusion We have shown that the control flow required to efficiently utilize multiple lightweight cores can be abstracted out of the program into a simple process algebra, which we use to label the middle layer of Coconut nested code graphs. We have drawn an analogy between this language and a RISC instruction set used in a superscalar processor. This analogy leads naturally to the idea that the multicore level of the graph be presented in order, for out-of-order execution. With this presentation, it was possible to verify the correctness of this level of parallelism in an efficient manner. Above all, Coconut is designed to make compiler optimizations transparent to the users who are responsible for performance. Out-of-order execution in superscalar cores is well understood and widely exploited by both the optimizing compiler community, and high-performance code tuners. The main difference between out-oforder execution within a CPU and our multicore model is that we do not see a hardware solution to the problem of maintaining in-order semantics. This turns the verification algorithm of this chapter from a requirement for exceptional safety-critical applications into an everyday tool required by anyone optimizing multicore applications under this model. Users working on parallelization strategies within Coconut will come to view verification as a useful build/debug tool. As evidenced by the extensive involvement of undergraduates in developing Coconut programs, even users who do not understand the theoretical underpinnings can derive significant benefit from tools which integrate verification mechanisms. At the SIMD level, verification was provided by symbolic evaluation and simulation. At the multicore level, users will be indirectly using process algebras to check the soundness of multicore schedules. Although we have developed this approach to multicore parallelism within the framework of Coconut, it could quite easily be adapted to other languages, replacing existing remote-procedure-call and synchronization libraries. The main challenge would be expressing patterns of concurrent parallelization. All functional programming languages have constructions which lend themselves to parallelization, as do environments like Matlab, Octave, Maple, and Mathematica. Other languages have been designed to support specific patterns of parallelism, notably MapReduce [12,30]. The next challenge for Coconut is to find combinations of patterns and target architectures for which more complex data flows can produce higher levels of performance. In the future, we will extend this language to handle nested parallelism (including multiple nodes each containing multiple cores). We will also address multiphase computations, including the changing use of memory, loading and unloading of computations, and the overlapping of fill/drain between phases.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 42 2008-10-20 #42
42
Process Algebra for Parallel and Distributed Processing
Acknowledgments We thank Kevin E. Browne and Shiqi Cao for significant help with the proof in Section 1.7. We are also grateful to the editors and the anonymous reviewers for numerous constructive comments. We especially thank Robert Enenkel of IBM Canada, Inc., for support and advice over several years. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc., in the United States, other countries, or both. IBM and PowerPC are registered trademarks of International Business Machines Corporation. Matlab is a trademark of The MathWorks, Inc. Maple is a trademark of Waterloo Maple Inc. Mathematica is a trademark of Wolfram Research, Inc.
References [1] E. Albert, M. Hanus, F. Huch, J. Oliver, and G. Vidal. Operational semantics for declarative multi-paradigm languages. Journal of Symbolic Computation, 40(1):795–829, 2005. [2] C. K. Anand and W. Kahl. A Domain-Specific Language for the Generation of Optimized SIMD-Parallel Assembly Code. SQRL Report 43, McMaster University, Hamilton, Ontario, Canada, May 2007. Available from http://sqrl. mcmaster.ca/sqrl reports.html. [3] C. K. Anand and W. Kahl. Multiloop: Efficient software pipelining for modern hardware. In CASCON ’07: Proceedings of the 2007 Conference of the Center for Advanced Studies on Collaborative Research, ACM, New York, 2007, pp. 260–263. [4] C. K. Anand and W. Kahl. Code graph transformations for verifiable generation of SIMD-parallel assembly code. In A. Schurr, M. Nagl, and A. Zundorf, (Eds.), Applications of Graph Transformations with Industrial Relevance, AGTIVE 2007, Lecture Notes in Computer Science, 2008. (to appear). [5] P. Bellens, J. M. Perez, R. M. Badia, and J. Labarta. CellSs: A programming model for the Cell BE architecture. IEEE, Tampa, FL, November 2006. [6] R. Bruni, J. Meseguer, U. Montanari, and V. Sassone. Functorial models for petri nets. Information and Computation, 170(2):207–236, 2001. [7] I. B. M. Corporation, S. C. E. Incorporated, and T. Corporation. Cell Broadband Engine Programming Handbook. IBM Systems and Technology Group, Hopewell Junction, New York, 1.0 edition. [8] A. Corradini and F. Gadducci. An algebraic presentation of term graphs, via gs-monoidal categories. Applied Categorical Structures, 7(4):299–331, 1999.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 43 2008-10-20 #43
Synthesizing and Verifying Multicore Parallelism
43
[9] A. Corradini, F. Gadducci, and W. Kahl. Term graph syntax for multi-algebras. Technical Report TR-00-04, Dipartimento di Informatica, Universit`a di Pisa, Pisa, Italy, 2000. [10] A. Corradini, F. Gadducci, W. Kahl, and B. K¨onig. Inequational deduction as term graph rewriting. Electronic Notes in Computer Science, 72(1):31–44, 2007. [11] A. Corradini, U. Montanari, F. Rossi, H. Ehrig, R. Heckel, and M. L¨owe. Algebraic approaches to graph transformation, part I: Basic concepts and double pushout approach. In G. Rozenberg, (Ed.), Handbook of Graph Grammars and Computing by Graph Transformation, Vol. 1: Foundations, Chapter 3. World Scientific, Singapore, 1997, pp. 163–245. [12] J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107–113, 2008. [13] A. van Deursen and P. Klint. Little languages: Little maintenance? Journal of Software Maintenance, 10:75–92, 1998. [14] A. van Deursen, P. Klint, and J. Visser. Domain-specific languages: An annotated bibliography. ACM SIGPLAN Notices, 35(6):26–36, 2000. [15] E. A. Emerson. Temporal and modal logic. In J. van Leeuwen, (Ed.), Handbook of Theoretical Computer Science, Volume B: Formal Models and Semantics. Elsevier Science Publishers, Amsterdam, the Netherlands, 1990, pp. 995–1072. [16] C. Flanagan and P. Godefroid. Dynamic partial-order reduction for model checking software. In Principles of Programming Languages, (POPL) 2005. ACM, Long Beach, CA, January 2005, pp. 110–121. [17] M. Frappier, A. Mili, and J. Desharnais. A relational calculus for program construction by parts. Science of Computer Programming, 26(3):237–254, May 1996. [18] F. Gadducci and U. Montanari. Axioms for contextual net processes. In K. Larsen et al., (Eds.), ICALP ’98, Aalborg, Denmark, Lecture Notes in Computer Science, 1443:296–308, 1998. [19] Gheorghe S¸tef˘anescu. Network Algebra. Springer, London, April 2000. [20] P. Godefroid. Partial-Order Methods for the Verification of Concurrent Systems: An Approach to the State-Explosion Problem, Vol. 1032. SpringerVerlag, New York, 1996. [21] G. Goumas, M. Athanasaki, and N. Koziris. Automatic code generation for executing tiled nested loops onto parallel architectures. In SAC ’02: Proceedings of the 2002 ACM Symposium on Applied Computing, ACM, New York, 2002, pp. 876–881.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 44 2008-10-20 #44
44
Process Algebra for Parallel and Distributed Processing
[22] G. Goumas, N. Drosinos, M. Athanasaki, and N. Koziris. Automatic parallel code generation for tiled nested loops. In SAC ’04: Proceedings of the 2004 ACM Symposium on Applied Computing, ACM, New York, 2004, pp. 1412– 1419. [23] M. Gschwind, H. P. Hofstee, B. Flachs, M. Hopkins, Y. Watanabe, and T. Yamazaki. Synergistic processing in cell’s multicore architecture. IEEE Micro, 26(2):10–24, 2006. [24] B. Hoffmann and D. Plump. Jungle evaluation for efficient term rewriting. In J. Gabrowski, P. Lescanne, and W. Wechler, (Eds.), Algebraic and Logic Programming, ALP ’88, Mathematical Research, 49. Akademie-Verlag, Gaussig, GDR (now Germany) 1988, pp. 191–203. [25] P. Hudak. Building domain-specific embedded languages. ACM Computing Survey, 28(4es):196, 1996. [26] P. Hudak. Modular domain specific languages and tools. In P. Devanbu and J. Poulin, (Eds.), Proceedings of the Fifth International Conference on Software Reuse. IEEE Computer Society Press, Victoria, British Columbia, Canada, 1998, pp. 134–142. [27] W. Kahl. The term graph programming system HOPS. In R. Berghammer and Y. Lakhnech, (Eds.), Tool Support for System Specification, Development and Verification. Wien, 1999. Springer-Verlag, Vienna (Wien),Austria,pp. 136–149. ISBN: 3-211-83282-3. [28] W. Kahl, C. K. Anand, and J. Carette. Control-flow semantics for assemblylevel data-flow graphs. In W. McCaull et al., (Eds.), 8th International Seminar on Relational Methods in Computer Science. St. Catherines, Ontario, Canada, Lecture Notes in Computer Science, 3929:147–160, 2006. [29] E. Koutsofios and S. C. North. Drawing graphs with dot. Technical Report, AT&T Bell Laboratories, Murray Hill, NJ, 1993. [30] R. L¨ammel. Google’s MapReduce programming model—revisited. Science of Computer Programming, 70(1):1–30, 2008. [31] S. Leonardi and D. Raz. Approximating total flow time on parallel machines. Journal of Computer and System Sciences, 73(6):875–891, 2007. [32] A. Mazurkiewicz. Trace theory. In Petri Nets: Applications and Relationships to Other Models of Concurrency, Advances in Petri Nets 1986, Part II Lecture Notes in Computer Science, 255:279–324, 1986. [33] P. S. Pacheco. Parallel Programming with MPI. Morgan Kaufmann, San Francisco, CA, 1997.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 45 2008-10-20 #45
Synthesizing and Verifying Multicore Parallelism
45
[34] D. Plump. Term graph rewriting. In H. Ehrig, G. Engels, H.-J. Kreowski, and G. Rozenberg, (Eds.), Handbook of Graph Grammars and Computing by Graph Transformation, Vol. 2: Applications, Languages and Tools, Chapter 1. World Scientific, Singapore, 1999, pp. 3–61. [35] V. Sassone. On the algebraic structure of petri nets. Bulletin of the European Association for Theoretical Computer Science, 72:133–148, 2000. [36] G. Schmidt and T. Str¨ohlein. Relations and Graphs, Discrete Mathematics for Computer Scientists. EATCS-Monographs on Theoretical Computer Science. Springer, New York, 1993. [37] M. Sleep, M. Plasmeijer, and M. van Eekelen, (Eds.). Term Graph Rewriting: Theory and Practice. Wiley, New York, 1993. [38] M. Snir, S. W. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra. MPI— The Complete Reference. Scientific and Engineering Computation Series. MIT Press, Cambridge, MA, 1996. [39] W. Thaller. Explicitly staged software pipelining. Master’s thesis, Department of Computing and Software, McMaster University, Hamilton, ON, 2006. Available at http://sqrl.mcmaster.ca/∼anand/papers/ThallerMScExSSP.pdf. [40] G. Uszkay. HUSC language and type system. Master’s thesis, Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada, 2006.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C001 Finals Page 46 2008-10-20 #46
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 47 2008-10-22 #3
Chapter 2 Semi-Explicit Parallel Programming in a Purely Functional Style: GpH Hans-Wolfgang Loidl, Phil Trinder, Kevin Hammond, Abdallah Al Zain, and Clem Baker-Finch Contents 2.1
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Semi-Explicit Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 From Algebraic Specification to Executable Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Operational Semantics of GpH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Operational Semantics Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Heaps and Labeled Bindings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Single-Thread Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Multithread Transitions and Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.5 Computation Relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6 Properties of the Operational Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6.1 Soundness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6.2 Adequacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6.3 Determinacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Program Equivalences and Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Pipeline Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Data Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Strategic Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 Cluster Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.5 Transforming Clustered Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.6 Strategic Clustering Version of sumEuler . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.7 Generic Clustering Version of sumEuler . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Implementation of GpH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Implementation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 Thread Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.3 Spark Stealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.4 Memory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Assessment of Parallel Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49 49 51 54 54 54 55 56 58 58 59 59 60 62 62 63 63 64 66 66 67 68 68 69 70 70 71 72 73
47
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 48 2008-10-22 #4
48
Process Algebra for Parallel and Distributed Processing
Declarative programming languages can play an important role in the process of designing and implementing parallel systems. They bridge the gap between a highlevel specification, with proven properties of the overall system, and the execution of the system on real hardware. Efficiently exploiting parallelism on a wide range of architectures is a challenging task and should, in our view, be handled by a sophisticated runtime environment. Based on this design philosophy we have developed and formalized Glasgow parallel Haskell (G P H), and implemented it as a conservative extension of the Glasgow Haskell Compiler (GHC). The high-level nature of declarative languages eases the task of mapping an algebraic specification down to executable code. In fact, the operational components of the specification can already be considered an implementation, with the associated properties acting as assertions in the program. Based on a formal model of the declarative language, the validity of these properties can be established by manual proof, which works on a level of detail similar to the specification language itself. Many operational aspects, usually complicating a proof of an implementation, do not come into the picture at this level. Most importantly, unnecessary sequentialization of the code is avoided. However, the goal of implicit parallelism has proven an elusive one. Often the automatically generated parallelism is too fine-grained to be efficient. In other cases, the data-dependencies between expressions prohibit the generation of a sufficient amount of parallelism. Thus, we employ an approach of semi-explicit parallelism, where only potential parallelism has to be annotated in a program, and all aspects of coordination are delegated to the runtime environment. A corresponding formal model, in the form of a structured operational semantics, handling pools of both realized and potential parallelism, is used to establish the correctness of programs employing semi-explicit parallelism. The runtime environment itself is capable of synchronizing parallelism, using automatic blocking on data under evaluation, and by simulating virtual shared memory across networks of machines. Being embedded into the optimized runtime environment for a sequential language, we achieve efficient execution of a high-level language, close to the original specification language, while minimizing the programmer effort in parallelizing the code and being scalable to large-scale applications that can be executed on heterogeneous networks and computational grids. This chapter summarizes research performed over more than a decade, covering language design [28], semantics [6], and implementation [21,22,29]. In particular this chapter elaborates on the semi-explicit programming model (Section 2.1), reflects on the mapping from specification to executable code (Section 2.2), presents a structural operational semantics (Section 2.3) for reasoning about these parallel programs (Section 2.4), discusses the main characteristics of the graph-reductionbased implementation (Section 2.5), underlines the usability of the system by assessing the performance of existing applications (Section 2.6), and concludes by recapitulating the role of our language and methodology as a tool for specification, transformation, and efficient parallel execution (Section 2.7).
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 49 2008-10-22 #5
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
49
2.1 Introduction One of the key problems of parallel programming is to identify work that may be suitable for parallel execution. Because of their side-effect-free character, it is relatively easy to identify independent expressions in purely functional languages, such as Haskell [24], and to then construct independent threads to evaluate these expressions. However, the costs of creating and synchronizing even ultralightweight threads can be extremely high relative to their runtime, and even projected multicore architectures such as Intel’s 80-core research testbed [13] will be unable to extract effective performance from more than a few hundred simultaneously running threads. Many approaches have therefore been proposed to introduce parallelism for functional languages [16], ranging from purely implicit parallelism to explicit, manual thread creation, communication, and load management. For the G P H variant of Haskell that is described here [29], and which targets a multithreaded parallel implementation, we have chosen a level of abstraction that hides most of the coordination aspects of the parallel execution, but that still enables the programmer to influence key aspects of thread creation. In this way it is possible to guide the granularity of G P H threads to avoid, for example, the creation of excessively fine-grained threads. The resulting semi-explicit parallel model of computation is elaborated below.
2.1.1 Semi-Explicit Parallelism Semi-explicit parallel languages [16] form an important class of notations, between explicit parallel languages where all coordination, communication, and control is made explicit (e.g., C extended with the MPI communications library or Concurrent ML [26]), and purely implicit notations where no parallelism control at all is provided (e.g., Sisal [27], Id [23], or NESL [9]). While aiming to provide high levels of abstraction and automatically managing many aspects of parallelism, as with purely implicit approaches, semi-explicit approaches require the programmer to include some directives to specify important aspects of the parallel coordination. Examples of such languages include Skil [10], Concurrent Clean [25], MultiLisp [14], and our own G P H notation. While semi-explicit approaches were often historically based around the use of annotations, more recent approaches, such as Caliban [18], provide compositional language constructs. This supports a more powerful, flexible, and often programmable, parallel programming methodology where (parallel) coordination is separated from (sequential) computation. Complete parallel programs are then orchestrated from lower-level components, which may themselves be broken down into parallel or sequential subcomputations, as required to implement the program. G P H [29] is a modest extension of Haskell with parallel and sequential composition as its two basic coordination primitives (Figure 2.1). Denotationally, both the par and seq constructs are projections onto the second operand, that is, the value of the first operand is not returned as the result of the construct, though it will usually be shared as part of some subcomputation in the second operand. Operationally,
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 50 2008-10-22 #6
50
Process Algebra for Parallel and Distributed Processing
par :: a → b → b seq :: a → b → b
—Parallel composition —Sequential composition
FIGURE 2.1: Types of the basic coordination constructs in G P H.
seq indicates that the first operand should be evaluated before the second operand, and par indicates that the first operand may be evaluated in parallel with the evaluation of the second. The latter operation is termed “sparking.” Unlike most parallel notations, the act of sparking an expression in G P H does not immediately force thread creation; rather the runtime environment determines which sparks are chosen to become parallel threads based on load and other information. It follows that programmers simply need to expose expressions in the program that they believe can usefully be evaluated in parallel. The runtime environment manages the details of parallel execution including thread creation, communication, workload balancing, etc., as described in detail in Section 2.5. Parallel implementations of G P H are publicly available from http://www.macs.hw.ac.uk/∼dsg/gph/. These two basic operations are then used to build higher-level constructs that help simplify parallel programming. Our early experience of implementing nontrivial programs in G P H showed that unstructured use of par and seq operators could lead to unnecessarily obscure programs. This problem can be overcome by using evaluation strategies [28]: nonstrict, polymorphic, higher-order functions that influence both the degree of evaluation of and the parallelism to be found in a G P H expression. Evaluation strategies provide a clean separation between coordination and computation. The driving philosophy behind evaluation strategies is that it should be possible to understand the computation specified by a function without considering its coordination. Figure 2.2 shows the basic operations that we provide for managing evaluation strategies. Strategies are defined as functions of type Strategy that take a polymorphic argument and return a unit value. The using construct then applies an evaluation strategy to a Haskell expression. We define two strategies for handling reduction of Haskell expressions to normal forms (NFs). The basic evaluation strategy rwhnf
type Strategy a = a → () using :: a → Strategy a → a
— Type of evaluation strategy — Strategy application
rwhnf :: Strategy a class NFData a where rnf :: Strategy a
— Reduction to weak head normal form — Class of reducible types — Reduction to normal form
FIGURE 2.2: Basic evaluation strategy operations in G P H.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 51 2008-10-22 #7
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
51
specifies that the associated expression is to weak head normal form (WHNF), in which no reduction is performed either under λ-expressions or within data structures. This corresponds to the default nonstrict evaluation order used by Haskell. The overloaded rnf strategy correspondingly specifies that the associated expression is to be reduced to full NF, representing a maximal evaluation degree. We provide instantiations of this strategy for all major standard Haskell types.
2.2 From Algebraic Specification to Executable Code Functional programming languages are designed to provide a high level of abstraction, focusing on what should be computed without committing the machine as to how the computation should be organized. This design principle has made functional languages a popular choice as execution languages for algebraic specifications. The main language features that contribute to this high level of abstraction are higherorder functions, polymorphism, and advanced mechanisms for modularisation, such as ML’s functors or Haskell’s type classes. Tight links between algebraic specification languages and execution languages can be identified in Extended ML [17], which maps down to ML, or in S PECTRUM [11], which uses advanced features such as type classes as found in Haskell. In our work, we exploit the proximity between algebraic specification languages and modern functional programming languages by attempting a direct mapping of the specification into Haskell code. In many cases, the algebraic specification can already be considered an executable specification without the need of further refinement steps. For more complex specifications, the process of mapping it to executable code may yield a number of proof obligations, stated in the axioms of the specification language. The proof obligations may be discharged using tools, including general theorem provers such as Isabelle or Coq, or specialized provers, such as Sparkle. Our model of software development does not tie the development to any particular tool, and the proof obligations may even be discharged manually. Some advanced language features provided by Haskell facilitate the mapping of a specification into a program. In particular, the sophisticated typing mechanism in Haskell, including type classes and functional dependencies, provides a powerful tool. Recent developments into the direction of dependent types [5] offer even more power to encode value-dependent properties into the type of a function, and move the programming language even closer to a specification language. These latest research directions aside, our current practice in mapping specification to code is, however, one of manually refining the specification to bring it into a rule-based format, suitable for a Haskell-like language. The proof obligations imposed by the axioms in the specification are proven either manually or with the help of a specialized theorem prover. With respect to the parallel execution of the resulting code, a functional language like G P H avoids the redundant sequentialization found in imperative languages. Nor do we mandate the specification of a parallelism structure in early stages of the
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 52 2008-10-22 #8
52
Process Algebra for Parallel and Distributed Processing m | n m⊥n ϕ(i) sumEuler(n)
= = = =
∃k ∈ N. m × k = n ¬ ∃k ∈ N. 1 < k ∧ k | m ∧ k | n | {m ∈ N | m < i ∧ m ⊥ i} | n i=1 ϕ(i)
FIGURE 2.3: Specification of the Euler totient function ϕ. refinement. As demonstrated by the running example used in this chapter, we generate rule-based code, which does not commit to a particular evaluation order. We then identify possible sources of parallelism and specify the parallel coordination using evaluation strategies [28]. Finally, we can exploit equational reasoning at this level to improve the parallel performance of the resulting program. It should be emphasized that this entire process is architecture-independent and Section 2.6 demonstrates good parallel performance of G P H applications on varying architectures from shared-memory machines to wide-area grid networks. As a running example, we will use a simple algorithm that computes the sum of the values of the Euler totient function for integers in the range from 1 to n, for some given n. The Euler totient function of a given integer i is the number of integers less than i that are relatively prime to i, as shown in Figure 2.3. Figure 2.4 shows the sequential Haskell code that implements this example. It directly maps the specifications in Figure 2.3 to the functions relprime, euler and sumEuler. In the code for relprime the following property is exploited: ∀ m n. gcd m n = 1 =⇒ m ⊥ n. Each function is preceded by a type signature, for example, mkList :: Int → [Int] specifies that mkList is a function that takes a fixed precision integer, of type Int, and returns a list of integers, of type [Int]. The notation [1..(n−1)] defines the
mkList :: Int → [Int] mkList n = [1..(n-1)] gcd :: Int → Int → Int gcd x 0 = x gcd x y = gcd y (rem x y) relprime :: Int → Int → Bool relprime x y = gcd x y == 1 euler :: Int → Int euler n = length (filter (relprime n) (mkList n)) sumEuler :: Int → Int sumEuler = sum . (map euler) . mkList
FIGURE 2.4: Sequential Haskell definition of sumEuler.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 53 2008-10-22 #9
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
53
sumEulerPar1 n = sum ((map euler (mkList n)) ‘using‘ parList rnf) sumEulerPar2 :: Int → Int → Int sumEulerPar2 c n = sum ([sum (map euler x) | x ← splitAtN c (mkList n)] ‘using‘ parList rnf) FIGURE 2.5: Two parallel versions of sumEuler in G P H.
list of integers between 1 and n − 1. In the definition of the greatest common divisor (gcd) we use the remainder function on integers (rem). The higher-order function filter selects all elements of a list that fulfil a given predicate, and map applies a function to all elements of a list. Figure 2.5 shows two simple parallel implementations of sumEuler that use evaluation strategies. The first parallel version, sumEulerPar1, simply applies the parList rnf strategy to the elements of the result list. The parList strategy is a parameterized strategy for lists that applies its argument (another strategy) to each element of the list in parallel. In this case, we will evaluate each element to full NF using a different thread. The parList strategy can easily be defined using the G P H par construct, as shown in Figure 2.6. The definition applies the argument strategy, strat, to each element of the list x:xs in parallel, returning the unit value () once the entire list has been traversed (: is list cons). The second parallel version, sumEulerPar2, is more sophisticated. It first splits the list elements into groups of size c using the splitAtN function. A list comprehension is used to bind each of the groups to the variable x in turn. Here, the list comprehension syntax [ e | v ← l ] builds a list of expressions whose value is e. The expression e depends on the value of v, which is drawn from each element of the list l in turn. The inner sum of the Euler totients for each group is then calculated, before all these results are summed. The definition thus exploits associativity of the underlying + operator. The reason for splitting the indexes in this way is so that each group of c list elements may be evaluated in parallel by its own individual thread, thereby increasing granularity. We will revisit this example in Section 2.4, developing versions with improved parallel performance.
parList :: Strategy a → Strategy [a] parList strat [] = () parList strat (x:xs) = strat x ‘par‘ (parList strat xs) FIGURE 2.6: Definition of parList in G P H.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 54 2008-10-22 #10
54
Process Algebra for Parallel and Distributed Processing
2.3 Operational Semantics of GpH In this section, we will present a parallel operational semantics for G P H, defined in terms of the call-by-need evaluation of a parallel extension to the λ-calculus, G P H- CORE. By making our semantics explicit in describing the way threads are managed and stored, we are able to reason accurately about the behaviour of G P H programs in terms of both coordination and computation.
2.3.1 Operational Semantics Overview The G P H operational semantics is a two-level transition semantics. At the lower level, there are single-thread transitions for performing the ordinary evaluation of expressions through, e.g., β-reduction. All candidate single-thread steps are performed simultaneously and in lockstep. They are then combined into a parallel computation super-step using coordination relations defined at the upper level. We follow Launchbury’s seminal work [19] by using a heap to allow sharing and by requiring all closures to be constructed by some let-binding. Where Launchbury uses a big-step natural semantics, in order to properly model coordination issues, we have chosen to use a small-step computational semantics. G P H- CORE is a simple subset of G P H, comprising the untyped λ-calculus extended with numbers, recursive lets, sequential composition, seq, and parallel composition, par. Expressions are normalized so that all variables are distinct, and the second argument to application and the first argument to par must be variables. x, y, z ∈ Variable n ∈ Number e ∈ Expression e ::= n | x | e x | λx.e | let {xi = ei }ni=1 in e | e1 seq e2 | x par e
2.3.2 Heaps and Labeled Bindings Following Ref. [19], we use a heap of bindings of expressions to variables. To deal with parallelism, each binding also carries a label to indicate its state. Thus, heaps are partial functions from variables to expression/thread-state pairs: H, K ∈ Heap = Variable ◦→ (expression, state) α, β ∈ State α ::= Inactive | Runnable | Active | Blocked We write individual bindings with the thread state appearing as an annotation on the binding arrow, thus α
x → e
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 55 2008-10-22 #11
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
55
A binding is Active (A) if it is currently being evaluated; it is Blocked (B) if it is waiting for another binding before it can continue its own evaluation; and it is Runnable (R) if it could be evaluated, but there are currently insufficient resources to evaluate it. All other bindings are Inactive (I). Bindings therefore correspond to heap closures and labeled bindings correspond to parallel threads. This is a rather simplified model of parallelism compared with the actual GUM implementation (Section 2.5): we assume idealized parallelism, with no communication costs; and unlike the actual implementation, threads are created instantly and may be migrated at no cost. However, it serves to provide limits on possible parallelism in the actual implementation. The computational semantics is specified as a relation on heaps, H =⇒ H . This is, in turn, defined in terms of a notion of single thread transitions (Section 2.3.3) and a scheduling relation (Section 2.3.4). The parallel operational semantics then builds on this to describe a reduction sequence from an initial global configuration to a final global configuration: A
I
(H, main → e) =⇒ . . . =⇒ (H , main → v) where main identifies the root expression for the program. Values, v, are in weak head NF, that is v ::= n | λx.e
2.3.3 Single-Thread Transitions The transition relation −→ of Figure 2.7 describes the computational step taken by each active binding in the heap. The left-hand side in each rule represents a heap A
with the active binding distinguished by H : z → e. Multilabel bindings, such as RAB
x → e in the block2 rule mean that the state is one of R, A, or B but not I. The right-hand sides of the rules are heaps comprising only those bindings changed or created by that computation step. Let: The let rule populates the heap with new bindings. These bindings are inactive since under call-by-need they may not necessarily be evaluated. Variables and blocking: In the var and blocki rules, z is a pointer to another closure (called x). If x has already been evaluated to WHNF (the var rule), then z simply receives that value. The notation vˆ indicates that all bound variables in v are replaced with fresh variable names. If x is inactive and has not yet been evaluated (the block1 rule), then z blocks at this point and x joins the pool of runnable bindings. Finally, if x is not inactive (the block2 rule), then z blocks but x is unaffected. Application: Evaluating e x involves reducing e to a function abstraction using app and then substituting x for the bound variable y using subst. Seq: The seq rule initially evaluates e1 without considering e2 . When and if e1 is reduced to WHNF, its value (but not any changes to the heap) is discarded by the seq-elim rule and evaluation then proceeds to e2 .
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 56 2008-10-22 #12
56
Process Algebra for Parallel and Distributed Processing I
A
A
H : z → let {xi = ei }ni=1 in e −→ ({xi → ei }ni=1 , z → e) I
A
A
(H, x → v) : z → x −→ (z → v) ˆ I
A
R
(var)
B
(H, x → e) : z → x −→ (x → e, z → x) RAB
A
B
(H, x → e) : z → x −→ (z → x) A
A
(subst)
α
H : z → e −→ (K, z → e ) A
(app)
α
H : z → e x −→ (K, z → e x) A
A
H : z → v seq e −→ (z → e) A
(seq-elim)
α
H : z → e1 −→ (K, z → e1 ) A
α
H : z → e1 seq e2 −→ (K, z → e1 seq e2 ) RAB
A
A
A
R
(H, x → e1 ) : z → x par e2 −→ (z → e2 ) I
(block1) (block2)
H : z → (λy.e) x −→ (z → e[x/y]) A
(let)
(seq)
(par-elim) A
(H, x → e1 ) : z → x par e2 −→ (x → e1 , z → e2 )
(par)
FIGURE 2.7: Single thread transition rules. Par: The par rule potentially introduces parallelism, i.e., suggests that an inactive binding could be made active by putting it into a Runnable state which may be promoted to Active later if sufficient resources are available. Nothing needs to be done if the binding is not inactive (par-elim).
2.3.4 Multithread Transitions and Scheduling The changes required for all active bindings are combined by the parallel rule (Figure 2.8) to create a full heap-to-heap transition. This is the key point in the semantics where reductions are carried out in parallel. We write H A to represent
A
H A = { xi → ei }ni=1
A
{H : xi → ei −→ Ki }ni=1 p H =⇒ H [ ni=1 Ki ]
FIGURE 2.8: Combining multiple thread transitions.
(parallel)
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 57 2008-10-22 #13
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH RA
B
RA
u
R
(H, x → v, z → ex ) −→ (H, x → v, z → ex ) RA
d
(unblock)
I
(H, x → v) −→ (H, x → v)
(deactivate)
|H A | < N R
a
57
(activate)
A
(H, x → e) −→ (H, x → e) FIGURE 2.9: Single-thread scheduling rules. A
all the active bindings in H , i.e., H A = {x → e ∈ H }. Hence in Figure 2.8 there are precisely n active bindings in H . The notation H [K] updates heap H with all new or changed bindings given by K. A more precise definition, together with a proof that conflicts do not arise between bindings can be found in Ref. [6]. The scheduling actions for individual threads are defined in Figure 2.9 as follows: (i) any binding that is immediately blocked on a completing thread is made runnable (unblock); (ii) any active or runnable binding that is in WHNF is made inactive (deactivate); and (iii) as many runnable bindings as resources will allow are made active (activate). In the unblock rule, the notation ex represents an expression that is immediately blocked on x, i.e., one of the three forms: ex ::= x | x y | x seq e Note that in the activate rule, N is a parameter to the semantics, indicating the total number of processors. This ensures that no more than N bindings are activated in any step. These rules do not, however, specify which bindings are activated: bindings are chosen nondeterministically during the activation phase. Since, however, this choice is at the coordination level, it does not change the actual values that are computed. The rules of Figure 2.10 extend the scheduling rules to multiple bindings. To achieve the maximum possible parallelism with respect to the available processors, it is necessary that all candidate threads are unblocked before deactivation and that deactivation takes place before activation. This sequence of actions is captured by the schedule relation of Figure 2.11.
†
H =⇒ H † −→∗
if:
H
†
i) H and ii) there is no H
such that H −→ H
. † ( is u, d or a.) FIGURE 2.10: Component scheduling relations.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 58 2008-10-22 #14
58
Process Algebra for Parallel and Distributed Processing s
=⇒ =
a
d
u
=⇒ ◦ =⇒ ◦ =⇒
=⇒ =
s
p
=⇒ ◦ =⇒
(schedule) (compute)
FIGURE 2.11: Overall scheduling and computation relations.
2.3.5 Computation Relation Finally, our full semantic computation relation, compute, is defined as a parallel p s transition =⇒ followed by a scheduling of bindings =⇒ (Figure 2.11). This ordering ensures that the heaps that appear in a reduction sequence are always fully scheduled. Since our semantics is parameterised on the number of processors, we decorate the computation relation with the number of processors where necessary: =⇒, so =⇒ N
1
indicates the single-processor case.
2.3.6 Properties of the Operational Semantics Abramsky’s denotational semantics of lazy evaluation [1] models functions by a lifted function space, thus distinguishing between a term (a nonterminating computation) and λx. to reflect the fact that reduction is to weak head NF rather than head NF. This is a widely used, simple, and abstract semantics. The properties and results developed in this section are expressed relative to this denotational semantics. Launchbury [19] shows a number of results relating his natural semantics of lazy evaluation to Abramsky’s denotational semantics. We borrow much of his notation and several of our proofs are inspired by him. Previously we showed that the 1-processor case of our semantics corresponds to Launchbury’s. There are three main properties that we expect of our semantics: soundness: the computation relation preserves the meanings of terms; adequacy: evaluations terminate if and only if their denotation is not ⊥; determinacy: the same result is always obtained, irrespective of the number of processors and irrespective of which runnable threads are chosen for activation during the computation. The denotational semantics of our language is given in Figure 2.12. The Val domain is assumed to contain a lifted version of its own function space. The lifting injection is lift and the corresponding projection is drop. The semantic function: [[. . .]] : Exp → Env → Val naturally extends to operate on heaps, the operational counterpart of environments: {{. . .}} : Heap → Env → Env The recursive nature of heaps is reflected by a recursively defined environment: {{x1 → e1 . . . xn → en }}ρ = μρ .ρ[x1 → [[e1 ]]ρ . . . xn → [[en ]]ρ ]
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 59 2008-10-22 #15
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
59
ρ ∈ Env = Var → Val [[λx.e]]ρ = lift λ .[[e]]ρ[x → ] [[e x]]ρ = drop([[e]]ρ )([[x]]ρ ) [[x]]ρ = ρ(x) [[let {xi = ei }ni=1 in e]]ρ = [[e]]{{x1 →e1 ...xn →en }}ρ ⊥ if [[e1 ]]ρ = ⊥ [[e1 seq e2 ]]ρ = [[e2 ]]ρ otherwise [[x par e]]ρ = [[e]]ρ FIGURE 2.12: Denotational semantics. We also require an ordering on environments: if ρ ≤ ρ then ρ may bind more variables than ρ but they are otherwise equal. That is: ∀x . ρ(x) = ⊥ ⇒ ρ(x) = ρ (x) The arid environment ρ0 takes all variables to ⊥. 2.3.6.1 Soundness Our computational relation H =⇒ H can be considered sound with respect to the denotational semantics in Figure 2.12 if the denotations of all the bindings in H are unchanged in H . The ≤ ordering on environments neatly captures this notion. PROPOSITION 2.1 If H =⇒ H then for all ρ, {{H }}ρ ≤ {{H }}ρ. PROOF
Induction on the size of H and the structure of expressions.
2.3.6.2 Adequacy We wish to characterize the termination properties of our semantics and Propositions 2.2 and 2.3 show an agreement with the denotational definition. The proofs are modeled on the corresponding ones in Ref. [19]. PROPOSITION 2.2 A I If (H, z → e) =⇒∗ (H , z → v) then [[e]]{{H }}ρ = ⊥. PROOF For all values v, [[v]]{{H }}ρ [[e]]{{H }}ρ = ⊥.
=
⊥ so by Proposition 2.1
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 60 2008-10-22 #16
60
Process Algebra for Parallel and Distributed Processing
PROPOSITION 2.3 A I If [[e]]{{H }}ρ = ⊥, there exists H , z, v such that (H, z → e) =⇒∗ (H , z → v). A proof of Proposition 2.3 is outlined in Ref. [6]. It is closely based on the corresponding proof in Ref. [19], working with a variant of the denotational semantics which is explicit about finite approximations. 2.3.6.3 Determinacy We now turn to the question of obtaining the same result irrespective of the number of processors and irrespective of which runnable threads are chosen for activation during the computation. Clearly, since the results above hold for any number of processors it follows that if an evaluation with N processors gives main a value then, depending on which threads are activated, an evaluation with M processors can give the same result in the sense of Proposition 2.1. a However, a consequence of the definition of =⇒ is that the main thread may be left runnable but never progress. It is possible that the main thread could be delayed or suspended indefinitely, if there is a constant supply of unneeded speculative threads being generated and scheduled in place of the main thread. This corresponds to the implementation of G P H, with the management of speculative evaluation the programmer’s responsibility [29]. It is possible to define an alternative activation a
relation =⇒ that requires that a runnable thread on which main is blocked (in a transitive sense) will be activated in preference to other runnable threads. We can be sure that there will always be a free processor in this circumstance because the blocking action has made one available. With this version of the activation relation, we can show that if any evaluation gives an answer for main then they all do, irrespective of the number of processors. a
For the 1-processor case, it is clear that the definition of =⇒ in Figure 2.13 ensures that there is always exactly one active binding and that the blocked bindings form a chain from main to that active binding. a
H =⇒ H 1. H
a −→∗
if:
H ; a
2. there is no H
such that H −→ H
and 3. req(main, H ) is active in H .
req(x, K) =
⎧ ⎨x,
RA
if x → e ∈ K
B y ⎩req(y, K), if x → e ∈K
FIGURE 2.13: Stronger activation relation.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 61 2008-10-22 #17
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
61
The following proposition demonstrates that all the closures activated in the one processor case will also be activated in the multiprocessor case. Recall that =⇒ is N
the computation relation assuming a maximum of N processors. PROPOSITION 2.4 Given N ≥ 1 processors, suppose A
(H, main → e) =⇒ H1 =⇒ H2 . . . and 1
1
A
(H, main → e) =⇒ K1 =⇒ K2 . . . N
N
If x is active in some Hi then there is a j such that x is active in Kj . PROOF
a
B
Suppose zk is active in some Hi . By =⇒ there is a chain main →
B
B
A
ez1 , z1 → ez2 , z2 → ez3 , . . . zk → e in Hi . By induction on the length k of this chain we can show that there must be some Kj where zk is active in Kj . Finally we can bring all these results to bear to prove that evaluation is deterministic in the sense that we get the same answer every time, for any number of processors, a
assuming the =⇒ activation relation. COROLLARY 2.1 A I For any number of processors N ≥ 1, if (H, main → e) =⇒∗ (H , main → v) 1
A
and (H, main → e) =⇒ K1 =⇒ K2 . . . then: N
N
I
1. there is some i ≥ 1 such that Ki = (Ki , main → v ); 2. [[v ]]{{Ki }}ρ0 = [[v]]{{H }}ρ0 PROOF 1. If there is no such Ki then main must remain active or blocked forever. In A
either case there must be some binding z → e that remains active and does not terminate. In that case the denotation of e in the context of the corresponding heap must be ⊥ by Proposition 2.3. But by Proposition 2.4 at some stage in the 1-processor evaluation z will be active and main will be (transitively) blocked on z. By Proposition 2.2 e will not reach a WHNF so main will remain blocked. (Unless main = z in which case the result follows immediately.) A
I
2. {{H, main → e}}ρ0 ≤ {{H , main → v}}ρ0 by Proposition 2.1, so in particular [[v]]{{H }}ρ0 = [[e]]{{H }}ρ0 . Similarly, [[v ]]{{Ki }}ρ0 = [[e]]{{H }}ρ0 .
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 62 2008-10-22 #18
62
Process Algebra for Parallel and Distributed Processing
2.4 Program Equivalences and Reasoning We will now demonstrate how we can perform program transformations on parallel programs in order to improve performance. We will use the semantic properties of the G P H par and seq constructs, defined in Section 2.3, to derive versions of the program that expose improved parallel behaviour. Our overall goal is to reduce the total execution time on some specific parallel platform. In working toward this goal, we will increase the degree of parallelism that can be found in the program by judicious introduction of evaluation strategies. However, since unlike the ideal situation considered by our operational semantics, in the real-world the costs and overheads of parallel execution and communication mean that maximal parallelism does not automatically lead to a minimal runtime. We continue with the sequential version of sumEuler below. Note that the three main worker functions (sum, map euler, and mkList) have been composed into a three-stage sequential pipeline using the function composition operator (.). sumEuler :: Int → Int sumEuler = sum . map euler . mkList
Despite the simplicity of this program, it is a good example because it exhibits a typical structure of symbolic applications: it uses fairly small auxiliary functions that are combined with function composition and higher-order functions. The overall program structure is a fold-of-map (where the sum function is a fold). Operationally, this structure suggests two possible ways of parallelisation: producer–consumer (or pipeline) parallelism, and data parallelism.
2.4.1 Pipeline Parallelism The function compositions give rise to producer–consumer (or pipeline) parallelism, where the initial producer (mkList) runs in parallel with its consumer (map euler), and this runs in parallel with the final consumer of the list (sum). We can specify pipeline parallelism by using a parallel variant of the function composition operator (.||), which computes both the producer and the consumer function in parallel. The sumEuler function can then be written as sumEuler =
sum .|| map euler .|| mkList
where the pipeline operator is defined as (f .|| g) x = let { x’ = g x ; y = f x’ } in x’ ‘par‘ y While pipeline parallelism can be easily expressed, in this case it is not very efficient, since the tight connection between producer and consumer leads to frequent synchronization between the generated threads, which can consequently result in poor performance. This behavior can be improved by attaching a strategy to the .|| combinator, ensuring that the producer generates whole blocks of list elements and thus reduces
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 63 2008-10-22 #19
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
63
the amount of synchronization. In order to further increase parallelism, we can use a parallel evaluation strategy to not only specify evaluation degree but also parallelism on the result of the producer. Here we use parallel, strategic function application $||, which applies a strategy to the argument of a function: sumEuler n = sum $|| (parList rnf) $ map euler $|| (parList rnf) $ mkList n
This is a mixture of pipeline and data parallelism, which we study in more detail in the following section. Notably, this pipeline construct makes it possible to specify parallelism on the top level, when combining sequential code: although mkList n is a sequential function, the application of parList rnf triggers the parallel evaluation of all list elements. For large-scale programming this approach of specifying parallelism when combining functions, rather than when defining them, reduces and localizes the amount of necessary code changes (see Ref. [22] for a more detailed discussion).
2.4.2 Data Parallelism More promising in this example is to use data parallelism, where the same operation is applied to multiple components of a data structure in parallel. In this case, the euler function, which is mapped over the list returned by mkList, is a good candidate for data parallelism. We can define a parallel variant of the map operation that builds on the parList strategy, and which abstracts over the pattern of parallelism defined in sumEulerPar1, as follows: parMap :: Strategy b → (a → b) → [a] → [b] parMap strat f xs = map f xs ‘using‘ parList strat
This clearly shows how higher level parallel constructs can be constructed in a layered manner, in order to maintain clear separation between coordination and computation: it is obvious from the definition that the value returned from a call to parMap is identical to a sequential map. We can use this equivalence to simply replace the map euler expression by parMap s euler, where s describes the evaluation degree on the elements of the result list. In this case, we choose rnf to increase the granularity of the generated tasks. Examining the behavior of this data-parallel code we find a large number of very fine-grained threads, since every list element gives rise to a new thread. In order to improve performance further, we need to combine computations on neighboring list elements into one, coarse-grained thread, as in the definition of parSumEuler2. We call this process clustering of evaluation and demonstrate how it can be derived from this code, and how it improves parallel performance.
2.4.3 Strategic Clustering Clustering is easily introduced into functions that return a list or other collection. For example, it is possible to construct a strategy that captures the splitting technique
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 64 2008-10-22 #20
64
Process Algebra for Parallel and Distributed Processing
parListChunk :: Int → Strategy a → Strategy [a] parListChunk c strat [] = () parListChunk c strat xs = seqList strat (take c xs) ‘par‘ parListChunk c strat (drop c xs)
FIGURE 2.14: A clustering strategy used in sumEuler. used in the definition of sumEulerPar2 above. The parListChunk strategy of Figure 2.14 introduces a number of parallel threads to operate on subsequences, or chunks, of an input list. Here, each thread evaluates a sublist of length c. The seqList strategy is a sequential analogue of parList, applying its argument strategy to every element of the list in sequence. Using a parListChunk c s xs strategy will therefore generate kc potential threads, where k is the length of the list xs. The expression map f xs can now be clustered as follows. parMapChunk c strat f xs = map f xs ‘using‘ parListChunk c strat
As before, the advantage of a purely strategic approach is that the clustered coordination operations can be captured entirely by the evaluation strategy and isolated from the computation.
2.4.4 Cluster Class As defined in Section 2.4.3, clustering works on the results of a function. However, many common functions do not return a list or other collection as their result. One common example is a fold function, that collapses a collection to a single value. In order to cluster such functions, we introduce a generic Cluster type class with functions to cluster the input data, lift the function to operate on the clustered data, and perhaps decluster the result. Although clustering changes the computation component of a parallel program, equivalence to the sequential program is maintained by introducing clustering systematically using semantics-preserving identities. We will discuss such transformations in the following sections. To be more precise, we require that every collection type c that is to be clustered is an instance of the Haskell MMonad class, as shown by the subclass dependency for Cluster below. We use a formulation of monads based on the functions munit, mjoin, and mmap [32], which is more suitable for our purposes than the usual Kleisli category (see Section 10.4 of Ref. [7]) with return and bind operations that is used in many of the standard Haskell libraries. class MMonad c where munit :: a → c a mjoin :: c (c a) → c a mmap :: (a → b) → (c a → c b)
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 65 2008-10-22 #21
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
65
We introduce a new Haskell class, Cluster, parametrized by the collection type c with four operations: singleton turns a collection into a 1 element collection of collections; the generalized variant cluster n maps a collection into a collection of subcollections each of size n; decluster flattens a collection of collections into a single collection; and lift takes a function on c a and applies it to a collection of collections. For singleton, decluster, and lift we can use existing definitions in the MMonad class to provide default definitions. class (MMonad c) => Cluster c where singleton :: c a → c (c a) cluster :: Int → c a → c (c a) decluster :: c (c a) → c a lift :: (c a → b) → (c (c a) → c b) singleton = munit decluster = mjoin lift = mmap All instances of the monad class come with proof obligations for the monad identities (see Rules (I)–(III), (i)–(iv) of Ref. [32]). From these identities we obtain the following equations relating lift, decluster, and singleton. decluster ◦ singleton = id decluster ◦ lift singleton = id decluster ◦ decluster = decluster ◦ lift decluster lift id lift (f ◦ g) lift f ◦ singleton lift f ◦ decluster
(M I) (M II) (M III)
= id = (lift f ) ◦ (lift g) = singleton ◦ f = decluster ◦ lift (lift f )
(M i) (M ii) (M iii) (M iv)
We further require for each n that cluster n is a one-sided inverse of decluster. decluster ◦ cluster n = id
(C I)
We now examine the properties of functions that modify the structure of the base domain. We call a function malg : : c a → a an (Eilenberg–Moore) algebra for the monad c if the following two identities hold malg ◦ munit = id malg ◦ mmap malg = malg ◦ mjoin
(A I) (A II)
The identities for an algebra can be shown as the two commuting diagrams. munit / ca a OO OOO OOO malg O id OOOO' a
c (c a) mjoin
ca
mmap malg
/ca
malg
/a
malg
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 66 2008-10-22 #22
66
Process Algebra for Parallel and Distributed Processing
Given two algebras α : : c a → a and β : : c b → b, a homomorphism between them is a function f : : a → b such that f ◦ α = β ◦ mmap f .
2.4.5 Transforming Clustered Programs The categorical identities on collections are useful for transforming clustered programs. This section discusses the two main identities we use (lift1 and lift2), and shows that they follow from those of monads and algebras stated in Section 2.4.4, together with the identity for cluster. We note that, for every a, the function mjoin : : c (c a) → c a is an algebra for the monad c (called the free algebra on a), and that, for every f : : a → b, mmap f is an algebra homomorphism between these free algebras. Clustering can be introduced into a program by the following identity that holds for all algebra homomorphisms, f : : c a → c b. Algorithmically the right-hand side splits the input into clusters, applies f to every cluster and flattens the result. f = decluster ◦ lift f ◦ cluster n c (cO a)
lift f
/ c (c b)
f
cluster n ca
(lift1)
decluster
/cb
Recall that mmap = lift and mjoin = decluster. Since f is an algebra homomorphism we know decluster ◦ lift f = f ◦ decluster, and (lift1) follows from (C I). The (A II) identity can be used as the following rewrite rule in order to apply the function malg to all clusters before combining the results using malg again. f = f ◦ lift f ◦ cluster n
(lift2)
Again, using mmap = lift and mjoin = decluster, this identity has an easy proof. Since malg is an algebra for the monad, the identity malg ◦ lift malg = malg ◦ decluster holds by (A II), and (lift2) follows again from (C I).
2.4.6 Strategic Clustering Version of sumEuler We will now exploit this generic clustering mechanism to provide an improved parallel definition of sumEuler. We first summarize the performance analysis of the (unclustered) data parallel version (see also Section 2.6). In the map-phase of sumEuler it is easy to exploit data parallelism, by computing every euler function in parallel. However, an unclustered version such as this will generally yield a large number of very fine-grained threads, resulting in a speedup that may be close to one. We have already seen how to improve the granularity of the algorithm by arranging for a whole list chunk to be computed by just one thread, using a parListChunk
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 67 2008-10-22 #23
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
67
sumEuler :: Int → Int → Int sumEuler c n = sum (map euler (mkList n) ‘using‘ parListChunk c rnf ) FIGURE 2.15: Strategic clustering version of sumEuler. strategy. We now apply this strategy to the sumEulerPar1 code in Figure 2.5 and arrive at a strategic clustering version of sumEuler, shown in Figure 2.15. This takes the cluster size c as an additional parameter and applies the parListChunk evaluation strategy to the inner expression. This then generates the list of result values that should be summed. Unfortunately, measurements in Ref. [21] show that the parallel performance of the resulting algorithm is still rather unsatisfactory on typical tightly connected networks. While sufficient parallelism is generated early on in the program, the sequential fold at the end of the computation becomes a serious bottleneck, giving typical speedups of only around a factor of 4 on a 16-processor network. We can identify two reasons for this poor performance. First, the sum computation is sequential and this will therefore inevitably generate a sequential tail to the computation, even if it is partially overlapped by the parallel map. Second, both the argument to and the result of each parallel thread is a list, and these may require a relatively large amount of time to communicate where large chunks are concerned.
2.4.7 Generic Clustering Version of sumEuler The derivation of a generic clustering version of sumEuler proceeds as follows. First we observe that a list is a monad with list-map as the mmap and list-append as the mjoin function. Since sum is defined as fold (+) 0 in the Haskell prelude, it is an algebra over lists. Finally, map euler is an algebra homomorphism between free algebras. sumEuler = = = =
sum ◦ map euler sum ◦ decluster ◦ lift (map euler) ◦ cluster z sum ◦ lift (sum) ◦ lift (map euler) ◦ cluster z sum ◦ lift (sum ◦ map euler) ◦ cluster z
(unfold) (lift1) (A II) (M ii)
The transformed sumEuler code in Figure 2.16 retains separation between the algorithmic and coordination code. In particular, the sequential code can be used as a worker function, and the clustering operations are wrapped around the (lifted) worker in order to achieve an efficient parallel version. This version exhibits a much improved average parallelism on the same modern 16-processor network as before. The sequential tail in the computation has been vastly reduced, to about 15% of the total runtime compared with about 80% of the total runtime in the strategic clustering version. As a consequence of this improved
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 68 2008-10-22 #24
68
Process Algebra for Parallel and Distributed Processing
sumEuler :: Int → Int → Int sumEuler z n = sum ((lift worker) (cluster z (mkList n)) ‘using‘ parList rnf) where worker = sum . map euler FIGURE 2.16: Clustered version of sumEuler. behavior we achieve a relative speedup of 14.3 in a 16-processor configuration. A detailed performance comparison can be found in Ref. [21]. On larger parallel machines it might be advantageous to use a different clustering scheme, which collects every zth element of the list into one block in order to achieve a better load balance. Notably, such a change would effect only the definition of (de)cluster but not the code of sumEuler itself. In summary, we have seen how a (manual) program transformation process, controlled by basic monad and algebra laws, was able to significantly improve the performance of a data parallel algorithm. The developed clustering is generic and can be applied to arbitrary algebras and monads. The parallel performance obtained for the unclustered, and both strategic and generic clustered, versions of the program is reported in Section 2.6.
2.5 Implementation of GpH G P H is implemented by the GUM runtime environment [29], a multithreaded parallel implementation that has been steadily evolved since the early 1990s, and which targets a variety of parallel and distributed architectures ranging from multicore systems through shared-memory machines and distributed clusters to wide-area computational grids [4,30].
2.5.1 Implementation Overview GUM uses a virtual shared memory model of parallelism, implementing a parallel graph reduction mechanism to handle nonstrict parallel evaluation of G P H programs. In this model, the program constructs a graph structure during execution. Each node in the graph represents a possibly shared (sub)expression that may need to be evaluated. Nodes are evaluated if (and only if) they contribute to the result of the program, realizing lazy evaluation. Following evaluation, the graph node is updated with the value of the subexpression and thus sharing of results is preserved. Parallelism is introduced by creating threads whose purpose is to evaluate nodes that have been marked using the par construct. This may cause other nodes to be evaluated if they are linked as subexpressions of the main node that is evaluated by the thread. Following evaluation, each graph node that has been evaluated by a thread is updated with the result of the evaluation. If one or more threads depend on this result and have therefore been blocked waiting for the node to be evaluated, they may now be notified of the value, and unblocked. We will elaborate on this below.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 69 2008-10-22 #25
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
69
Since we use a virtual shared graph model, communication can be handled implicitly through accesses to globally shared nodes rather than through explicit system calls. When a thread needs the value of a globally shared node, and this is not available on the local processor, the processor which owns the master copy will be contacted. In the simplest case, when the master copy has already been evaluated, the value of the result is returned immediately, and recorded as a locally cached copy. Similarly, if the master copy has not yet been evaluated, it is returned in an unevaluated form to the requesting processor. The local node will now become the master copy, and the thread will continue by evaluating this node. If the master copy is currently under evaluation, the thread that requested the value becomes blocked until the result is produced, at which point it will be notified of the result. An important feature of the GUM implementation is the use of a local heap in addition to a global heap. A local heap is used to hold cached copies of global values that will not change in future (the use of a purely functional language ensures there are no cache coherence issues—once produced, the value of a graph node is fixed and will never change), and to hold a graph that is not reachable globally. Since the majority of a graph that is produced falls into the latter category, this is a major advantage. The dual-level approach allows fast independent local collection, integrated with a slower, but much less frequently used global collection mechanism, currently based around distributed reference counting.
2.5.2 Thread Management A thread is a virtual processor that executes a task to evaluate a given graph node. GUM threads are extremely lightweight compared to operating system constructs such as pthreads. They are implemented entirely within the language’s runtime environment, and contain minimal state: a set of thread-specific registers plus a pointer to a dedicated stack object. Threads are allocated to processing elements (PEs), which usually correspond to the cores or CPUs that are available on the target system. Each PE has a pool of runnable threads. At each scheduling step, the runtime scheduler selects one of these threads for execution. This thread then runs until either it completes, it blocks, or the system terminates as the result of an error condition (such as insufficient memory). This unfair scheduling approach has the advantage of tending to decrease both the space usage and the overall execution time [12], which is beneficial for parallel execution. However, it is not suitable for concurrency, or for handling speculative threads, since in both cases it is necessary to interleave thread execution. In G P H, parallelism is introduced using par constructs in the source program. When the expression e1 ‘par‘ e2 is evaluated, e1 is sparked for possible future evaluation, and then e2 is evaluated. Sparking involves recording the graph node associated with e1 in the current PE’s spark pool. At a future point, if there is insufficient workload, sparks may be selected from the spark pool and used to construct threads to actually evaluate the closure. Sparking a thunk (an unevaluated expression) is thus a cheap and lightweight operation compared with thread creation, usually involving adding only a pointer to the spark pool.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 70 2008-10-22 #26
70
Process Algebra for Parallel and Distributed Processing
2.5.3 Spark Stealing If a PE becomes idle, it will extract a previously saved spark from its local spark pool if it can, and use this to create a new thread. If the spark is no longer useful (e.g., because the node it refers to has already been evaluated by another thread), it will be discarded, and another one chosen for execution. If the process fails and there are no useful local sparks, then the PE will attempt to find work from another PE. It does this by generating a FISH message that is passed at random from PE to PE until either some work is found, or it has visited a preset number of PEs. If no work is found, then the message is returned to the originating PE, and after a short, tunable delay, a new FISH message is generated. If the PE that receives a FISH has a useful spark it sends a SCHEDULE message to the PE that originated the FISH, containing the corresponding graph node packaged with a tunable amount of nearby graph. The spark is added to the local spark pool, and an ACK message is sent to the PE that donated the spark.
2.5.4 Memory Management Parallel graph reduction proceeds on a shared program/data graph, and a primary function of the runtime environment of a parallel functional language is to manage the virtual shared memory in which the graph resides. In GUM, most sequential execution is exactly as in the standard sequential GHC implementation. This allows the GUM implementation to take advantage of all the sequential optimizations that have been built into GHC. Each PE has its own local heap memory that is used to allocate graph nodes and on which it performs local garbage collections, independently of other PEs. Local heap addresses are normal pointers into this local heap. Global addresses (GAs) are used to refer to remote objects. A GA consists of a (PE identifier, local identifier) pair. Each PE then maintains a table of in-pointers mapping these local identifiers to specific local addresses. The advantage of this approach over simply using local addresses as part of a GA is that it allows the use of a copying garbage collector, such as the generational collector used in GHC. The garbage collector treats the table of in-pointers as additional roots to the garbage collection. The corresponding local address is then updated to reflect the new location of the object following garbage collection. An important part of this design is that it is easy to turn any local address into a GA (so that it can be exported to another PE), simply by creating a new in-pointer. In order to allow in-pointers to be garbage collected, we use a weighted reference counting scheme [8], where GAs accumulate some weight, which is returned to the owner node if the remote copy of the node is garbage collected. If all weights are returned, the in-pointer becomes garbage, and the local node may be a candidate for recovery during garbage collection. The only garbage not collected by this scheme consists of cycles that are spread across PEs. We plan ultimately to recover these cycles, too, by halting all PEs and
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 71 2008-10-22 #27
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
71
performing a global collective garbage collection, but we have not yet found the need for this in practice, even in very long-lived applications.
2.6 Assessment of Parallel Performance Taking sumEuler as our running example, Figure 2.17 compares the relative speedups of the unclustered version and both strategic and generic clustering versions developed in Section 2.4. The figure reports results on a 16-node Beowulf cluster of Linux RedHat 6.2 workstations with 533 MHz Celeron processors and 128 MB of DRAM connected through a 100 Mb/s fast Ethernet switch. It shows that the unclustered version produces hardly any speedup at all, due to the extreme fine granularity of the generated parallelism. A naive strategic version, which combines the execution of neighboring elements, produces only speedups up to 3.7, mainly owing to the sequential sum operation at the end. The improved generic clustering version avoids this sequential bottleneck at the end and shows a good speedup of 14.3 on 16 processors. We have used our high-level program development approach on applications from different application areas, and Table 2.1 summarizes key aspects of some example programs. The first column shows the parallel paradigm. The second and third columns show the application domain and program name, respectively. The fourth and fifth columns present the total size of the program and size of the code specifying the parallel coordination, both measured in Source Lines of Code (SLOC). The last two columns report the maximal parallel performance and give a reference to detailed performance measurements. Parallel performance is measured as maximum relative speedup, i.e., speedup over the runtime of the parallel program executed on a single processor, on a given number of PEs. Speedup of sumEuler 16 Generic clustering Strategic clustering No clustering
14
Speedup
12 10 8 6 4 2 0
2
4
6
8
10
12
14
16
Processors
FIGURE 2.17: Speedups of different versions of sumEuler. (From Loidl, H-W., Trinder, P., and Butz, C., Parallel Process. Lett., 11, 471, 2001. With permission.)
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 72 2008-10-22 #28
72
Process Algebra for Parallel and Distributed Processing
TABLE 2.1: Program characteristics and performance summary. Applications
Parallel Paradigm
Data parallel
Size
Domain
Name
Code
Coord.
Max Speed Speedup/PEs
Ref.
Symbolic computation Graphics
linSolv smallGroup raytracer sumEuler matMult Queens
121 52 80 31 43 21
22 8 10 5 9 5
11.9/16 26.9/28 8.9/16 9.1/10 3.4/4 7.9/16
[20] [2] [20] [31] [20] [20]
47k
13
1.4/2
[22]
5k
6
2.5/5
[22]
Numeric Divide and conquer
AI
Nested parallelism
Natural lang.processing
Lolita Naira
A key aspect of G P H programming reflected by the fifth column of Table 2.1 is that the parallelization requires only small code changes, localized in only a few modules. This is in stark contrast to the pervasive changes required by a lower-level parallel programming models. The applications require a range of parallelism paradigms. The majority of the applications are symbolic in nature and exhibit irregular parallelism, i.e., both varying numbers and sizes of tasks. Hence the parallelism results should not be directly compared with more regular problems where near-optimal parallelism is achievable. The results show that, despite their very different computational structures, G P H delivers respectable parallel performance for all of the programs and on widely varying architectures, from modern multicores to very high latency computational Grids [4,22,30]. The performance results show that we can obtain good parallelism on local networks, such as Beowulf clusters, e.g., 26.9 on 28PEs [22]. More recent results exhibit almost linear speedup on an 8-core machine [15], and acceptable performance on a wide-area network composed of three local networks: we achieve speedups of 7 on 7 nodes, and up to 16 on 41 nodes for a challenging graphics application [3,4]. Thus, the parallel performance of G P H scales well even on heterogeneous high-latency architectures. We attribute the largely architecture-independent nature of the achieved speedup to the high-level of abstraction provided by our parallel programming model. Unlike most programming models it does not tie a concrete parallel implementation to the specifics of the underlying architecture. Rather, a parallel runtime environment is in charge of coordinating the parallelism by adjusting the coordination to dynamic properties such as the current workload.
2.7 Conclusion On the one hand, the inherent complexity of designing parallel systems calls for a high-level approach of specifying its behavior. On the other hand, one of the main goals for using parallelism is to improve performance through the coordinated use of many processors on the same application. Thus, an efficient implementation of the specified parallelism is almost as crucial as the correctness of the implementation
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 73 2008-10-22 #29
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
73
itself. To reconcile the tension between high-level design and low-level performance, we use G P H, a parallel extension of Haskell, as both specification and as programming language. As the former, it is a useful intermediate step from a high-level algebraic specification to tuned executable code. In this role, it exploits the formal foundations of the purely functional programming language Haskell, and enables the programmer to extensively use code transformations to modify the degree of parallelism in the code. As the latter, it profits from an efficient sequential implementation (based on the highly optimizing GHC), augmented with advanced concepts such as a virtual shared heap, yielding a system of efficient, largely architecture-independent parallelism. We have demonstrated our high-level program development approach on applications from a range of application areas such as numerical analysis, symbolic computation, and natural language processing. These applications use a range of parallelism paradigms and deliver good parallel performance on a range of widely varying architectures, from modern multicores to very high latency computational Grids [4,22,30]. We have shown that G P H parallelization requires minimal, local refactoring rather than the pervasive changes required by lower-level approaches. We attribute the largely architecture-independent performance of G P H to its high-level parallel programming model. In Ref. [20] we have outlined how G P H’s performance compares favorably with both conventional parallel technologies and with other parallel functional languages. Our experience in realizing numerous parallel applications in various application areas underlines that efficient parallel code can be developed through a transformation-based programming methodology. In this process, a set of tools for controlling and examining the parallel behavior has proven indispensable. Our measurement results indicate good performance both on shared-memory machines and clusters of workstations. The emerging architectures of large-scale, computational grids on the one hand, and multicore, parallel machines on the other hand, are a hard stress-test for the underlying parallel runtime environment. Even though we already achieve acceptable performance on both of these new kinds of architectures, we are currently enhancing the features of the runtime environment to better deal with these new architectures, by adding support for dynamically adapting the chosen policies for work distribution and scheduling.
References [1] S. Abramsky. The lazy lambda calculus. In Research Topics in Functional Programming, Reading, MA, Addison Wesley, 1990, pp. 65–117. [2] A. Al Zain, K. Hammond, P. Trinder, S. Linton, H.-W. Loidl, and M. Costanti. SymGrid-Par: Designing a framework for executing computational algebra systems on computational grids. In International Conference on Computational Science (2), Reading, MA. Lecture Notes in Computer Science, 4488:617–624, 2007.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 74 2008-10-22 #30
74
Process Algebra for Parallel and Distributed Processing
[3] A. Al Zain, P. Trinder, H-W. Loidl, and G. Michaelson. Supporting high-level grid parallel programming: The design and implementation of grid-GUM2. In UK e-Science All Hands Meeting, Nottingham, U.K., EPSRC, September 10– 13, 2007, pp. 182–189. [4] A. Al Zain, P. Trinder, G. Michaelson, and H.-W. Loidl. Evaluating a highlevel parallel language (GpH) for computational grids. IEEE Transactions on Parallel and Distributed Systems, 19(2):219–233, 2008. [5] D. Aspinall and M. Hofmann. Dependent types. In Advanced Topics in Types and Programming Languages, Cambridge, MA, MIT Press, 2005, pp. 45–86. [6] C. Baker-Finch, D. King, and P. Trinder. An operational semantics for parallel lazy evaluation. In ICFP’00—International Conference on Functional Programming, Montreal, Canada, ACM Press, September 2000, pp. 162–173. [7] M. Barr and C. Wells. Category Theory for Computing Science, New York, Prentice-Hall, 1995. ISBN 0133238091. [8] D.I. Bevan. Distributed garbage collection using reference counting. In PARLE’87—Parallel Architectures and Languages Europe, Eindhoven. Lecture Notes in Computer Science, 259:176–187, Eindhoven, June 12–16, 1987. [9] G.E. Blelloch. Programming parallel algorithms. Communications of the ACM, 39(3):85–97, March 1996. [10] G.H. Botorog and H. Kuchen. Skil: An imperative language with algorithmic skeletons for efficient distributed programming. In HPDC’96— International Symposium on High Performance Distributed Computing, Syracuse, New York, IEEE Computer Society Press, August 1996, pp. 243–252. [11] M. Broy, C. Facchi, R. Grosu, R. Hettler, H. Hußmann, D. Nazareth, F. Regensburger, and K. Stølen. The Requirement and Design Specification Language S PECTRUM: An Informal Introduction. Technical Report TUM 19311/2, Institut f¨ur Informatik, TU M¨unchen, 1993. [12] F.W. Burton and V.J. Rayward Smith. Worst case scheduling for parallel functional programming. Journal of Functional Programming, 4(1): 65–75, January 1994. [13] A.A. Chien. Parallelism drives computing. In Manycore Computing Workshop, Seattle, WA, June 2007. On-line proceedings. [14] R. Halstead. Multilisp: A language for concurrent symbolic computation. ACM Transactions on Programming Languages and Systems, 7(4):106–117, October 1985. [15] K. Hammond, A. Al Zain, G. Cooperman, D. Petcu, and P. Trinder. SymGrid: A framework for symbolic computation on the grid. In EuroPar’07—European Conference on Parallel Processing, Rennes, France. Lecture Notes in Computer Science, 4641:457–466, August 2007.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 75 2008-10-22 #31
Semi-Explicit Parallel Programming in a Purely Functional Style: GpH
75
[16] K. Hammond and G. Michaelson, (Eds.) Research Directions in Parallel Functional Programming. Springer, 1999. ISBN 1-85233-092-9. [17] S. Kahrs, D. Sannella, and A. Tarlecki. The definition of extended ML: A gentle introduction. Theoretical Computer Science, 173:445–484, 1997. [18] P.H.J. Kelly. Functional Programming for Loosely-Coupled Multiprocessors. Research monographs in parallel and distributed computing, Cambridge, MA, MIT Press, 1989. ISBN 0262610574. [19] J. Launchbury. A natural semantics for lazy evaluation. In POPL’93— Principles of Programming Languages, Charleston, SC, ACM Press, 1993, pp. 144–154. [20] H.-W. Loidl, F. Rubio Diez, N. Scaife, K. Hammond, U. Klusik, R. Loogen, G. Michaelson, S. Horiguchi, R. Pena Mari, S. Priebe, A. Rebon Portillo, and P. Trinder. Comparing parallel functional languages: programming and performance. Higher-order and Symbolic Computation, 16(3):203–251, 2003. [21] H.-W. Loidl, P. Trinder, and C. Butz. Tuning task granularity and data locality of data parallel GpH programs. Parallel Processing Letters, 11(4):471–486, December 2001. [22] H.-W. Loidl, P. Trinder, K. Hammond, S.B. Junaidu, R.G. Morgan, and S.L. Peyton Jones. Engineering parallel symbolic programs in GpH. Concurrency— Practice and Experience, 11(12):701–752, 1999. [23] R.S. Nikhil. The parallel programming language Id and its compilation for parallel machines. In Workshop on Massive Parallelism: Hardware, Programming and Applications, Amalfi, Italy, 1989. CSG Memo 313. [24] S.L. Peyton Jones, editor. Haskell 98 Language and Libraries: The Revised Report. Cambridge University Press, 2003. ISBN 0521826144. Available at http://www.haskell.org/. [25] M.J. Plasmeijr, M.C.J.D. van Eekelen, E. N¨ocker, and J.E.W. Smesters. The concurrent clean system—Functional programming on the MacIntosh. In International Conference of the Apple European University Consortium, Paris, 1991, pp. 14–24. [26] J.H. Reppy. Concurrent ML: Design, application and semantics. In Functional Programming, Concurrency, Simulation and Automated Reasoning, New York, Lecture Notes in Computer Science, 693:165–198, 1993. [27] S. Skedzielewski. Sisal. In Parallel Functional Languages and Compilers, Frontier Series, ACM Press, New York, 1991, pp. 105–158. [28] P. Trinder, K. Hammond, H.-W. Loidl, and S.L. Peyton Jones. Algorithm + strategy = parallelism. Journal of Functional Programming, 8(1):23–60, January 1998.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C002 Finals Page 76 2008-10-22 #32
76
Process Algebra for Parallel and Distributed Processing
[29] P. Trinder, K. Hammond, J.S. Mattson Jr., A.S Partridge, and S.L. Peyton Jones. GUM: A portable parallel implementation of Haskell. In PLDI’96— Programming Languages Design and Implementation, Philadelphia, PA, May 1996, pp. 79–88. [30] P. Trinder, H.-W. Loidl, E. Barry Jr., K. Hammond, U. Klusik, S.L. Peyton Jones, and A. Rebon Portillo. The multi-architecture performance of the parallel functional language GpH. In Euro-Par 2000—Parallel Processing, Munich. Lecture Notes in Computer Science, 1900:739–743, 2000. [31] P. Trinder, H.-W. Loidl, and R. Pointon. Parallel and distributed Haskells. Journal of Functional Programming, 12(4, 5):469–510, July 2002. Special Issue on Haskell. [32] P. Wadler. Comprehending Monads. Mathematical Structures in Computer Science, 2:461–493, 1992.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 77 2008-10-20 #3
Chapter 3 Refinement of Parallel Algorithms Fredrik Degerlund and Kaisa Sere Contents 3.1 3.2
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Refinement Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Specification Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Weakest Precondition Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Refinement of Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Refinement of Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Action Systems Formalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Parallel Programming with Action Systems . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1.1 Action Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1.2 Partitioned Action Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Derivation of Action Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Case Study: Integer Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Problem Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1.1 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1.2 Tackling the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1.3 Parallelization Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Derivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77 78 79 79 80 80 81 81 81 81 83 85 86 86 86 87 88 94 95
3.1 Introduction In this chapter, we describe how to design correct parallel algorithms using the refinement calculus framework. This is a framework that supports the stepwise derivation of algorithms from their abstract high-level description down to an implementation on some platform using correctness-preserving program transformations. The abstract high-level description of the target parallel algorithm can be, for instance, a sequential algorithm or a mathematical formula that can be used as an initial specification of the problem. We take the view that the developed algorithms will be executed on grid- or cluster-type of networked platforms or processor farms where we have some central resource coordinating the computation, while the computation itself is divided into mutually independent tasks which can be executed
77
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 78 2008-10-20 #4
78
Process Algebra for Parallel and Distributed Processing
on any node of the network. Tasks are typically computationally demanding. The methodology we present is, however, of a more general nature as will be discussed below, and indeed applies to a wide range of parallel and distributed systems. The refinement calculus [6] generalizes the weakest precondition technique of Dijkstra [10] to prove the correctness of refinement steps during program construction. It is based on the weakest precondition predicate transformer semantics of the underlying programming language. All correctness reasoning is carried out within the calculus relying ultimately on the predicate transformers. However, within the calculus we can define transformation rules and base the derivation on these rules. The rules can be general purpose or targeted and domain specific. The correctness of the transformation rules is proven within the calculus. In this chapter, we will take this approach and define a small collection of transformation rules particularly suitable for derivations within the field of scientific computing. Parallelism is often formalized in terms of processes. Each process consists of a more or less sequential program. The processes execute in parallel and communicate with each other by sending and receiving messages or through shared memory. Examples of such process-oriented process algebras are CSP [12] and CCS [16]. The reasoning about the behavior of an entire system is based on the behaviors of the single processes. We put forward an alternative approach and model the behavior of parallel and distributed programs in terms of the actions which the system carries out. The action systems formalism we use [4] is a state-based approach in contrast to the event-based process algebras. Moreover, an action system even though modeling parallel computation has an interpretation as a sequential program. The design of the logical behavior of the parallel system is carried out within the context of this purely sequential interpretation, hence simplifying considerably the tedious task of formally verifying and reasoning about parallel programs. The refinement calculus is used as the formal framework for action systems. Event B [15] and UNITY [8] are related formalisms to action systems. Event B has actually been heavily influenced by action systems and it relies on a very similar refinement basis whereas UNITY is based on a temporal logic. We start in Section 3.2 by describing the refinement calculus for parallel programs and define a specification and programming language for this. Thereafter in Section 3.3, we describe a programming methodology together with a set of transformation rules for designing parallel algorithms. In Section 3.4, we illustrate the methodology with the derivation of a parallel algorithm for integer factorization. We end in Section 3.5 with some concluding remarks.
3.2 Refinement Calculus In this section, we describe the foundations of the refinement calculus for parallel programs. We define the notion of correct refinement between statements and describe the method of refinement.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 79 2008-10-20 #5
Refinement of Parallel Algorithms
79
3.2.1 Specification Language We use the classic guarded commands language of Dijkstra [10], with some extensions as our specification language. In this language, we have two syntactic categories, statements and actions. Statements S are defined by S ::=
| | | | |
x := e {Q} S1 ; ...; Sn if A1 []...[]Am fi do A1 []...[]Am od |[ var x; S ]|
((multiple) assignment) (assert statement) (sequential composition) (conditional composition) (iterative composition) (block with local variables)
Here A1 , . . . , Am are actions, x is a list of variables, e is a list of expressions, and Q is a predicate. An action (or guarded command) A is of the form: A ::= g → S where g is a boolean expression (the guard of A, denoted gA) and S is a statement (the body of A, denoted sA). We will say that an action is enabled in a certain state if its guard is true in that state. The assert statement {Q} acts as skip if the condition Q holds in the initial state. If the condition Q does not hold in the initial state, the effect is the same as abort. This statement can be used for specification purposes as it allows us to express conditions on the values of the state variables in a convenient way. The other statements have their usual meanings.
3.2.2 Weakest Precondition Semantics The refinement relation between statements and between actions is defined with respect to the weakest precondition semantics of these. The weakest precondition wp(S, R) for a statement S and any predicate R is a predicate that describes the set of all states such that execution of S begun in any of them is guaranteed to terminate in a state satisfying R. Hence, this is a total correctness semantics requiring statements to terminate. The weakest preconditions of a statement S are listed below [10]: wp(x := e, R) wp({Q}, R) wp(S1 ; ...; Sn , R) wp(if A1 []...[]Am fi, R) wp(do A1 []...[]Am od, R) wp(|[var x; S]|, R)
= = = = = =
R[e/x] Q∧R Sn , R)) wp(S 2 ;...; m 1 , wp(S m gA ∧ i i=1 i=1 gAi ⇒ wp(sAi , R) ∃k.k ≥ 0.Hk (R) ∀x.wp(S, R)
The conditions Hk (R) above are given by H0 (R) = R ∧ ¬(∃j.1 ≤ j ≤ m.gAj )
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 80 2008-10-20 #6
80
Process Algebra for Parallel and Distributed Processing
and for k > 0 Hk (R) = wp(if A1 []...[]Am fi, Hk−1 (R)) ∨ H0 (R) The weakest precondition for an action A = g → S is defined as wp(g → S, R) = (g ⇒ wp(S, R)) The semantics of the do–od statement has also been discussed elsewhere in the literature. Hehner [11] advocates the use of recursion as a means of achieving repetitive behavior, whereas Back [2] discusses the do–od construct in the presence of a nondeterministic assignment statement. In neither of these cases is Dijkstra’s original definition optimal. In our case, do–od is mainly used as a means of making parallelism possible, and we find that Dijkstra’s definition is the most suitable for our purposes.
3.2.3 Refinement of Statements A statement S is said to be (correctly) refined by statement S , denoted S S , if for every postcondition Q, wp(S, Q) ⇒ wp(S , Q) In other words, refinement means that whatever total correctness criterion S satisfies, S will also satisfy this criterion (S can satisfy other total correctness criteria also, which S does not satisfy). Intuitively, a statement S is refined by a statement S , if (1) whenever S is guaranteed to terminate, S is also guaranteed to terminate, and (2) any possible outcome of S for some initial state is also a possible outcome of S for this same initial state. This means that a refinement may either extend the domain of termination of a statement or decrease the nondeterminism of the statement, or both. The refinement relation is reflexive and transitive. This means that (1) a statement is always refined by itself (reflexivity) and (2) refinements can be chained or ordered such that a statement in the chain is always refined by the statements occupying a place further down the chain (transitivity). This formally supports the stepwise refinement approach. Moreover, the statement constructors considered here are monotonic with respect to the refinement relation. This allows us to refine substatements in the context of other statements.
3.2.4 Refinement of Actions Refinement between statements can be extended to a notion of refinement between actions as follows. Let A and A be two actions. Action A is refined by action A , A A , if for every postcondition Q, wp(A, Q) ⇒ wp(A , Q)
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 81 2008-10-20 #7
Refinement of Parallel Algorithms
81
When considering the definition of the weakest precondition for an action, we have an action A is refined by an action A if and only if (1) whenever A is enabled, the body of A is refined by the body of A (i.e., {gA }; sA sA ) and (2) A is enabled whenever A is enabled (i.e., gA ⇒ gA).
3.3 Action Systems Formalism We model parallel computations in terms of action systems. In this section, the action systems formalism will be presented. We will describe how the formalism is used to model parallel computation and give several alternative interpretations for action systems.
3.3.1 Parallel Programming with Action Systems 3.3.1.1 Action Systems An action system A is a statement of the form: A = |[var x; S0 ; do A1 []...[]Am od]| : z on state variables y = x ∪ z. The global variables z are indicated explicitly for notational convenience. Each variable is associated with some domain of values. The set of possible assignments of values to the state variables constitutes the state space. The initialization statement S0 assigns initial values to the state variables. Due to the total correctness semantics we follow, only the input/output behavior of A with respect to z is of interest. The behavior of an action system is as follows: the initialization statement is executed first; thereafter, as long as there are enabled actions, one action at a time is nondeterministically chosen and executed. Actions are atomic, i.e., simultaneously enabled actions can be executed in any order, and only their input–output relationship is of interest. Hence, they can be executed in parallel if they do not have any variables in common. 3.3.1.2 Partitioned Action Systems An action system can be viewed as a parallel program, when a process network is associated with it. Basically this can be done either by (1) assigning each action to a process (“partitioning with respect to actions”), or by (2) assigning each state variable to a process (“partitioning with respect to variables”). The processes then execute the actions concurrently, guaranteeing that the atomicity requirement of actions is respected. In this chapter, we will only consider partitioning with respect to actions, or rather an extension of that approach. Consider the action system A above. Let ℵ = {A1 , ..., Am } be the set of actions and let the multiset P= {P1 , ..., Pp } be p a partitioning of ℵ, i.e., Pi ⊆ ℵ, Pi = ∅, for i = 1, ..., p and i=1 Pi = ℵ. Each
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 82 2008-10-20 #8
82
Process Algebra for Parallel and Distributed Processing
action Ai in the partitioned action system (A, P) is thus associated with at least one process Pj that it can be executed in. An action can be associated with more than one process, and it may be executed in any one of those processes. We have that do A od do A[]A od Since the partitions do not have to be mutually disjoint, our solution is not strictly a partitioning in the established mathematical sense of the word. When there are several legal options, an implementation can decide in which of the processes the action will be executed. However, conflicts between variables have to be taken into account when actions are scheduled. Two actions may not be executed at the same time if they have variables in common, since atomicity must be respected. We consider detection of variable conflicts as an implementation issue, the details of which are outside the scope of this chapter, but we assume that there are mechanisms for communication between processes. Degerlund et al. [9] have proposed methods for implementing action systems partitioned with respect to actions and have developed a proof-of-concept scheduler for executing such action systems. Example 3.1 To illustrate partitioning, consider the following action system for performing exchange sort: S = |[ x.1, ..., x.n := Xi , ..., Xn do [] x.1 > x.2 → x.1, x.2 := x.2, x.1 ... [] x.(n − 1) > x.n → x.(n − 1), x.n := x.n, x.(n − 1) od ]| : x.1, ..., x.n ∈ integer
(E.1) ... (E.(n − 1))
The system contains n − 1 sorting actions: E.1, ..., E.(n − 1), each of which swaps pairs of numbers if they are not in ascending order. We especially note that two consecutive actions (e.g., E.1 and E.2) cannot be executed simultaneously, since they have variables in common. On the other hand, if the actions are not consecutive (e.g., E.1 and E.3), they have no common variables, and their execution can take place at the same time. The (extended) partitioning with respect to actions can, for example, be done in the following ways: P1 = {{E.1, ..., E.(n − 1)}} P2 = {{E.1}, ..., {E.(n − 1)}} P3 = {{E.1, ..., E.(n − 1)}, ..., {E.1, ..., E.(n − 1)}},
|P3 | = n − 1
In partitioning P1 , there is only one process, and all actions have to be scheduled on that one. In P2 , there are as many processes are there are actions (n − 1), and each action is tied to a particular process. If there were no variable conflicts, all
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 83 2008-10-20 #9
Refinement of Parallel Algorithms
83
actions could be executed in parallel in their respective processes. However, in our example there are variable conflicts, so this degree of parallelism is not possible since atomicity has to be taken into account. In partitioning P3 , there are n − 1 processes, and any action can be executed in any process (as long as variable dependencies are respected).
3.3.2 Derivation of Action Systems Action systems can be developed in a stepwise manner [5,18] with the target execution platform or process network in mind. Here, the main focus is on grid-like target platforms, where any action can in principle be executed on any processing unit, to the extent permitted by the partitioning. However, the rules we present below apply to a wider class of platforms. Moreover, according to our experience we believe that specific rules for other types of platforms can be developed based on the refinement calculus for action systems. Because action systems are just a special kind of sequential statements in our specification language, we can use the refinement calculus for stepwise refinement of action systems. However, as the goal is to transform a more or less sequential algorithm into an action system that can be executed in a parallel fashion, we need special kinds of transformation rules that introduce parallelism into the execution. As an action system is the unit for which we give an interpretation in terms of parallel activity, we need (i) Rules to turn arbitrary statements into do–od constructs (ii) Rules for merging sequentially composed action systems (iii) Rules for changing variables in statements The first class of rules (i) makes it possible to create action systems out of any statement in our language. An example of a class (i) rule is Sequence-to-loop. This rule states that the following refinement can be done: n n n gi ; S1 ; gi ; S2 ; ...; gn ; Sn i=1
i=2
j =n
do g1 → S1 []...[]gn → Sn od provided that 1. Each gi → Si terminates with gi false 2. gi → Si and gj → Sj can be executed in any order for all i, j, i = j 3. gi → Si cannot affect the value of gj for all i, j, i = j Notice that we use the assert statement here to require that every action in the loop is initially enabled. When inserting an assert statement we use the fact that the statement acts as skip when the predicate evaluates to true, having no effect on the
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 84 2008-10-20 #10
84
Process Algebra for Parallel and Distributed Processing
state. Moreover, we have to guarantee that each action is executed only once in the loop. Using the rules in class (ii) we can take full advantage of the divide-and-conquer style of program design and introduce parallelism into execution by merging the action systems for subproblems. One of the rules belonging to class (ii) is the Merge-into-loop rule. We have that do g1 → S1 od; {g2 }; S2 do g1 → S1 []g2 → S2 od provided that 1. S2 cannot affect the value of g1 2. g1 → S1 and g2 → S2 can be executed in any order or g2 → S2 cannot precede g1 → S1 3. do g2 → S2 od terminates when g1 is true The rule also generalizes to n (n > 2) actions. A similar class (ii) rule is Merge loops, which combines two sequentially composed do–od loops into a single one. We have that do g1 → S1 od; do g2 → S2 od do g1 → S1 []g2 → S2 od provided that 1. S2 cannot affect the value of g1 2. g1 → S1 and g2 → S2 can be executed in any order or g2 → S2 cannot precede g1 → S1 3. do g2 → S2 od terminates when g1 is true This rule also generalizes to more than two actions. Even though the rules in (i)–(ii) allow us to create actions and action systems out of arbitrary statements, they do not necessarily introduce any real parallelism into an execution. For this, we need actions which are independent from each other in the sense that they do not have variables in common and that they can be simultaneously enabled. The rules in class (iii) make actions independent of each other and make it possible to change the way in which the program state is represented by variables in the system, in order to replace a centralized representation by a distributed one. Typically, we replace one variable by a number of variables, for example, one for each process in the partitioning. The new variables hold the same information as the original variable, but permit the information to be accessed in a distributed fashion.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 85 2008-10-20 #11
Refinement of Parallel Algorithms
85
This makes the processes less dependent of each other so that more activity can go on in parallel. One of the rules belonging to this class is the Exchange variables rule. We have that {v1 = v2 }; S S[v2 /v1 ] where v1 and v2 are variables, and S is a statement containing variable v1 . The conditions for applying the rule is that v1 is free in S and v2 is not free is S. A similar rule is the Exchange expressions rule. Considering two expressions, e1 and e2 , it states that if the assertion {e1 = e2 } holds, an expression S containing the expression e1 can be refined by S , which is the same expression as S except for all occurrences of e1 having been replaced by e2 . We furthermore need general rules and methods for turning a program into a form where the above methods can be applied and also for improving the efficiency of the program in general. Many of these rules are not specific to action systems but apply to any kind of statements. An example of a general rule is the Add new variables rule: |[var x; S]| |[var x, y; S1 ; S; S2 ]|. Here, we have the condition that S1 and S2 do not write to old variables, and S1 and S2 need to terminate. Examples of how the rules presented here can be used in practice are given in the following section.
3.4 Case Study: Integer Factorization We present the methodology via a case study about integer factorization. Even though we focus on this particular problem, it follows a general derivation pattern that can be used in other examples as well: 1. We first specify the problem using mathematics and discuss the target parallel platform. Thereafter, 2. the mathematical formulae are manipulated using equivalent representations which more reflect the target architecture for the parallel algorithm that we want to develop. When we feel satisfied with the mathematical representation, 3. we turn the formulae directly into an equivalent specification, usually a sequential statement, in our specification language. Thereafter, 4. we use the divide-and-conquer paradigm and identify the independent tasks of the future computation (rules in class (iii)). Finally, 5. the tasks are turned into loops (class (i) rules) and the loops are merged into a single loop (class (ii)) having the form of an action system.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 86 2008-10-20 #12
86
Process Algebra for Parallel and Distributed Processing
The main intellectual inventions are done at steps (2) and (4) where we actually invent the parallel algorithm. As for our factorization case study, steps (1) and (2) will be described below in Section 3.4.1, whereas steps (3)–(5) will be handled in Section 3.4.2.
3.4.1 Problem Scenario We will now give a simple example of how action systems can be used to derive a parallel program. Given a number num ∈ {2, 3, ...} the problem we want to solve is finding the greatest number, f actor ∈ {1, ..., num − 1}, by which num is evenly divisible, i.e. num mod f actor = 0. Such a number exists for any accepted values of num, since ∀num ∈ {2, 3, ...} : num mod 1 = 0. This task can be expressed by the following formula: f actor = max({i ∈ {1, ..., num − 1} | num mod i = 0})
3.4.1.1 Topology We will do the development with a specific target architecture in mind, for which we will gradually adapt the system. We assume that our architecture contains n identical processors for task computation (“slave”) and one additional processor for coordination. We consider a star topology where the coordinating processor should be able to communicate directly with each of the slave processors, but where the slaves do not need to be able to communicate directly with each other. We also assume that one process is executing on each processor. In practice, this can be a grid scenario where the slave processes, as well as the coordination process, are run on computers connected to the Internet. Another possibility that is equally plausible is execution on computers in a local area network (LAN), or even on a single computer containing several processors (SMP) or on a computer consisting of one processor containing several computational cores. Our topology is illustrated in Figure 3.1. 3.4.1.2 Tackling the Problem Our formula states what we want to achieve in a precise mathematical manner. We will use transformation rules to gradually work toward more concrete statements, of which the final one will not only be a true action system but also be adapted to parallel implementation. During the refinement, we do not state specifically how the factorization is performed. However, we do have the method of trial division in mind, and we will tackle the problem accordingly. Consider the set of divisors, {1, ..., num − 1}, by which we will try to divide num. We will split this set into smaller chunks, or subsets, so that these can be distributed to slave processors for trial division. The granularity of the splitting into subsets is given by a parameter step ∈ {1, ..., num − 1}, and each of the subsets will contain step elements. If num − 1 is not evenly divisible by step, some subsets will contain negative numbers
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 87 2008-10-20 #13
Refinement of Parallel Algorithms
87
Slave proc. 1 Slave proc. n . . .
Slave proc. 2
Coordination process Slave proc. 3
Slave proc. 5
Slave proc. 4
FIGURE 3.1: Process topology used in our example.
that will not affect the final result. We are fully aware of the fact that trial division is not particularly efficient, and that there are more sophisticated factorization algorithms available, such as the general number field sieve algorithm. Trial division is, however, very intuitive, and serves our purposes well for showing the general pattern of how parallelism can be introduced when developing software using program refinement.
3.4.1.3 Parallelization Strategy Our goal is to refine action systems in such a way that n subsets can be tested in parallel on the n processors, i.e., one subset for each processor. We will call such a trial division of n subsets a round. Given the parameters num, n, and step, the num−1 total number of rounds needed will be r = roof st ep∗n . After each round (except the last one), n new subsets can be distributed to the processors. Since actions with common variables are not allowed to execute in parallel, we will refine the system in such a way that the computing actions involved in the same round will not have any variables in common in the final step. Figure 3.2 shows how the splitting takes place when num = 39, n = 3, and step = 5, in which case three rounds will be needed. The last subset (i.e., the one for processor 3 in the last round) contains only negative values and will, in practice, be superfluous. We especially want to emphasize that we work “backward” from the largest number (num − 1) down toward 1 (and finally negative numbers if needed for filling the last few subsets in the last round). This means that, for example, processor 1 during round 1 will be assigned the subset containing the largest numbers, i.e., {num−step, ..., num−1}. The general formulae
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 88 2008-10-20 #14
88
Process Algebra for Parallel and Distributed Processing Proc 3
Proc 2
Proc 1
Round 3
−6 to −2
−1 to 3
4 to 8
Round 2
9 to 13
14 to 18
19 to 23
Round 1
24 to 28
29 to 33
34 to 38
FIGURE 3.2: Splitting into computational chunks (num = 39, n = 3, and step = 5). for deciding the start and stop values for a particular subset as a function of processor and round number are given below: start (round, proc) = num − 1 − (round − 1) ∗ n ∗ step − (proc − 1) ∗ step stop(round, proc) = start (round, proc) − step + 1 We wish to point out that since we work from higher numbers toward lower ones, the start value is greater than the stop value. In particular, and in full accordance with what we stated above, start (1, 1) = num − 1 and stop(1, 1) = num − step.
3.4.2 Derivation We will now define a function for finding the maximum (positive) divisor, by which num is evenly divisible, in an interval {low, high} (1 ≤ low ≤ high): search(low, high) = max({0} ∪ {i ∈ {low, ..., high} | num mod i = 0}). In case such a divisor does not exist, the value 0 is returned. We notice that with the parameters low = 1, high = num − 1, the search function will yield the expression for factor. The search function does contain the number 0 in the max expression, which our factor expression does not. However, with legal values of num, low, and high, a (positive) result will always be found, so it is irrelevant whether the number 0 is there or not. Mathematically, the expressions will thus coincide. We can therefore use the search function to specify of first version of our statement: A0 = |[factor := search(1, ..., num − 1) ]| : factor ∈ Z+ We are now ready to start rewriting the statement to better suit our needs. We will start by splitting the search space into several parts. We split the set {1, ..., num − 1} into subsets of the size step numbers. We will also consider rounds, so that n sets are examined in each round. Furthermore, the new variables f ound11, ..., f oundrn are introduced to indicate what factor (or 0, if not found) each of the n searches per round has found. Our new version, A1 , is shown below. Note that meta notation
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 89 2008-10-20 #15
Refinement of Parallel Algorithms
89
for should be interpreted as a replication of the previous statement, with sequential composition (;) as the delimiter. Later on, we will also use for to denote replication where the delimiter is nondeterministic choice ([]). In those cases, we indicate the delimiter by writing [] before the statement to be replicated. A1 = |[ varfound11 , ..., foundrn ∈ Z+ [(foundij := search(stop(i, j ), start (i, j ))) for(1 ≤ j ≤ n) ] for(1 ≤ i ≤ r); {search(1, ..., num − 1) = max(found11 , ..., foundrn )}; factor := max(found11 , ..., foundrn ) ]| : factor ∈ Z+ As compared to A0 , we have replaced the only assignment with n ∗ r + 1 sequential assignments. The correctness of the refinement step is proven as follows. The introduction of new variables, as well as statements involving only these variables (f oundij := ...), is justified by the Add new variables rule. Having introduced these new statements as shown in A1 , we can show mathematically that the assertion {search(1, ..., num − 1) = max(found11 , ..., foundrn )} holds prior to the factor := ... statement. Thus, the condition for applying the Exchange expressions rule holds. By using this rule to replace the old expression search(1, ..., num − 1) with max(found11 , ..., foundrn ), we achieve an alternative assignment to the factor variable, i.e., factor : = max(found11 , ..., foundrn ). Since we have used two rules (Add new variables, Exchange expressions), we have actually performed two transformation steps when refining A0 into A1 . Next, we introduce partial analysis after each round by adding the new variables partial1 , ..., partialr and logically restructure our block statement: A2 = |[ var found11 , ..., foundrn , partial1 , ..., partialr ∈ Z+ [(foundij := search(stop(i, j ), start (i, j ))) for(1 ≤ j ≤ n); partiali := max(foundi1 , ..., foundin )] for(1 ≤ i ≤ r); factor := max(partial1, ..., partialr ) ]| : factor ∈ Z+ Each round will now be analyzed separately and the result of round i will be stored in a variable partiali . After the final round, the variable f actor will be assigned the greatest of partial maximum values found after each round, i.e., the maximum value of partial1 , ..., partialr . The correctness of this step is proven in a similar manner as the last one, using exactly the same rules. To illustrate how A2 works intuitively, Figure 3.3 shows the order in which computations take place as well as what partial results are generated at what stage. The various shades of grey in the figure illustrate what takes place at what time, so that a lighter shade means that the block (subset of numbers) in question is
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 90 2008-10-20 #16
90
Process Algebra for Parallel and Distributed Processing Foundrn Round r
Proc n
Proc n −1
Proc n −2
Proc 3
Proc 2
Proc 1
Partialr −1
Stop
Round r −1 Round 2 Start
Round 1
Found11
Factor
FIGURE 3.3: Computational structure of A2 . processed at a later time than a darker block. For example, the first subset of numbers (in the lower right-hand corner) will be processed first, and its result (the greatest factor found, or 0 if none is found) will be stored in variable found11 . Since we have not yet introduced nondeterminism, the subsets (blocks) still cannot be processed in parallel. Next, we will introduce a do–od loop using the Sequence-to-loop rule. In this way, we will achieve nondeterministic choice between the search statements within a round. The new variables comp1, ..., compn are added for use in the guards that will be introduced for the search statements. This can be written as A3 = |[var found11 , ..., foundrn , partial1 , ..., partialr ∈ Z+ , comp1, ..., compn ∈ boolean+ comp1, ..., compn := true; ( do []compj → foundij := search(stop(i, j ), start (i, j )); compj := false for (1 ≤ j ≤ n) od; partiali := max(foundi1 , ..., foundin ); comp1, ..., compn := true) for(1 ≤ i ≤ r); factor := max(partial1, ..., partialr ) ]| : factor ∈ Z+ As for using the Sequence-to-loop rule, we have that assertions mentioned in the rule are true at the required points. The guards comp1 , ..., compn disable themselves after their respective action has been executed, the actions in question can be executed in any order, and they do not affect each other’s guards. Thus, all conditions are fulfilled and the rule can be applied. The intuitive interpretation of the new behavior in A3 is that the do–od loop will terminate after each of the n actions has been running. After
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 91 2008-10-20 #17
Refinement of Parallel Algorithms
91
this has taken place, a partial maximum value is computed in the same way as in A2 , before the next round starts. In the next refinement step, we insert the partial maximum computation into the do–od loop. Technically, this is justified by using the Merge-into-loop rule. Furthermore, we add the new variables round1 , ..., roundn and roundpart. Each roundj variable keeps track of which round processor j is currently in. The roundpart similarly indicates the round for which a partial result should next be computed. The resulting A4 looks as follows: A4 = |[ var found11 , ..., foundrn , partial1 , ..., partialr ∈ Z+ , comp1, ..., compn ∈ boolean, round1 , ..., roundn , roundpart ∈ Z+ comp1, ..., compn := true; round1 , ..., roundn := 1; roundpart := 1; ( do [] roundj = i ∧ compj → foundij := search(stop(i, j ), start (i, j )); compj := false; roundj := roundj + 1 for(1 ≤ j ≤ n) [] roundpart = i ∧ ¬(comp1 ∨ ... ∨ compn ) → partial := max(foundi1 , ..., foundin ); comp1, ..., compn := true; roundpart := roundpart + 1 od ) for(1 ≤ i ≤ r); factor := max(partial1, ..., partialr ) ]| : factor ∈ Z+ We have now achieved r do–od loops, each of which contains actions for the computations of one round. In order to conform to the requirements of an action system, there should however only be one (outer) do–od loop. Therefore, we merge the loops by using the Merge loops rule repeatedly, after verifying that its conditions of use hold, until we achieve A5 : A5 = |[ var found11 , ..., foundrn , partial1 , ..., partialr ∈ Z+ , comp1, ..., compn ∈ boolean, round1 , ..., roundn , roundpart ∈ Z+ comp1, ..., compn := true; round1 , ..., roundn := 1; roundpart := 1; do [] ([] roundj = i ∧ compj → foundij := search(stop(i, j ), start (i, j )); compj := false; roundj := roundj + 1 for(1 ≤ j ≤ n) [] roundpart = i ∧ ¬(comp1 ∨ ... ∨ compn) → partial := max(foundi1 , ..., foundin ); comp1, ..., compn := true; roundpart := roundpart + 1) for(1 ≤ i ≤ r) od; factor := max(partial1, ..., partialr ) ]| : factor ∈ Z+ We are almost finished at this stage, but we still have to insert the final maximum computation into the do–od loop. After verifying that the conditions for using the Merge-into-loop are fulfilled, we apply the rule and achieve our final system A6 :
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 92 2008-10-20 #18
92
Process Algebra for Parallel and Distributed Processing A6 = |[ var found11 , ..., foundrn , partial1 , ..., partialr ∈ Z+ , comp1, ..., compn ∈ boolean+, round1 , ..., roundn , roundpart ∈ Z+ comp1, ..., compn := true; round1 , ..., roundn := 1; roundpart := 1; do [] ([] roundj = i ∧ compj → foundij := search(stop(i, j ), start (i, j )); compi := false; roundj := roundj + 1 for(1 ≤ j ≤ n) (Searchij ) [] roundpart = i ∧ ¬(comp1 ∨ ... ∨ compn ) → partial := max(foundi1 , ..., foundin ); comp1, ..., compn := true; (P artiali ) roundpart := roundpart + 1) for(1 ≤ i ≤ r) [] 1≤j ≤n roundj > r ∧ roundpart > r → factor := max(partial1, ..., partialr ) (Final) od ]| : factor ∈ Z+
In this final version, A6 , we note especially three facts: first, A6 is a correct refinement of our initial statement A0 , i.e., A0 A6 . This is a direct result of the transitivity property of the refinement relation. Second, the system fully adheres to the action system requirements, which includes that there is a single do–od loop, consisting of actions separated by the nondeterministic choice delimiter ([]). Third, search actions (Searchij , 1 ≤ i ≤ r, 1 ≤ j ≤ n) corresponding to the same round (i.e., with the same value of i) do not contain any shared variables. This implies that they can be scheduled for simultaneous execution, as long as we use a proper partitioning. A suitable partitioning would be P1 = {{Search11, ..., Searchr1 }, ..., {Search1n, ..., Searchrn }, {P artial1 , ..., P artialr , F inal}} In P1 , search actions with the same second index (corresponding to processor number in terms of search subsets) are part of the same partition. In this way, each of these partitions contains one action corresponding to each computations round. There is also one partition that contains the actions responsible for computing the partial results after each round, as well as the final result. Partitioning P1 can be conveniently mapped to the topology we had in mind when developing the action system. This mapping is illustrated in Figure 3.4. The system can also be partitioned in an alternative way as follows: P2 = {{Search11, ..., Searchrn }, ..., {Search11, ..., Searchrn }, {P artial1 , ..., P artialr , F inal}}, |P2 | = n + 1 Here, all search actions are included in all partitions, except for a partition that is dedicated for coordination. This partitioning can be mapped onto our topology as shown in Figure 3.5. In this way, specific actions are not tied to specific processors,
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 93 2008-10-20 #19
Refinement of Parallel Algorithms
Slave 1:
Slave n:
Search11 ,..., searchr 1
Slave 2:
Search1n ,..., searchrn
Coord.:
Search12 ,..., searchr 2
Slave 5:
Partial1 ,..., Partial r , final
Slave 3:
. . . Search15 ,..., search r5
Slave 4:
Search13 ,..., searchr 3
Search14 ,..., searchr 4
FIGURE 3.4: Partitioning P1 of the final system.
Slave 1:
Slave n:
Search11 ,..., searchrn
Slave 2:
Search11 ,..., searchrn
Coord.:
Search11 ,..., searchrn
Slave 5:
Partial1 ,..., partialr , final
Slave 3:
. . . Search11 ,..., searchrn
Slave 4: Search11 ,..., searchrn
FIGURE 3.5: Partitioning P2 of the final system.
Search11 ,..., searchrn
93
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 94 2008-10-20 #20
94
Process Algebra for Parallel and Distributed Processing
but the actions can be executed on any available processor (except for the coordinating one). As mentioned in Section 3.3.1, we consider the practical details concerning communication between processes to be outside the scope of this chapter, but we assume an implementation providing the required interprocess communication mechanisms. Such a framework for implementing action systems has been discussed by Degerlund et al. [9]. Furthermore, Bostr¨om and Wald´en [7] have proposed a specification language for facilitating development of grid systems. Their language, Distributed B, is based on Event B, which in turn is similar to the action systems formalism.
3.5 Concluding Remarks In this chapter, we have presented a design methodology for the derivation of provably correct parallel algorithms expressed as action systems. We have shown how the combination of refinement calculus and action systems formalism gives a foundation for a design methodology. The method is based on systematic construction of programs by stepwise refinement. Each step in the derivation can be formally proven correct. The main emphasis is on a methodology that allows the programmer to start with a traditional, purely sequential algorithm and gradually turn it into a parallel version. We argue that the transformation rules discussed allow the designer to uncover the potential parallelism in the program in a more innovative and general manner than what a parallelizing compiler can do. Our presentation was mainly targeting grid- and cluster-type parallel platforms. The main difficulty of grid programming [13], task parallelization of the algorithm, is handled here via the program transformations. The main novelty here is the focus on correctness and formal program transformation when parallelizing a system. The starting point can be a problem expressed using mathematical formulae (as in the case study of Section 3.4), or an already existing algorithm, that may be sequential, and which is turned into a parallel program using transformations. The presented method is rather general and the target platform can be defined freely via the variable or action partitioning as seen via the derivations. As was observed, we started from the mathematical definitions of the problems at hand and used abstract, high-level constructs to give the initial specifications. In this way our language resembles the programming language Fortress [19] which also allows mathematical notation. Moreover, our approach is independent of programming language compared to many other systems where more concrete Java or C/C++ programming is required [1,17,20]. Indeed, we can define classes of action systems, which can be implemented on different execution platforms and using different programming languages [5,9,18]. Furthermore, the focus here was on total correctness reasoning about parallel algorithms. Action systems are also used to model reactive behavior [3]. Related to this are formalisms such as TLA+ [14] and input/output automata, as well as Event B [15].
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 95 2008-10-20 #21
Refinement of Parallel Algorithms
95
References [1] M. Aldinucci, M. Coppola, M. Danelutto, N. Tonellotto, M. Vanneschi, and C. Zoccolo. High level grid programming with ASSIST. Computational Methods in Science and Technology, 12(1):21–32, 2006. [2] R.J.R. Back. Proving total correctness of nondeterministic programs in infinitary logic. Acta Informatica, 15(3):233–249, 1981. [3] R.J.R. Back. Refinement calculus. Part II: Parallel and reactive programs. In Stepwise Refinement of Distributed Systems. Models, Formalisms, Correctness (REX Workshop Proceedings), Mook, the Netherlands, Lecture Notes in Computer Science, 430:67–93, 1990. [4] R.J.R. Back and R. Kurki-Suonio. Decentralization of process nets with centralized control. In PODC ’83: Proceedings of the Second Annual ACM Symposium on Principles of Distributed Computing, New York, ACM Press, 1983, pp. 131–142. [5] R.J.R. Back and K. Sere. Stepwise refinement of action systems. Structured Programming 12, pp. 17–30, 1991. [6] R.J.R. Back and J. von Wright. Refinement Calculus: A Systematic Introduction. Springer-Verlag, New York, 1998. [7] P. Bostr¨om and M. Wald´en. An extension of Event B for developing grid systems. In ZB 2005: Formal Specification and Development in Z and B, Guildford, U.K., Lecture Notes in Computer Science, 3455:142–161, 2005. [8] K.M. Chandy and J. Misra. Parallel Program Design: A Foundation. AddisonWesley Publishing Company, Reading, MA 1988. [9] F. Degerlund, M. Wald´en, and K. Sere. Implementation issues concerning the action systems formalism. In David S. Munro, Hong Shen, Quan Z. Sheng, Henry Detmold, Katrina E. Falkner, Cruz Izu, Paul D. Coddington, Bradley Alexander, and Si-Qing Zheng, editors, Proceedings of the Eighth International Conference on Parallel and Distributed Computing Applications and Technologies (PDCAT’07), IEEE Computer Society Press, 2007, pp. 471–479. [10] E. Dijkstra. A Discipline of Programming. Prentice Hall International, Englewood Cliffs, NJ, 1976. [11] E. Hehner. do considered od: A contribution to the programming calculus. Acta Informatica, 11(4):287–304, 1979. [12] C.A.R. Hoare. Communicating Sequential Processes. Prentice-Hall International, Englewood Cliffs, NJ, 1985. [13] D. Laforenza. Grid programming: some indications where we are headed. Parallel Computing, 28(12):1733–1752, 2002.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C003 Finals Page 96 2008-10-20 #22
96
Process Algebra for Parallel and Distributed Processing
[14] L. Lamport. Specifying Systems: The TLA+ Language and Tools for Hardware and Software Engineers. Addison-Wesley Longman, Boston, MA 2002. [15] C. M´etayer, J.-R. Abrial, and L. Voisin. Event-B Language. RODIN Deliverable D7. EU Project RODIN (IST-511599), 2005. http://rodin.cs.ncl. ac.uk/deliverables.htm. [16] R. Milner. A Calculus of Communicating Systems. Lecture Notes in Computer Science, 92, 1980. [17] T.-A. Nguyen and P. Kuonen. Programming the grid with POP-C++. Future Generation Computer Systems, 23(1):23–30, 2007. ˚ Akademi [18] K. Sere. Stepwise derivation of parallel algorithms. PhD thesis, Abo ˚ University, Abo, Finland, 1990. [19] G. Steele. The Fortress parallel programming language, 2006. Presentation slides from 2006 GSPx multicore applications conference, http://research.sun.com/projects/plrg/Publications/GSPx-Lecture2006 public.pdf. [20] R. van Nieuwpoort, J. Maassen, T. Kielmann, and H.E. Bal. Satin: Simple and efficient Java-based grid programming. Scalable Computing: Practice and Experience, 6(3):19–32, 2005.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 S002 Finals Page 97 2008-10-15 #3
Part II
Distributed Systems
Alexander/Process Algebra for Parallel and Distributed Processing C6486 S002 Finals Page 98 2008-10-15 #4
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C004 Finals Page 99 2008-10-22 #3
Chapter 4 Analysis of Distributed Systems with mCRL2 Jan Friso Groote, Aad Mathijssen, Michel A. Reniers, Yaroslav S. Usenko, and Muck van Weerdenburg Contents 4.1 4.2
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 mCRL2 Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.2.1 Process Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.2.1.1 Multiactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.2.1.2 Basic Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.2.1.3 Parallel Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.2.1.4 Additional Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.2.1.5 Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.2.2 Data Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.2.2.1 Basic Data Type Definition Mechanism. . . . . . . . . . . . . . . . . . . . 106 4.2.2.2 Standard Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.2.2.3 Type Formers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.2.2.4 Sort References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.2.2.5 Standard Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.2.2.6 Additional Language Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.3 Verification Methodology and the mCRL2 Toolset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.4 Examples of Modeling and Verification in mCRL2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.4.1 Dining Philosophers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.4.2 Alternating Bit Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.4.3 Sliding Window Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 4.5 Historical Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.1 Introduction The language mCRL2 [14] is a process algebra with data and time suitable for the specification and verification of a broad range of (typically real-time distributed) systems. Specific characteristics of the mCRL2 process algebra are local communication, multiactions, and communication-independent parallelism. Local communication
99
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C004 Finals Page 100 2008-10-22 #4
100
Process Algebra for Parallel and Distributed Processing
allows one to specify communications within subcomponents of the system without affecting the rest of the system. This makes the language particularly suitable for component-based and hierarchical systems. Global communication, which is typically used in other languages, only allows communication to be defined on a global level for the system as a whole. Multiactions enable the specification of (not necessarily related) actions that are to be executed together. Most process algebras only allow a single action to be executed atomically thus forcing an order on the execution of actions. By allowing multiple actions to be executed together, certain systems (such as low-level hardware) can be modeled in a more straightforward manner. These multiactions also allow the separation of parallelism and communication. When two actions can execute at the same time, a multiaction with those actions is the result. Communication can then be applied to these multiactions to make certain actions communicate with each other. Besides the process algebra, mCRL2 also has a higher-order data language. Processes and actions can be parameterized with this data, which is essential for the description of most practical systems. Besides several built-in data types (e.g., integers, lists, and sets) the user can define his or her own abstract data types in a straightforward manner. Typical functions, such as equality, are automatically generated for such user-defined data types. As the data language is higher-order, functions are first-class citizens and can therefore be used just as easily as other data. For practical use we have developed a toolset for mCRL2. This toolset contains numerous tools for manipulation of specifications, state space exploration and generation, simulation, verification, and visualization. To be able to work efficiently a specification is transformed into a specific form called a linear process. Such a linear process can be seen as a compact symbolic representation of the (possibly infinite) state space of the system. Many optimization and verification techniques have been developed for these linear processes (see Section 4.3). A typical practice in verification via model checking is to generate a state space of a specification and use that state space to check certain properties. The mCRL2 toolset supports this practice, but also offers an alternative. Given a specification and a (modal) formula expressing a desired (temporal) property, the toolset can generate a parameterized boolean equation system (PBES) [19,21]. Such a PBES can then be solved by specific provers of the toolset. The use of PBESs is particularly interesting for systems that are too large or complex for model checking via explicit state spaces. The mCRL2 toolset has been successfully used in various academic and industrial case studies; see Refs. [27,36,43], or Ref. [29] for an overview. mCRL2 is the successor of µCRL [15,16] and its timed version timed µCRL [13,37]. The languages are very similar: any process description in (timed) µCRL can be translated straightforwardly into a description of the same process in mCRL2. For a discussion on the differences and their motivation we refer the reader to Section 4.5 and Refs. [14,20]. In Section 4.2, we introduce the mCRL2 language. In Section 4.3, we give an in-depth overview of the toolset. We further illustrate the possibilities offered by the mCRL2 language and toolset by means of some examples in Section 4.4. In Section 4.5, we place mCRL2 in its historical context.
Alexander/Process Algebra for Parallel and Distributed Processing C6486 C004 Finals Page 101 2008-10-22 #5
Analysis of Distributed Systems with mCRL2
101
4.2 mCRL2 Language We define the syntax of the mCRL2 process language together with an intuitive explanation of the semantics. After that, we discuss the data language that is used by the process language.
4.2.1 Process Language This section gives an overview of the mCRL2 process language. The precise semantics—in terms of timed transition systems and by means of structural operational semantics (SOS)—can be found in Ref. [20, Section 3]. The axioms of mCRL2 can be found in Ref. [20, Section 2]. 4.2.1.1 Multiactions The primary notion in the mCRL2 process language is an action, which represents an elementary activity o