Concurrent Systems Engineering Series Editors
M.R. Jane, J. Hulskamp, P.H. Welch, D. Stiles and T.L. Kunii
Volume 60 Previously published in this series: Volume 59, Communicating Process Architectures 2001 (WoTUG-24), A. Chalmers, M. Mirmehdi and H. Muller Volume 58, Communicating Process Architectures 2000 (WoTUG-23), P.H. Welch and A.W.P. Bakkers Volume 57, Architectures, Languages and Techniques for Concurrent Systems (WoTUG-22), B.M. Cook Volumes 54–56, Computational Intelligence for Modelling, Control & Automation, M. Mohammadian Volume 53, Advances in Computer and Information Sciences '98, U. Giidukbay, T. Dayar, A. Gursoy and E. Gelenbe Volume 52, Architectures, Languages and Patterns for Parallel and Distributed Applications (WoTUG-21), P.H. Welch and A.W.P. Bakkers Volume 51, The Network Designer's Handbook, A.M. Jones, N.J. Davies, M.A. Firth and C.J. Wright Volume 50, Parallel Programming and JAVA (WoTUG-20), A. Bakkers Volume 49, Correct Models of Parallel Computing, S. Noguchi and M. Ota Volume 48, Abstract Machine Models for Parallel and Distributed Computing, M. Kara, J.R. Davy, D. Goodeve and J. Nash Volume 47, Parallel Processing Developments (WoTUG-19), B. O'Neill Volume 46, Transputer Applications and Systems '95, B.M. Cook, M.R. Jane, P. Nixon and P.H. Welch Transputer and OCCAM Engineering Series Volume 45, Parallel Programming and Applications, P. Fritzson and L. Finmo Volume 44, Transputer and Occam Developments (WoTUG-18), P. Nixon Volume 43, Parallel Computing: Technology and Practice (PCAT-94), J.P. Gray and F. Naghdy Volume 42, Transputer Research and Applications 7 (NATUG-7), H. Arabnia Volume 41, Transputer Applications and Systems '94, A. de Gloria, M.R. Jane and D. Marini Volume 40, Transputers '94, M. Becker, L. Litzler and M. Trehel Volume 39, Transputer/Occam Japan 6, S. Noguchi, M. Ishizuka and M. Ota Volume 38, Progress in Transputer and Occam Research (WoTUG-17), R. Miles and A. Chalmers Volume 37, Parallel Computing and Transputers (ATOUG-6), D. Arnold, R. Christie, J. Day and P. Roe Volume 36, Transputer Applications and Systems '93, R. Grebe, J. Hektor, S.C. Hilton, M.R. Jane and P.H. Welch Volume 35, Transputer/Occam Japan 5 (OUGJ-5), S. Noguchi and M. Yamamoto Volume 34, Transputer Research and Applications 6 (NATUG-6), S. Atkins and A.S. Wagner Volume 33, Transputer and Occam Research: New Directions (WoTUG-16), J. Kerridge Volume 32, Networks, Routers and Transputers, M.D. May, P.W. Thompson and P.H. Welch Volume 31, Transputers and Parallel Applications (ATOUG-5), J. Hulskamp and D. Jones Volume 30, Transputing in Numerical and Neural Network Applications, G.L. Reijns and J. Luo Volumes 28 and 29, Parallel Computing and Transputer Applications (parts I, II), (Pacta '92), M. Valero, E. Onate, M. Jane, J.L. Larriba and B. Suarez Volume 27, Transputer/Occam Japan 4 (OUGJ-4), S. Noguchi and H. Umeo Volume 26, Transputers '92 - Advanced Research and Industrial Applications, M. Becker, L. Litzler and M. Trehel
ISSN: 1383-7575
Communicating Process Architectures 2002 Edited by
James Pascoe Department of Computer Science, The University of Reading, United Kingdom
Peter Welch Computing Laboratory, The University of Kent at Canterbury, United Kingdom
Roger Loader Department of Computer Science, The University of Reading, United Kingdom
and
Vaidy Sunderam Department of Mathematics and Computer Science, Emory University, United States of America
WoTUG-25 Proceedings of the 25th WoTUG Technical Meeting, 15–18 September 2002, University of Reading, United Kingdom
/OS
Press Ohmsha
Amsterdam • Berlin • Oxford • Tokyo • Washington, DC
© 2002, The authors mentioned in the Table of Contents All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 1 58603 268 2 (IOS Press) ISBN 4 274 90539 X C3055 (Ohmsha) Library of Congress Control Number: 2002111139
Publisher IOS Press Nieuwe Hemweg 6B 1013 BG Amsterdam The Netherlands fax:+3120 620 3419 e-mail:
[email protected]
Distributor in the UK and Ireland IOS Press/Lavis Marketing 73 Lime Walk Headington Oxford OX3 7AD England fax:+44 1865 75 0079
Distributor in the USA and Canada IOS Press, Inc. 5795-G Burke Centre Parkway Burke, VA 22015 USA fax: +1 703 323 3668 e-mail:
[email protected]
Distributor in Germany, Austria and Switzerland IOS Press/LSL.de Gerichtsweg 28 D-04103 Leipzig Germany fax: +49 341 995 4255
Distributor in Japan Ohmsha, Ltd. 3-lKandaNishiki-cho Chiyoda-ku, Tokyo 101-8460 Japan fax:+813 3233 2426
LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS
v
Contents Preface James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam Programme Committee
vii viii
Papers Semantics of prialt in Handel-C™ Andrew Butterfield and Jim Woodcock
1
Acceptances, Behaviours and Infinite Activity in CSPP Adrian Lawrence
17
HCSP: Imperative State and True Concurrency Adrian Lawrence
39
Consolidating the Agreement Problem Protocol Verification Environment James Pascoe and Roger Loader
57
On the Complexity of Buffer Allocation in Message Passing Systems Alex Brodsky, Jan Bcekgaard Pedersen and Alan Wagner
79
Java PastSet - A Structured Distributed Shared Memory System Kei Simon Pedersen and Brian Vinter
97
Synchronous Active Objects Introduce CSP's Primitives in Java Claude Petitpierre
109
Configurable Collective Communication in LAM-MPI John Markus Bjorndalen, Otto J. Anshus, Brian Vinter and Tore Larsen
123
Cache-Affinity Scheduling for Fine Grain Multithreading Kurt Debattista, Kevin Vella and Joseph Cordina
135
A Predicate Transformer Semantics for a Concurrent Language of Refinement Ana Cavalcanti and Jim Woodcock
147
Reconnetics: A System for the Dynamic Implementation of Mobile Hardware Processes in FPGAs Ralph Moseley
167
vi
Performance Analysis and Behaviour Tuning for Optimisation of Communicating Systems Mark Green and A li E. A bdallah
181
Configuration Discovery and Mapping of a Home Network Keith Pugh
191
Cluster Computing and JCSP Networking Brian Vinter and Peter Welch
203
View-Centric Reasoning for Linda and Tuple Space Computation Marc L. Smith, Rebecca J. Parsons and Charles E. Hughes
223
A Graphical Modeling Language for Specifying Concurrency based on CSP Gerald Hilderink
255
The "Honeysuckle" Programming Language: Event and Process Ian East
285
The "Honeysuckle" Programming Language: Object and Protocol Ian East
301
A Communicating Threads (CT) Case Study: JIWY Dusko Jovanovic, Gerald Hilderink, Jan Broenink
311
Prioritised Dynamic Communicating Processes - Part I Fred Barnes and Peter Welch
321
Prioritised Dynamic Communicating Processes - Part II Fred Barnes and Peter Welch
353
Implementing a Distributed Algorithm for Detection of Local Knots and Cycles in Directed Graphs Geraldo Pereira de Souza and Gerson Henrique Pfltscher
371
Author Index
387
vii
Preface Natural concurrency surrounds us. The concept is so prevalent that we interact and benefit from concurrent operations all the time and think nothing unusual of it. The simultaneous management of multiple tasks is how everyday business is conducted. Communication plays a paramount role in keeping things synchronised and, for sentient beings, happens without much pre-planning - it's automatic. However, for computers, it's not so easy. All computer systems contain concurrent architectures at many levels of their design. Over a long period, the WoTUG series of meetings have provided a major forum for the presentation and discussion of novel directions in the field of concurrency and communication. Communicating Process Architectures 2O02 extends this series by adding the high quality papers contained in these proceedings. The discussions are wide ranging and encompass many aspects of concurrent systems - from theoretical foundations, through software and hardware tools (including languages, libraries, kernels and processors) and on to applications. The 22 accepted papers will be presented in a single stream during the day, with (unplanned) fringe sessions in the evenings. Normal sessions will not concentrate on particular themes - except that closely related presentations will be kept together. The aim is to create a workshop athmosphere with all participants engaging in as much as possible, especially in talks that may appear to be outside their immediate interests. Selected papers will be invited for further development and submission to a (CPA) Special Edition of IEE Proceedings - Software, for journal publication in Spring 2003. Our best wishes and gratitude go to the authors for submitting their work and responding to the feedback provided by the referees. We also thank the CPA-2002 Programme Committee for their effort in reviewing the papers (and note that, where papers were submitted by members of that committee, they were reviewed in isolation with authors neither choosing nor knowing who their referees were). One individual deserves our special gratitude - Fred Barnes of UKC - without whose time andfiercesomecopy-editing, LaTeX and Word skills, these proceedings would have been in a much poorer state. Finally, we thank all the staff at Windsor Hall for their assistance in the local arrangements. This occasion is our Jubilee - the twenty-fifth in the series of WoTUG conferences and we intend to celebrate! They have consistently proven to be a valuable meeting place for all those interested in the problems and opportunities thrown up by parallel computing and concurrency. The participants reflect a wide spectrum of disciplines - theoreticians, software engineers, hardware engineers, tool builders and applications specialists. We meet in a relaxed, exciting, friendly and productive atmosphere ... often till very late the next morning ... the trick being to remember what was solved the night before ... and how we solved it! We like to thank you for your interest and welcome you to Reading. We very much hope you enjoy Communicating Process Architectures 2002 !! James Pascoe (University of Reading, UK) Peter Welch (University of Kent at Canterbury, UK) Roger Loader (University of Reading, UK) Vaidy Sunderam (Emory University, Atlanta, USA)
viii
Programme Committee Professor Peter Welch (Programme Chair), University of Kent at Canterbury, UK Dr. James Pascoe (Conference co-Chair), University of Reading, UK Dr. Roger Loader (Conference co-Chair), University of Reading, UK Professor Vaidy Sunderam (Conference co-Chair), Emory University, Atlanta, USA Dr. Alastair Allen, University of Aberdeen, UK Professor Hamid Arabnia, University of Georgia, Athens, Georgia, USA Richard Beton, Roke Manor Research Ltd., UK Professor Jan F. Broenink, University ofTwente, The Netherlands Dr. Alan Chalmers, University of Bristol, UK Professor Peter Clayton, Rhodes University, South Africa Dr. Barry Cook, University ofKeele, UK Ruth A. Ivimey-Cook, ARM Ltd., Cambridge, UK Gerald Hilderink, University ofTwente, The Netherlands Christopher Jones, BAES, Warton Aerodrome, UK Professor Jon Kerridge, Napier University, UK Dr. Tom Lake, InterGlossa, UK Dr. Adrian E. Lawrence, Loughborough University, UK Dr. Jeremy Martin, Oxagen Ltd., Oxford, UK Stephen Maudsley, Esgem Ltd, UK Professor David May FRS, University of Bristol, UK Dr. Majid Mirmehdi, University of Bristol, UK Dr. Henk Muller, University of Bristol, UK Professor Chris Nevison, Colgate University, New York, UK Professor Patrick Nixon, University of Strathclyde, Glasgow, UK Dr. Brian O'Neill, Nottingham Trent University, UK Dr. Roger Peel, University of Surrey, UK Professor Nan Schaller, Rochester Institute of Technology, New York, USA Professor G. S. Stiles, Utah State University, Utah, USA 0yvind Teig, Navia Maritime AS, Division Autronica, Norway Professor Rod Tosten, Gettysburg University, USA Dr. Stephen J Turner, Nanyang Technological University, Singapore Professor Paul Tymann, Rochester Institute of Technology, New York, USA Paul Walker, 4Links Ltd, UK Dr. Hugh Webber, Defence Evaluation Research Agency, Malvern, UK Professor Jim Woodcock, University of Kent at Canterbury, UK
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
Semantics of prialt in Handel-C™ Andrew BUTTERFIELD1 and Jim WOODCOCK2 1
Dublin University University of Kent at Canterbury
[email protected] e 2
Abstract. This paper discusses the semantics of the prialt construct in HandelC[l]. The language is essentially a static subset of C, augmented with a parallel construct and channel communication, as found in CSP. All assignments and channel communication events take one clock cycle, with all updates synchronised with the clock edge marking the cycle end. The behaviour of prialt in Handel-C is similar in concept to that of Occam [2, 3], and to the p-priority concept of Adrian Lawrence CSPP[4]. However, we have to contend with both input and output guards in Handel-C, unlike the situation in Occam, although prialts with conflicting priority requirements are not legal in Handel-C. This makes our problem simpler than the more general case including such conflicts considered in [4]. We start with an informal discussion of the issues that arise when considering the semantics of HandelC's prialt construct. We define a resolution function (H) that determines which requests in a collection of prialts become active. We describe a few properties that we expect to hold for resolution, and discuss the issue of compositionality.
1 Introduction This paper discusses the semantics of the prialt construct in Handel-C[l], a language originally developed by the Hardware Compilation Group at Oxford University Computing Laboratory. It is a hybrid of CSP [5] and C, designed to target hardware implementations, specifically field-programmable gate arrays (FPGAs) [6, 7, 8, 9]. The language is essentially a static subset of C, augmented with a parallel construct and channel communication, as found in CSP. The type system has been modified to refer explicitly to the number of bits required to implement any given type. The language targets largely synchronous hardware with a multiple clock domains. All assignments and channel communication events take one clock cycle, with all updates synchronised with the clock edge marking the cycle end. All expression and conditional evaluations are deemed to be instantaneous, effectively being completed before the current clock-cycle ends. 1.1 Notation In Handel-C, a typical program fragment showing two pr ial ts in parallel might appear as: par { prialt a!11 b?x }; prialt b!22 c?y
{ : PI ; break ; : P2 ; break ; { : P3 ; break ; : P4 ; break ;
A. Butterfield and J. Woodcock I Semantics ofpri alt in Handel-C
Here, PI, P2, P3, and P4 denote the continuation code that gets executed if the corresponding guard is deemed to be active. In CSPP [4], this could be written as: ((a!ll
5 (blx - P 2 (x))) || ((W22 - P3) 5 (cly -» P 4 (j/)))
We adopt a simpler notation, in which we ignore the guard values, variables and continuation processes, viewing each prialt simply as a sequence of guards, and those prialts in parallel simply being collected into a set, to give:
/ (a!, 6?) We do this because the concern here is how the decision is made regarding which guards are deemed to be true. We are not interested in this paper in the consequent actions, both of the guard communications and the subsequent process actions. 1.2 The Problem The behaviour of prialt in Handel-C is similar in concept to that of occam [2, 3], and to the p-priority concept of Adrian Lawrence CSPP[4]. However, we have to contend with both input and output guards in Handel-C, unlike the situation in Occam. In relation to ppriority, Handel-C does not admit processes with conflicting priorities, so presents a simpler problem semantically, than that found in CSPP. The synchronous nature of Handel-C also makes it easy to establish what prialt statements are participating in a synchronisation event at any given point in time. A particular consequence of this is that all the key events are deemed to occur simultaneously. Our interest is in determining the outcome when a number of prialts are simultaneously ready to run, i.e the outcome of the following situation:
k
(9nl,9n2,---,9nj,---,9nmn)
,
The guards can have only one of the following two forms: input guard c? or output guard c!. Handel-C allows the use of a def aul t guard as the last guard in a pr ial t, but again this is something which we will ignore for current purposes. Another constraint present in Handel-C is that a channel may appear only once in any given prialt. Handel-C also admits communication statements outside of pr ial ts ("naked communication"). However these can be modelled by a singleton pr ial t with a null continuation process: c!e
=
prialt{c!e:break}
In CSPP, semantics is given using Acceptances which are partial functions from traces to relations between sets of of offered and accepted events, where the domain of a partial function equals the set of traces of the process it denotes. A process is therefore described by describing how, after any of its traces (s), with an environment offering a set of events (E), it is willing to accept a specific set (A) of events. The notation used for this is s : £~~> A
A. Butterfield and J. Woodcock I Semantics ofprial t in Handel-C
3
However, in Handel-C, it seems to make more sense to view the prialts as making specific offers of sequences of events, rather than sets of events. We can view the prial t statement: ( g i , g - z , . . . , gn} as offering the events gi,g2,...,gnin that order. In effect what we have done here is to abstract the prial t notation to reflect precisely the prioritised list of events that it is willing to offer/accept. A general resolution mechanism then determines which events actually occur. This paper is a description of the formal semantics of that resolution mechanism, in a form sufficiently general that it can be incorporated into a variety of different semantics for HandelC—denotational [10], operational [11] or otherwise. As prial t resolution in Handel-C is deterministic, having the form of a total function from sets of prialts to their resolution, we adopt a formal notation most suited to describing functions, that of the "Irish School" of the VDM (VDM*) [12]. This can be described briefly as a functional subset of standard VDM (VDM-SL [13]), using equational reasoning rather than the logic of partial functions. A notation guide is provided as an appendix, as well as by explanations running through the text. Despite the Handel-C language's superficial similarity to CSP, we did not adopt CSP as a semantic modelling language here, because it lacks any notion of priority. Also, it was not clear when the work started if it is possible to capture this at all in CSP. Clear evidence for this is the work of Adrian Lawrence [14, 4, 15], which requires extensions to CSP in order to capture the notion of priority, and simultaneous events.
2 Informal Description We start with an informal discussion of the issues that arising when considering the semantics of Handel-C 's prial t construct. We consider the situation where we have n prial t statements which are to be resolved, and we assume, as already stated, a simplified model where prialts consist solely of the communication statements, hereinafter called requests. Each prialt is therefore a nonempty sequence of requests. We assume all prialts are well-formed, in that any given channel identifier occurs at most once in any given pr ial t. 2. 1 Priority: Absolute or Relative ? Initially, with Handel-C, it was assumed that a prial t of the form
assigned priority levels 1, 2, . . . m to guards g\, g%, . . . gm respectively. However all attempts to reconcile this absolute priority view with Handel-C's actual behaviour resulted in failure. After experimenting with prialts in Handel-C it became very clear that the notion of absolute priority was completely incorrect. Instead, each prialt simply defines a relative priority for the channels to which it refers. This is a strong indicator that p-priority, rather than e-priority [14], is the correct way forward. Merging the (locally total) ordering for each pr ial t results in either an overall partial order, or a relation with cycles. In the former case we use the ordering to determine the successful requests. In the latter case Handel-C reports an error. The simplest example of this is {(a!, 6!}, (6?, a?)} In [4], provision is made to give a meaning to this expression, but in Handel-C, which is deterministic, this is an un-recoverable compile-time error. If we try to resolve this in the
4
A. Butterfield and J. Woodcock / Semantics ofpri alt in Handel-C
Handel-C semantics by nondetenninistically choosing to "break" cycles, we end up losing some nice laws regarding the sequencing in pr ial ts and the way in which an offered event masks all subsequent events in a given prialt: (pi, . . . , & , . . . 0m) = (0i, . . . , &) when gt is "offered" This law does not hold if we resolve priority conflicts by breaking priority cycles. 2.2 Deriving Channel Partial Ordering Given prialts, we first take each constituent prialt and convert it into the corresponding relation (total linear ordering). We then merge all the relations together, one from each prialt, using relational union, and take the transitive closure to get one big relation (p). This relation captures the priorities between all participating channels. We then examine the (graph of) the relation for cycles. If present, we should flag an error condition at this point—the processes involved all diverge. If there are no cycles, then we have a partial ordering on all the channels which can then be used as the basis for determining the successful requests. Also this relation will in general have many minimal elements. At this stage we have all the prialts, and an induced partial ordering on the channel identifiers. We identify active channels (offered events) as being those which appear at least twice in (different) prialts, with complementary directions. 2.3 Pathological Cases Note that a channel may occur multiple times, in multiple prialts. This is satisfactory as long as all but two complementary requests for that channel are effectively masked by other channels with requests at higher priority. An example is:
which is equivalent to: {
, (<*?>,<&!>, In the event that we have one writer to a channel and multiple receivers, as in
the outcome (assuming all these channel instances are deemed active) is that each reader gets their own copy of the single written value. What happens in the case of multiple writers ?
In Handel-C Version 2.1 [16], the compiler flagged this with a warning, and in simulation, the value obtained from the channel was the bitwise logical-OR of the underlying representations of all the inputs ! It is clear that a multiplexor (which ends with a logical OR gate) is getting multiple input streams switched through simultaneously to the output. This is bizarre, but at least the compiler (i) informed the user and (ii) maintained the order independence of parallel composition of prialts: Pi II ^2 = Pi II Pi However, in Handel-C Version 3.0 [1], the compiler supplies no warning at all, and proceeds to accept data for the channel write occurring in the prialt which is first in the program text ! This means that { (c?:r), , (c!22) } £ { (c?rr), (c!22),
A. Butterfield and J. Woodcock / Semantics ofpri alt in Handel-C
5
The first writes 11 to x, the second writes 22 ! To handle this properly, we would have to talk about sequences of prialts being resolved, rather than sets. Also we would have to sacrifice the following very nice property of Handel-C (which was true for 2.1, but is now false for 3.0): P, || P2 = P2 || A We shall not concern ourselves overly with this case, but rather consider such programs at present to be ill-formed, with no obligation on our part to supply a formal semantics for them. They should certainly not be the result of any refinement methodology for calculating Handel-C programs from specifications, which is the ultimate goal of this work. 2.4 Identifying Active Channels Once we have identified the minimal active channels and removed them and the prialts in which they appear, we repeat the entire process of constructing the ordering afresh, and selecting the minimal active channels, with the remaining prialts. We continually repeat this until no further changes occur.
3 Formal Treatment We are defining a resolution function (72.) that determines which requests in a collection of prialts become active. 3.1 Input Types We start by considering as given a collection of channel identifiers: c € Ch
With every I/O request in Handel-C there is an associated direction: d <E Dir = {IN,OUT} This allows us to define an input-output request as a channel/direction pair: Request:
r € Req = Ch x Dir These "requests" are in fact our guards, as all such guards in Handel-C are of this form. In general we shall write c! and c? as shorthands for (c, OUT) and (c, IN), respectively. Having abstracted away from the details of the processes that are guarded in prialts, we view them as sequences of requests, where each channel occurs at most once: p e PriAlt = Req+ inv-PriAlt(a) = (# o elems o 7Ti*)cr = len cr The notation inv-PriAlt is the VDM* notion of "invariant", which gives a boolean expression defining which elements of the space are considered well-formed, which is analagous to the "healthiness conditions" of CSP/CSPP. What we are resolving is simply a set of pr ial ts: P € PriGrp = PPnAlt This structure is the input to our resolution process.
6
3.2
A. ButterGeld and J. Woodcock I Semantics ofprialtin Handel-C
Output Types
We now consider the information that needs to be output from the resolution process. Firstly, we need to know which channels are going to be active, i.e. able to communicate. We need to know the prial ts with which these channels are associated, so we can identify the specific guards involved, and consequently which guarded processes will be involved. We cannot determine this information from knowledge of which channels are active alone, because a given channel can be active in a proper subset of those pr ial ts in which it is mentioned. To see this, consider the following prialt collection, including guarded processes, (ignoring directions, with prial ts labelled for ease of reference):
2 : (a -» P3) 3 : 6 -> P4 4 : b -> P5
Here, both events a and b will proceed. Event a will involve prialts 1 and 2, leading to the subsequent execution of PI and P3. Event b will involve prialts 3 and 4, leading to P4 and P5 being activated subsequently. However, even though prialt 1 mentions 6, and 6 takes place, it is masked by the higher priority a event in that pr ial t, which is also active. Hence we need to couple a channel (event) identifier with the prialts which are actually involved. Our key output structure is therefore a map from channels to prial t-sets: 7 e CPMap = Ch -> PriGrp In the example above, we expect to see an output of / {a -> { (a -» Plt 6 -> P2), (a - P3) }} I {b~{(b^P4),
We can then determine the active processes by using the channel value as a lookup key in the associated set of prialts. So, for
we perform a lookup of a in both (a —> P! , 6 —> P2) and (a —» P3) to get PI and P3 respectively. Of course, in this paper we are ignoring the guards processes P, mentioned in this example, but at least this example shows in principle how that information can be incorporated. A key aspect highlighted by this example is a well-formedness constraint on channelprial t-set pairs. We expect the channel component to be present in every prial t in the corresponding set: inv-CPMap : CPMap -» B inv-CPMap 7 = Vc e dom 7 • inv-CPSet~,(c) inv-CPSet-,(c} = V/o € 7(0) • c e (elems o n^)p
A. Butterfield and J. Woodcock / Semantics ofpri alt in Handel-C
3.3
Intermediate Types
In order to describe the resolution process, it will be necessary to have some key intermediate structures. Firstly, we note that the input to the resolution process is very pria It-centric, but that it will be helpful to have a structure that is more channel-centric. Each channel present occurs at most once in any given prialt, in association with a corresponding direction. With any given channel, over all the prialts in which it appears, we would like to associate the directions of the corresponding requests. We therefore map each channel to a set of the request directions associated with it: K € ChReqs = Ch ->• IPDir As described above, we need build a relation over channels. We choose to model relations here as set- valued functions, as is conventional in VDM*: •R € ChRel = Ch -> PCh This formulation, while not very conventional outside VDM*, does make it very easy to extract the minimal elements, which denote the channels of highest priority. 3.4
Intermediate Operators
We now look at some operators for our intermediate semantic domains. We start with a function that converts the input of pr ial t sets into a channel request mapping: PriGrp —* ChReqs
o i*))P
Basically every entry p in P of the form
((ci.d.M^flfe),...,^,^)) is converted to an entry of the form {Ci i-» { d{ }, C'2 •-> { (I? }, . . - , Cfe i-> { 4
}}
All of these (one for each p 6 P) are then joined together using Q to provide one relation. The Q operator is a lifted (indexed) form of union, and has the effect of acting as relational union. In particular, we get
Once we have this channel request mapping, it becomes very easy to see if any channel is active. We simply look it up, and deduce that the channel is active if both directions are present: Act : ChReqs -» Ch -» B ActKc = K(C) = {IN, OUT}
8
A. Butterfield and J. Woodcock I Semantics of pr i a 1 1 in Handel-C
The central intermediate structure is that which captures the partial ordering on channel priorities induced by the collection of prial ts. Consider the following input structure:
For each sequence in P, we then discard the direction data (TTI*), and then convert the resulting sequence into the corresponding total ordering relation (using order), modelled as a set valued mapping, but shown symbolically here:
S' =
{ C.-1 -< Cj2 -< ---- <
-<
. {
Cifc, }
-< C«j; -< C,,*,, } ,
We then reduce this set with relational union (Q) to form a single relation, and take its transitive closure (TC)1. All of this is combined into the following operator: toChRel
:
PriGrp -> ChRel
toChRel = TC o ©/ o P(order o TTJ*) We can define order as follows: order order A order (c) order (c : c': a)
: = = =
C* -»(C ->PC) 0 {c H^ 0} {c H^ { c' }} 0 order (c': a]
3.5 Input-Output Relation We now describe the overall relationship between the inputs and outputs of the prialt resolution process (7£). The input is a set of prial ts, while the output is a pair (Resltn) consisting of a map from all active channels to the set of associated pr i a 1 ts, and the set of prial ts which are found to be inactive: (7, P) e Resltn = CPMap x PriGrp n : PriGrp —> Resltn As a precondition, we insist that the underlying relation on channels be acyclic (i.e. a partial order): pre-K p = (->o acyclic o toChRel)P We supply a partial postcondition which captures certain key properties we expect of the output. Let us assume that applying 7£ to P results in output (7. B), i.e
1
It is quite likely that the transitive closure is not necessary for this to work
A. Butterfield and J. Woodcock / Semantics ofpri alt in Handel-C
9
First consider 7, which is a map from of channel to prialt-set. All the prialts mentioned in 7, given by A = (u/ o rng)7 are active prialts. The collection of inactive prialts (B) should consist precisely of those not mentioned in A: Secondly, consider the set of all actually active request granted) channels (G):
G = dom 7 We expect this to be a subset of the potentially active channels (Q). These are determined by taking the channel-centric view of the input: K = toChReqs P taking its domain, and then filtering this with the active-channel predicate, itself parameterised by K: Q — (f ilter[Act«] o dom)« We expect AC Q. This is all collected together in the postcondition: post-H(P) *-+ (7, B) => dom 7 C (f liter [Acts] o dom)« A B = <*[A]P where K = toChReqs P A = (7°rng) 7 Observe that (0, P) satisfies the above post-condition, and the post-condition does not fully define the resolution process, but rather serves to specify some well-formedness criteria for the outcome. In particular, no mention is made of priority. We can define an invariant on the output tuple (7, B) which states that the prialts occurring in 7 do not appear in B: inv-Resltn : Resltn —>• B inv-Resltn(^, B) = (u/ o rng)7 n B = 0
3.6
Input-Output Function
We now look at the actual (functional) relationship between the input and outputs for resolution. We shall view resolution as repeatedly applying a resolution step CR£} to an initial "output value" (0, P) until no change occurs: K(P) = iterate (=) TIS (9, P) Note that "US preserves the invariant inv-Resltn. We observe that US repeatedly transforms the input as follows: (9, P) -> (71, A) - (72, P2) - * • • • - » (7«, Pi) -> (7n, Pn) The iteration stops when we reach (7n, Pn] such that KS(in, Pn] — (7«, Pn)- If we take (7o; PO) = (0, -P)» we can state that 7^9 has the following properties:
10
A. BufterfieW and /. Woodcock / Semantics of pri alt in Handel-C
1 . If 7&S changes the input, then 7 is larger and P is smaller: ftS(7n Pi) = (7H-1, Pi+i) =* 7, C 7,+i A Pi D PI+1
2. If 7&S does not change its input, then P contains no active channels, and v.v.: n, Pn) = (7n, Pn) = { IN, OUT } g rng(toChReqs P,,) The resolution step function applied to a tuple (7, P) identifies active prialt channels of highest priority in P and removes them, putting the corresponding channel and prialt information into 7: The key feature is how 7' and P' are computed. First from P we obtain the partial order (p) and channel-centric view (K): p = toChRel P and
/c = toChReqs P
We then identify all those channels that are active in p and then filter them to retain only those which are minimal (^4'): A' = mins(f ilter[Act K ]p) Given our representation of the partial order relation by Ch —» WCh (rather than the more traditional P(C7i x Ch)) allows us to give a concise definition of mins: mins p ==-<3[(u/ o rng)p](dom p) We simply take the domain, and remove from it anything that occurs in any of the range sets, thus leaving every channel which points to (lower priority) channels, but is not itself so pointed to. For every channel in A', we extract and map them to the set of all the pr ial ts in which they appear: (7' , P') = extr 7 P A' Here "extr" scans through channels in A', using them to identify the relevant pri alts in P which are then accumulated together with their channel in 7: extr 7 P 0 = (7, P) extr 7 P (Au{a}) = extr (7 U {a •-» <[Ha]P}) (-*[Ha]P) Ha p = a € (elems o KI*)P
A
Here
Note that the channel partial order is computed afresh for each iteration of 1ZS. In each iteration we simply extract all the highest priority active channels (i.e those lowest in the ordering).
A. Butterfield and J. Woodcock I Semantics ofpri alt in Handel-C
11
4 Resolution Properties We now mention a few properties that we expect to hold for resolution. In what follows we assume throughout that K = toChRel P and
ftP=(7,£) and these properties hold if all variables are identically decorated (subscripts or superscripts).
4.1 Termination K terminates The key to proving this is to recognise the crucial role played by A' in the definition of ~RS. If A' is empty, then extr leaves its argument unchanged. extr 7 P 0 = (7, P) In this case, the iterate function terminates. If A' is non-empty, then extr does at least one iteration, which increases 7 and decreases P. As P is finite, its reduction in size must eventually terminate. This proof is an informal one, but basically boils down to an assertion that the size of P supplies the well-founded measure required to show termination. 4.2 Resolution Progress As long as the set of prial ts to be processed is non-empty and contains active requests, then US will always make progress by enlarging 7 and shrinking P. If P is empty or has no active channels, then KS leaves its input unchanged: (7',P')
= 725(7, P) =$ dom 7' D dom 7 A P' C P V 7' = 7 A P' = P
Also: (7, P) = ^£(7, P) == (7 o P(f liter [Actfe]))P = A where K is derived from P in the usual way. 4.3 Resolution Chains Repeated application of 1ZS results in the following (finite) chain: (0,P0) - (71, Pi] - - - - - ( 7 i,Pi) - - - - - ( 7 n,Pn) where nS(jn, P,,) = (7,,, P?l) So we can define an ordering:
An open question: can we find (7a, Pa) and (75, P&), such that neither (7a, Pa) X (7^, P&), or vice versa, but there exists (7', P'), and numbers n and m such that
If not, then each member of the chain (6, P()) —> • • • —> (7,,, Pn) is uniquely determined by the first element, and hence by P0.
12
A. Butterfieldand J. Woodcock/Semanticsofprialt inHandel-C
4.4 Resolution Invariant The set of all pr ial ts present is invariant under the TtS operation. We can define the set of prialts present as: pAltOf(7, P) = P U (7 o rngb We then assert that: PAltOf(7,
P) = (pAltOf o US] (7, P)
We can extend this to the overall resolution process by asserting: = (7', P1} =
4.5 Resolution Validity We can define a predicate which determine when a Resltn structure is valid, namely that it is the result of iterating ~RS a finite number of times on a starting resolution structure with no mapping and a set referring to all prialts present: is Valid : Resltn -* B isValid(7,P) = 3n • ( 7 ,P) - 7l<Sn(0,pAltOf(7,P))
4.6 Active Channel Screening An active channel at a position in a prial t effectively removes the possibility of a subsequent channel in that prialt being selected.
where Ac = toChReqs(PLJ{p}) It is this property which is lost if we admit cycles in the priority relation and then allow for a non-deterministic breaking of such cycles—consider breaking the above just after p\i\. The entry p[i + 1] (if it exists) is no longer "screened" by the zth entry and in fact becomes minimal in the new relation. 4. 7 Disjoint PriAlt Groups If we have two disjoint collections of prial ts (channels distinct) then we can resolve them together or separately. dom KI n dom «2 = 0 => K(P\ U P2) = (71 U 72- #1 U B2)
A. Butterfield and J. Woodcock I Semantics ofprial t in Handsl-C
13
5 Compositionality of prialt In order to achieve full compositionality, we need tofindan operator | such that
This property is necessary in order to provide a denotational semantics, and makes it much easier to produce a set of useful laws for reasoning about Handel-C programs. It turns out that this is very easy to achieve, at least in a technical sense. Given the output of a resolution, we can use pAltOf to give us the original input data, so we can define \ as: (7', P'} | (<J', Q') =ft(pAltOf(V,P') U pAltOf(<5', Q')) This definition, while technically correct, seems unsatisfactory, in the sense that it involves taking both resolutions, undoing them, throwing all the original pr ial ts back into the pot, and redoing the entire resolution process again. This idea that composing resolutions involves retracting previously granted requests seems a little unsatisfactory. However, as observed in [14], when processes are combined such that relative priorities get changed, then events accepted by the combined process my be different from those accepted by the individual process. This is in fact inherent in the whole concept of priority. Consider the following example: (a!, 6?)
(a?,c?)
In both cases, channel a has highest priority, but is inactive, so channels 6 and c are selected: K(P] U(Q)
= ({&->{ (a!, &? = ({<:->{ (a?, c?>,(d)}},0)
However, when we combine P and Q:
we find the channel a is now active, and removing its prialts from the pool results in channels b and c being inactive, when the leftover pr ial ts are considered: K(P U Q) = ({a ^ { R 6?}, (a?, c?) }}, { <6!},
14
A. Butterfieldand/. Woodcoclc/Semanticsof prialt inHandel-C
channels are totally ordered within those disjoint subsets. All we have to do is to add one new prialt which ranks the highest priority channel from one subset as lower than the lowest priority channel of the other subset to completely alter the ordering relationship. It appears very difficult to handle a case like this without re-computing the resolutions for the entire set of prialts.
6 Conclusions and Future Work We have described a formal process for resolving pr ial t communication requests, suitable for handling the type of prialts found in the Handel-C programming language. We have shown this process to be compositional in nature, at least in a technical sense, and we have described some key properties. The first task to be done at this stage is to integrate this material with the existing semantics for Handel-C [10, 11] without prialt, and then to use the denotational semantics to validate a collection of useful language-level laws for Handel-C. The long term goal is to develop a development methodology for deriving Handel-C programs from appropriate specifications. This involves the development of a suitable refinement calculus, which will be integrated into the unifying theory of Hoare & He [17], using the Circus refinement calculus [18]. On a more speculative note, we believe that it may be possible to encode a semantics of Handel-C in standard CSP, by using the functional language component as found in machine readable CSP (see [19, Appendix C]) to encode the resolution. This would use a mechanism of first lodging and then resolving requests similar to that described in [10] for non-prial t communication. The reason that the extra machinery of CSPP might not be required is simply the constrained deterministic nature of Handel-C. We do not see this as being possible for prialt in a more general setting, as described in [4] References [1] Celoxica Ltd. Handel-C Language Reference Manual, v3.0,2002. [2] A. W. Roscoe. Denotational Semantics for occam. In S. D. Brookes, A. W. Roscoe, and G. Winskel, editors, Seminar on Concurrency, volume 197 of LNCS. Carnegie-Mellon University, Pittsburgh, PA, Springer-Verlag, July 1984. [3] A. W. Roscoe and C. A. R. Hoare. The laws of occam programming. Technical Monograph PRG-53, Oxford University Computing Laboratory Programming Research Group, February 1986. [4] A. E. Lawrence. Acceptances, Behaviours and infinite activity in CSPP. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 17-38, Amsterdam, Sept 2002. IOS Press. [5] C.A.R. Hoare. Communicating Sequential Processes. Intl. Series in Computer Science. Prentice Hall, 1990. [6] I. Page and W. Luk. Compiling Occam into field-programmable gate arrays. In W. Moore and W. Luk, editors, FPGAs, Oxford Workshop on Field Programmable Logic and Applications, pages 271-283. Abingdon EE&CS Books, 15 Harcourt Way, Abingdon OX14 1NV, UK, 1991. [7] M. Spivey and I. Page. How to design hardware with Handel. Technical report, Oxford University Hardware Compilation Group, December 1993. [8] J. P. Bowen, He Jifeng, and I. Page. Hardware compilation. In J. P. Bowen, editor. Towards Verified Systems, volume 2 of Real-Time Safety Critical Systems, chapter 10, pages 193-207. Elsevier, 1994.
A. Butterfield and/. Woodcock/Semantics ofprial t in Handel-C
15
\9] Adrian Lawrence, Andrew Kay, Wayne Luk, Toshio Nomura, and Ian Page. Using reconfigurable hardware to speed up product development and performance. In Will Moore and Wayne Luk, editors, FieldProgrammable Logic and Applications, pages 111-119. Springer-Verlag, Berlin, August/September 1995. Proceedings of the 5th International Workshop on Field-Programmable Logic and Applications, FPL 1995. Lecture Notes in Computer Science 975. [10] Andrew Butterfield. Denotational semantics for prialt-free Handel-C. Technical Report TCD-CS-2001-53, Dept. of Computer Science, Trinity College, Dublin University, 2001. ftp://ftp.cs.ted.ie/pub/tech-reports/reports.01/TCD-CS-2001-53.pdf. [11] Andrew Butterfield. Interpretative semantics for prialt-free Handel-C. Technical Report TCD-CS-2001-54, Dept. of Computer Science, Trinity College, Dublin University, 2001. ftp://ftp.cs.tcd.ie/pub/tech-reports/reports.01/TCD-CS-2001-54.pdf. [ 12] Micheal Mac an Airchinnigh. Conceptual Models and Computing. PhD dissertation, University of Dublin, Trinity College, Department of Computer Science, 1990. [13] ISO/IEC. Information technology - Programming languages, their environments and system software interfaces — Vienna Development Method - Specification Language — Part 1: Base language. International Standard 13817-1, International Standards Organisation, 1996. www. iso. ch. [14] A. E. Lawrence. CSPP and event priority. In Majid Mirmehdi Alan Chalmers and Henk Muller, editors, Communicating Process Architectures 2001, Concurrent Systems Engineering, Amsterdam, September 2001. IOS Press. [15] A. E. Lawrence. HCSP: Imperative State and True Concurrency. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 39-55, Amsterdam, Sept 2002. IOS Press. [16] Embedded Solutions Ltd. (now Celoxica Ltd.). Handel-C Language Reference Manual, v2,l, 2000. [17] C. A. R. Hoare and He Jifeng. Unifying Theories of Programming. Series in Computer Science. Prentice Hall, 1998. [18] Jim Woodcock and Ana Cavalcanti. The Semantics of Circus, volume 2272 of LNCS. 2nd International Conference of B and Z Users, Grenoble, France, Springer, January 2002. [19] A. W. Roscoe. The Theory and Practice of Concurrency, international series in computer science. Prentice Hall, 1997.
16
A. Butterfield and J. Woodcock I Semantics ofprialtin Handel-C
A Notation and Definitions We provide a brief glossary of notations and definitions of the less familiar VDJVi* notations used in this paper. Symbol
Description Examples Cardinah'ty p/ Set Mapping P/{ *,y, *} = {/(*). /(y)./(*)> el ems Sequence Elements elems{6, a, c, a) = {a, 6, c} len Sequence Length Sequence Mapping P/(*,y, «) = {/(*), /(y),/(*)> zth Projection T T i ( a i , . . . a i } . . . , a n ) = a, VTi ff[i] Sequence Indexing U Map Extension Map Override t dom Map Domain rng Map Range //(x) Map Application (lookup) Set or Sequence Reduction +/ (3, 5, 4} = 12 V Transitive Closure Operator TC Sequence to Total Order order acyclic Acyclic Predicate filter Sequence Filter ~Q[S] Map Removal w.r.t. S -<}[{a,c}]{a H-* 1,6 •—»• 2,ci—> 3} = {61—»• 2} Minimal Elements of a P.O. mins
#
r
Given any binary operator * : A x A —> A and an arbitrary indexing set X, we "index" the operator, calling it Q : (X — » A ) x (X —> A) —> (X —> A) and defining its action so that: ( lx
, f /^U{xi->a}, ifx £ dom/i »-»• a)} = < , f f . ! .. \ p, ^ {x >-> n(x) * a}. tfx€ dom p.
Effectively * has been "lifted" from acting on A to acting on X —»• A, essentially operating on the map range elements. The iterator application iterate P f x repeatedly applies / to x and uses P to examine the before and after values to determine when to stop: iterate : (A x A -» B) - » ( A -f A) -> A -» A iterate P / x = let x' - f ( x ) in if P(x, x') then x' else i terate P f x'
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
11
Acceptances, Behaviours and Infinite Activity in CSPP Adrian E. LAWRENCE Department of Computer Science, Loughborough University, Leicestershire, LEU 3TU UK A. E. Lawrence @ Iboro. ac. uk Abstract. The denotational semantics presented here defines an extension of CSP called CSPP. It includes a full description of infinite behaviour in one simple model using only finite traces. This is true for both finite and infinite alphabets. The structure is a complete lattice, and so also a complete partial order, under refinement. Thus recursion is defined by fixed points in the usual way. It is also a complete restriction metric space so this provides an alternative treatment of recursion for contraction mappings. Keywords: CSP; CSPP; Denotational semantics; formal methods; concurrency; parallel systems; occam; hardware compilation; priority; p-priority.
CSP is a process algebra which describes processes which engage in events. In the original version of CSP presented in [1] and reprinted in [2] processes were sequential, but could run in parallel at an outer syntactic level. They could communicate by unbuffered message passing. Hence the name Communicating Sequential Processes. The communication was more like 7r-calculus than modern CSP: channels did not appear explicitly. The second version of CSP is described in a very accessible way in Hoare's book [3]. In this version, parallelism is ubiquitous, and the name Communicating Sequential Processes is not so obviously appropriate. However, the concurrency is interleaving, so events occur in sequence and the historical name can be defended on those grounds. The language has spawned many variants and even competitors, but the mainstream version has not changed substantially in recent years and is described comprehensively in Roscoe's landmark text [4]. CSP also spawned the concrete language occam [5], which was designed by May in consultation with Hoare and other CSP researchers. It is an imperative language and so had to encompass notions which did not appear explicitly in the more abstract CSP. occam was one of the first practical concurrent imperative languages with a mathematical foundation. That foundation was CSP extended with algebraic semantics to cover aspects of state. An almost complete denotational semantics is given by Goldsmith, Roscoe and Scott in [6] and [7]. occam, originally closely associated with the transputer, inspired and enthused a world wide community. It is so innovative, simple and elegant that it revolutionised the approach of many practitioners. It must be said that those very same qualities, coupled with a longstanding lack of support for conventional tools and environments, since rectified, alienated others wedded to tradition. When occam was introduced its close connections with CSP were seldom mentioned, and occam practioners were typically unfamiliar with the algebra. The occam community, who were in effect the main 'applied-CSP' practioners, developed many techniques and insights largely independently from main stream CSP. occam was regarded from the beginning as merely an initial foundation for more powerful developments, particularly in raising the level of abstraction. The language has indeed
18
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
evolved from proto-occam to occam-3, although the level of abstraction has not changed significantly so far. It also has small extensions for hardware compilation: see [8], [9] and [10]. But so far it has proven to be remarkably resistant to major change or extension: it seems that the core language is so robust, simple and transparent that modification is almost always a backward step. The work on CSP reported below originated in this context: • occam includes a way to give priority to some CSP events: conventional CSP abstracts from such concerns. This problem became a serious concern when the author attempted to prove that his hardware, software and firmware design in the HARP( [11],[12]) codesign circuit boards and Handel-AS compiler was correct. • Hardware compilation involves true concurrency. • The semantics in [6] and [7] is rather difficult to use informally: a simpler way to understand imperative state in the same way as CSP events is very attractive. • The characteristic features of occam stem from its mathematical foundations. In seeking successors or extensions, it is essential to have a full mathematical theory on which to build. This led to two incremental extensions of CSP: CSPP and TiCSP. CSPP extends CSP by including priority: the trailing P stands for priority. TiCSP is a further extension of CSPP and includes true concurrency and additional constructors for hardware compilation: the leading H stands for hardware. These ideas were first presented informally at the Twente WoTUG-20 technical meeting. Readers with only passing acquaintance with CSP may wish to be reminded that there are two elementary processes Skip and Stop. Neither process does anything, but Skip does it successfully! That is Skip terminates: it will pass control on to a successor, while Stop is a deadlock. The process representing livelock div might also be included as an 'elementary' process: that represents a situation when a broken program goes into an infinite internal loop. A common way of building more complex programs is to use prefixing: if a process performs an event a before stopping, that is written as a —> Stop. And the process that performs a before passing control to a successor is a —>• Skip. Notice that a is an event while both a —> Skip and Skip are processes. Processes just engage in events. Almost: as in the case of Stop, they might refuse to engage in events. A slightly more general prefixing is exemplified by/i : {1,2,3} —»• Skip, a process that is willing to engage in the event n where n is drawn from {1,2,3}. Notice that here we are regarding ordinary integers as CSP events. That is fine: CSP does not specify the nature of events. Here we interpret such an event as the reception of a number from some sender. Our process is willing to accept one of the three numbers and then terminate. This is an example of external choice: the sender decides on whether to send 1,2 or 3, and our receiver process follows that decision. It is 'driven'—the choice is made—externally: hence 'external choice'. The more general form of external choice is written D, so n : {1,2,3} -> Skip = (1 -» Skip) D (2 -> Skip) D (3 -» Skip), for example. Now consider (3 —> 1 —»• Skip) n (4 —» 2 —> Skip). A most important aspect of CSP is that it supports abstraction—hiding of detail. Thus ((3 -> 1 -»• Skip) D (4 -» 2 -* Skip)) \ {3,4} is our same process, but now with the events 3 and 4 hidden. Because 3 and 4 are hidden, they are unconstrained, and can happen freely as far as the inner process is concerned. The
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
19
justification for this idea is not difficult, but must be omitted from this brief overview. It is now an internal matter whether 3 or 4 occurs, so the external view of the process is non deterministic. We may observe a process that is prepared to accept the number 1 and will refuse 2, or the reverse. In fact ((3 -> 1 -» Skip) D (4 -» 2 -> Skip)) \ {3,4} = (1 -> Skip) n (2 -+ 5/tzp) . Thus n is the internal or non deterministic choice operator. Concurrency has been implicit in all our examples so far: the 'receiver' n : {1,2.3} —» Skip needed to interact with a 'sender'. So the receiver and sender are running in parallel. Thus we may have (1 -> Skip)
(n : {1,2,3} -» Sfcp) = (!->• 5*ip),
where | is the parallel operator which requires that every event be a joint event of both partners. A more general version is ||: the parallel partners synchronize on the joint events in E
E, but are otherwise independent. This is enough to express any sort of parallelism including interleaving 11 which is just . 0
Sequential composition is written as ?, so (1 -»• Skip) 5 (2 -> Stop) = 1 -» 2 -> Stop . The last of the basic CSP operators is recursion. /zP • n : {1,2,3} —»• P is the process that will accept an unending sequence of events drawn from {1,2,3}. CSPP extends CSP by the addition of biased and neutral versions of the external choice and parallel operators. Since prefixing is a form of external choice, that also gets extended as does interleaving which is a particular case of the general parallel operator. Thus (1 —> Skip) D (2 —>• Skip) is a process that will choose to accept a 1 when there is a choice. Frequently, there is no choice: a sender is only prepared to send a single integer on any given occasion. Then (1 —» Skip) O (2 —> Skip) behaves in just the same way as the ordinary (1 —> Skip) n (2 —> Skip). All this has been described in previous papers presented in this series of conferences: [13],[14], [15] and [16]. Merely adding extra syntax to CSP does not achieve very much in its own right. It is necessary to ensure that a consistent and meaningful structure emerges. Since we are building an algebra, we must be able to identify what algebraic laws are obeyed, at the very least. The original and traditional way to do that for CSP is by way of a denotational semantics. That is a mapping from syntax into a mathematical structure embedded in a known theory typically incorporating some sort of fixed point property. The standard denotational semantics of CSP is based on Failures and has a long history some of which is recounted in [4]. It took a while for the right concepts to emerge, and various difficulties to be recognised and sidestepped. But there have always been severe problems in handling certain sorts of infinite behaviour connected with unbounded non determinism. These have been largely overcome mainly by outstanding work by Roscoe, see [17]. Yet it has to be said that when the full theory is included, the whole semantics is somewhat convoluted. And the treatment of termination is a little contrived. It is believed that the semantics presented in this paper completely overcomes all of those problems as well as having additional merits. The design of CSPP was driven by the acceptance denotational semantics below. This too evolved, and various versions were explored and presented in this conference series. A number of difficulties were encountered but have now been resolved. The version in this
20
A.E. Lawrence /Acceptances, Behaviours and Infinite Activity
paper is pleasingly simple and appears to have solved all the outstanding problems with a single theory. Tune will tell whether it will become the standard semantics for CSP and its extensions. This semantics defines HCSP including true concurrency with almost no change: that is the subject of the companion paper [18]. 1
Introduction
The theory introduced here is a significant advance over earlier denotational semantics for CSP: • It captures infinite behaviour in the simplest and only model. • It is a complete partial order, indeed a complete lattice, under refinement in all cases. • / is a token rather than a first class event; a new token X is introduced to represent divergence. • It deals with certain 'awkward' processes like Skip D P in a simple and natural way. • It is expressive enough to capture extensions to CSP, in particular priority. The main family of denotational semantics for CSP is based on Failures, see [3], [4],[19], [20] and the references therein. The principal idea is to characterise a process by what it can refuse to do after it has performed some trace, that is a sequence of events. The idea is elegant and economical, if somewhat anti-intuitive. However there are some difficulties, especially when infinite behaviour is included. In particular, refinement then fails to be a complete partial order. In contrast to Failures, the Acceptance semantics below is based on the idea of simultaneously offering a process a number of mutually exclusive events, and observing which event or events can be chosen or accepted. A problem which seems to be inherent in a semantics based on Failures is an inability to describe conditional refusal. So if a process prefers to engage in an event a when it is available, but otherwise will perform b, then b will sometimes be refused and sometimes accepted. The best that a simple Failures model can do is to model that non deterministically. CSPP extends CSP to include priority, so the last observation is very pertinent: acceptance semantics was devised as a means of defining CSPP precisely. Traditionally CSP abstracts from priority, modelling it with nondeterminism, presumably arising from the underlying Failures semantics. And sometimes it is said that CSP is not appropriate for modelling 'fine detail' like priority. Yet the correctness of some systems, including those built directly in hardware, depend upon priority. So there is every reason to capture them in a formal and precise way in a single unified language. This is a prime motivation for extending CSP and giving it a rigorous definition. Nondeterminism is fundamental and is modelled here in a rather direct way by allowing a process to have multiple behaviours. That is a type of internal choice: a parallel partner cannot control which is selected. Nondeterminism also arises when a particular behaviour does not make a unique choice of an event from those offered to it. This is a extension of external choice: such a response indicates a behaviour which is prepared to be flexible or compliant: it is prepared to negotiate or be driven by a parallel partner into the refinement of the choice down to a particular event. If there is no such partner, or the response of the partner is also compliant, then the final selection of a particular event is nondeterministic. The entity 'offering' the choice of events to a process is abstracted as the environment. This is generally the set of parallel partners of the process. But even if our process represents
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
21
a whole system, it can at least be observed, perhaps passively, from the outside: one expects it to engage in at least one event if it is of any use. In this sense, there is always an outermost environment. Those familiar with the standard semantics of CSP will recognise much of this picture, but the interpretation in Acceptances is enriched. Events remain atomic and require the joint participation of all partners. It is useful to think of some such partners as 'always ready and compliant' as in a passive observer. 2 Extended CSP CSP originated in the context of concurrent software typically implemented on a single sequential processor. Major issues were synchronisation and communication. And CSP unified these in die notion of an atomic event which involved the joint participation of, normally two, processes. The joint participation involved both partners in such events mutually blocking. The event could only proceed when both were ready: there was a 'handshake' between partners. As technology advanced, CSP came to be used to describe situations involving true concurrency. In particular, the theory was used to design and program massively parallel arrays of transputers, although since individual transputers were not usually synchronous across arrays, it might be argued that the concurrency involved was still interleaving. In consort, CSP was used to design occara, and that introduced new concepts. It introduced parallel assignment as in p, q : = q,p. The semantics was equivalent to true concurrency although it was not expressed in that way. However, the assignments were not regarded as CSP events, so that did not immediately raise the question of true concurrency in CSP. But occam also introduced priority as PRI PAR and PRI ALT, and these were obviously direct extensions of the CSP operations || or 11| and D. However there was no formal semantics for these extensions. The first attempt to provide such was made by Fidge in [21], but since fundamental issues of recursion and fixed points were not considered it was incomplete and so did not succeed. The topic of priority is sometimes said to be a 'real-time' issue rather than a matter of 'logical' design, and therefore legitimately separated from matters of correctness. This is an appealing 'separation of concerns' argument at first sight: it seems to have more force when applied to || and ||, the parallel operators, rather than to D and D, the external choice operators. But many programs, especially those involving PRI ALT, depend on priority for their correctness. It would be absurd to require a second precise language with another rigorous semantics to establish what PRI ALT meant and to be able to prove the correctness of programs utilising it. Far better to extend CSP rigorously which is what CSPP achieves. And that is done in a way that maintains 'separation of concerns'. For PRI ALT is a refinement of ALT. That is if we establish the correctness of a program that employs ALT, then it follows that ALT can be replaced by PRI ALT, and the program is still correct. Although CSP and occam were used in the design of hardware including the transputer, direct use in hardware compilation was a later development. The hardware was usually a synchronous circuit and was truly concurrent, occam simultaneous assignment was used extensively. And when two processes engaged in a joint event, the 'handshake' was typically implemented with a signal in each direction. But there were some circuits where the receiver was 'always ready': it was a waste of hardware to provide redundant signals. So the idea of a process that was 'always ready' to engage in an event arose. Although this idea seems disturbing to some only familiar with software, it is still a valid interpretation of CSP and
22
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
CSPP. We can maintain the fiction that such a ready process could in principle inhibit the event, but it 'chooses' not to do so. Once one has admitted that interpretation of an event, it opens the door to the more radical idea that 'actions' like assignments might also be regarded as events. Quantum physicists are very familiar with the idea that a measurement can affect the subsequent observed behaviour of a system. And that measurement involves the reception of information. While conventional assignment can be described perfectly well by CSP, that normally requires an explicit model of a variable. By extending the idea of an event to include assignments where an observer is 'always ready' to see the action, we can capture a large part of the semantics of an occam-like language within the framework of an extended CSP. Since sequential execution is included, this provides a radical unifying theory for imperative programming in general. There is a further extension of CSPP which covers such matters. For now, merely note that events in the theory below may have non traditional interpretations. Hardware also throws up the idea of events that involve more than two processes. In software, events normally include precisely two processes joined by a channel (which is formally merely a set of events). The most obvious example of an event involving many processes in hardware is a clock edge in a synchronous circuit. The underlying theory of CSP did not restrict events to two processes. In fact the number of processes involved in an event is not even well defined as evidenced by the identity (a -* Skip) || (a -» Skip) = (a -»• Skip)
.
Almost all descriptions of CSP aimed at software deal with events as channels. Below we concentrate on the underlying events for the reasons indicated above, and because the semantics is more obvious. Channels are used in applications and examples. Here only the modest extension of CSP to CSPP is addressed, but this also serves as a foundation for further extensions including TiCSP. Only interleaving concurrency is covereded here. TiCSP includes true concurrency. The main extensions to standard CSP here are: • A miraculous process, T. This is an unimplementable process introduced for technical reasons: it ensures that the structure is a complete lattice under refinement. • Priority and compliant processes. Thus D has refinements D , D and D, for example. • 'Fair' refinements of standard operations like interleaving. Thus ||| is a refinement of interleaving that must eventually favour each partner. This is related to priority and infinite behaviour and expresses ideas found in temporal logic. 3 Why denotations! semantics? Much recent work in CSP concentrates on operational semantics. That work is largely concerned with modelling and proving systems correct particularly with the aid of model checking [22]. FDR [23] is a model checker for CSP which is built around such an operational semantics. Here there is a different emphasis: the main thrust is in building tools and designing languages for codesign, especially for hardware design. In particular transformation laws, in effect algebraic semantics, are required which can be used in compilers. Acceptance semantics is abstract and simple, yet can capture more detail than standard CSP and can be applied to a wide range of situations. It can establish the algebraic identities needed for practical tools. And the insight that it affords in aiding understanding and recognising new realms of application should not be dismissed.
A.E. Lawrence /Acceptances, Behaviours and Infinite Activity
23
4 Some intuitions underlying Acceptances 4.1
Events and Traces
Classical CSP models concurrency by interleaving: no more than one event can occur at any instant. Since such events may be separated by arbitrarily small times, at least in untimed CSP, this suffices for many purposes. If we wish to model two events that occur together, that is done by supposing that closer examination would reveal that the events actually did happen at different times, and so can be represented by a trace: a particular sequence. In fact, the more precise measurement of the order is not available, so that information is missing, and consequently the order is indeterminate. Thus if we wish to model two events a and b that appear to happen together either because that is the reality, or because we cannot determine the times of occurrence sufficiently precisely, then that is handled as the pair of orders (ab) and (ba). The corresponding elementary process is (a —> Skip] 111 (b —> Skip) = (a —» b -—> Skip) D (b —> a —* Skip): one or other order will happen, we know not which. This identification is established rigorously in section 5.9 on page 28. On occasions this model is inadequate. Modelling synchronous hardware in full detail including explicit clocks requires a more realistic model. HCSP [18] is an extension ofCSPP which is further extended to include true concurrency. But CSPP is a more conservative extension of standard CSP and retains interleaving semantics. 4.2
Priority, simultaneous offers and concurrency
CSPP is aimed at capturing priority. Consider two events a and b again, and a process that gives priority to a in preference to b. If the process is presented with both a and b simultaneously, then the process will select a. At first sight, we seem to be faced with the need for true concurrency once again. Yet priority arises in real software systems implemented on entirely sequential processors, so that cannot be the case. The explanation here is that the state of the system, in particular what events are available, is sampled at various points. Acceptance semantics captures these ideas, but can also be extended as in HCSP to handle true concurrency. It is intended to be close in spirit to Failures semantics, but easier to understand: it is simpler to identify what a process accepts rather than what it refuses. Hence the name. 4.3
Environments, simultaneous offers, compliant responses, and events.
A basic idea is that of an environment with which a process interacts. That environment is formed by the concurrent partners of the process if any. But at the outer level, it may be thought of as a passive observer. The only purpose of the environment is to engage in common events. At the outermost level, a passive observer merely 'accepts' - observes - any event that the process chooses to perform. This is an example of a 'compliant' environment. Thus if a particular process has a repertoire of events a, b and c, then a compliant environment might make the compliant 'offer' X = {a, b, c}. That means that the environment is prepared to engage in precisely one of the events in X. It is compliant in that it leaves the choice to the process. If the process opted for event a, then we will describe that by saying that the process 'accepted' {a} in response to the offer (a, b, c}. But the environment is also the mechanism for capturing the interaction of cooperating (and also contending) processes. If a process is placed in parallel with another process only prepared to engage in the shared event a, that would in effect constitute an environment offering X = {a}. If our process is only prepared to perform another event b, then deadlock follows.
24
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
An offer like {a, fr, c} is a simultaneous presentation of a choice among 3 events. This is quite distinct from simultaneous events which are only present in TiCSP : the compliant offer is to perform exactly one of the events. This is just what we need to capture priority. Notice that explicit priority is sometimes only required when true concurrency is not available: if both events a and b are available and we can execute them together, there is no need to use priority to select just one. But there are uses for explicit priority even when true concurrency is available: there may still be mutually exclusive choices with a preference when both are available. 5
Simple examples
The examples here serve to introduce notation and fix ideas. It is assumed that the reader has had some previous exposure to CSP.
5.1 Stop Consider Stop. Like all processes, it starts with an empty trace {}: initially it has done nothing: And Stop continues to do nothing: if it is offered the set of events X, then it accepts nothing. We write the acceptance semantics as {) : X ~» 0
or
Stop ::: {} : X ~» 0
when we need the process name.
Stop is an example of a process which has only one behaviour, namely {} : X —+ 0. That will be explicit in section 13.1 on page32. 5.2
Prefixing: a —» Stop
a —> Stop also has a single behaviour. If it is offered the event a initially, it accepts it:
{) : {a} - {*} More generally
Once the only behaviour of a —» Stop has accepted a, the trace of its past actions is (a). It subsequent behaviour is (a) : X ~» 0
.
X ~» {a} can be pronounced X accepts {a} or X may accept a, but X ~* 0 is probably better pronounced as X may refuse or X may accept nothing. A behaviour has a set of traces, here just {{), (a)}, and associates each trace with the responses to all possible offers X. 5.3
Nondeterminism: Stop l~l (a —» Stop)
Consider Stop fl (a —»• Stop). This is a process that may behave like Stop or like (a —» Stop). And the choice is internal: the environment cannot influence which. An implementation might consist of either process alone in which case Stop n (a —> Stop) can be regarded as a specification. Or a system might be capable of behaving like either component process, but makes arbitrary choice between the options when it is run.
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
25
These two possibilities are mapped very literally here: there is a distinct behaviour matching each of the component processes. The process is modelled by this set of two behaviours Even when a is among the events offered initially, it may be refused: {} : X -~* 0. But we still have {} : X —> {a} when a € X as well. Each corresponds to one of the two behaviours of the component processes. We write bi :: {) b2 ::{} b'2 :: (a)
: X ~» 0 : X - X D {a} : X ~» 0
for the two behaviours b\ and b^. Since these behaviours are uniquely associated with the component processes here, we can abuse notation in such cases to write: Stop (a-* Stop) (a -» Stop)
:: {} :: {) :: (a)
: X ~* 0 : X ~» X n {a} : X ~» 0
So in general a process is identified with a set of partial functions each of which represents a possible behaviour. A behaviour b takes a possible trace s and yields another function which describes what is accepted when a set of events X is offered. Thus b :: {} : X ~» 0 means that is the function X i—> 0. More precisely
where S is the set of all events. 5.4 Termination: / and Sfap The processes illustrated above simply cease activity, but useful processes normally terminate. That is how a process passes control to a successor on successful completion. In CSP this is done with a special token written as /. An associated process is Skip which has a single behaviour which does nothing except terminate:
In CSPP / has a special status: it cannot be offered to an event, and it cannot appear in a trace. In classical CSP, / is treated as an ordinary event for most purposes so it may appear in traces. To do this consistently requires considerable ingenuity and awkwardness in Failures semantics. These problems do not arise in Acceptance semantics. A characteristic property of / is (a -»- Skip) § (b -> Skip) = (a -> b -* Skip) This has a behaviour with (ab) : X -
The / which passes control from a —*• Skip to b —> Skip is a hidden synchronisation between the two processes, and does not appear in the trace: rather it is associated with the instance of the sequential constructor | which "glues" the two processes together.
26 5.5
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity Recursion: // P • a —» P
CSP includes solutions to recursive equations like
Q = a-+Q . which defines an infinite process. This can be written Q = nP*a-*P liP • f(P) denotes denotes the unique solution of the equation P = f(P) if one exists. If there is more than one solution, then it selects the least, that is the most non deterministic, of the available solutions, if any. Determining under what conditions such solutions exist is one of the primary tasks in setting up a denotational semantics for any CSP variant. Acceptance semantics based on behaviours shows that CSPP constitutes a complete metric space. This shows that all functions/ which are contracting with respect to that metric have unique solutions. With the addition of a 'miraculous' T process, CSPP is also a complete lattice and therefore also a complete partial order. The order relates processes with common behaviours where those that are more deterministic are 'better than' those which exhibit more internal choice. This refinement order is defined below. It then follows from standard results about fixed points that all sensible recursions in CSPP have 'best' - most deterministic - solutions. Be that as it may, note for the moment that p,P • a —» P is an infinite process with traces consisting of a sequence of a's. Its semantics is
(a") : X ~* {a} n X where (a") is a trace of n consecutive a's. It is unbounded: it can perform more than n events for any n. This is our first example of a behaviour with an infinite domain. The domain is the set of all finite traces consisting of sequences of as. This ability to represent infinite behaviour using only finite traces is simple, yet significant. It is the extra layer of structure at the individual behaviour level that yields the expressive power to distinguish truly infinite behaviour. 5.6 Hiding, divergence and/: (a —> b —» Stop) \ {a} An important feature of CSP is that it includes hiding. An example: (a -» b -» Stop) \ {a} = b -» Stop. So a becomes an 'internal event' invisible to the environment. This is the primary abstraction mechanism. It introduces nondeterminism and divergence. So (a^(lMp.b^p))\{b},
(1)
is a process which performs a, but then goes into an infinite loop which no longer interacts with its environment, and there is no way to exercise control. This livelocked process is said to be divergent in analogy with uncontrolled infinite behaviour. Such situations are captured with the aid of another pseudo-event X. The notation is intended to indicate undesirable non-terminating behaviour contrasting with /. For equation (1) the behaviour is: () : X ~» {a} n X (a): X -» {X}.
A.E. Lawrence /Acceptances, Behaviours and Infinite Activity
21
When priority is present, processes like ((a -> Stop)H(b -> Stop)) \ {a} = Stop, arise, (a —> Stop) D (& —* Stop) is a process which always performs a when it is available, but otherwise performs b. But ((a -> Stop) D (6 -». Stop)) \ {a} = Stop n (fc -» Stop). Here (a —» Stop) O (6 —> Stop) treats the events a and b on an equal basis. Both of the above equations are true in the semantics presented below. 5.7
External choice: (a —* Stop) n (b —> Stop)
A simple example of external choice is, (a —> Stop) D (& —* Stop). It is a process which is partly controlled by the environment as we can see in bi:: 62 ::
{} : X ~» {a, £} H X {) : X ~» {a} * a € X *• {fc} D X
X ~» {b} + b € X > {a} D X X -» {&} : X ~+
(2)
0 0.
EI •« boolean *• E2 is a notation borrowed from CSP itself: if the boolean is true the result is the expression E\, otherwise it is E^. Our process is nondeterministic because there are in general three possible responses to an initial offer of X. The last two lines in equation (2) represent the responses common tobl,b2 and b3. bl :: () : {a, b} ~-> {a} shows that the process may choose to perform the event a when given a choice between a and b. But suppose that we have a process which is compliant in the sense that it wishes to conform to the selection made by a parallel partner in its shared environment. It expresses that by responding with both a and b: bl :: {) : {a,b} -^ {a,b}. Thus D allows any of these possibilities: it abstracts from those details. Priority is a means of choosing just one class of these behaviours. 5.8
The pseudo even ts { /, X}
/ and A model termination and livelock respectively. We work with a global alphabet E U {/, X} where S is a set which includes all the 'ordinary' events that may arise. Later we will encounter the odd process Skip D div. It always accepts {/,/}. That is X —•» {/, X} for every offer X. {/,X} has the form of a compliant response, but since an environment can only offer real events, it cannot 'select' between / and X: there is no way to make Skip D div comply with a request to terminate because the environment has no way of making such a request. How then does Skip D div differ from Skip n div? The acceptance semantics certainly differ. The first has a single compliant behaviour; the second two deterministic behaviours. The nondeterminism in Skip F! div is between the two possible behaviours. We interpret any unresolved compliance also as a nondeterministic outcome. So Skip D div is deterministic in that there is only one behaviour, but nondeterministic in that no environment can choose between / and X in the response of that behaviour. So no single experiment can distinguish Skip D div and Skip n div.
28
5.9
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
Interleaving: (a -> Stop) \\\ (b -+ Stop)
The interleaving (a —> Stop) 1 1 1 (£ —»• Stop) has 3 behaviours characterised by bl::():
X ~»
bl : : ( ) : X
{a,b}DX
~» {a} 4 a € X »> {ft} DX
6 3 : : { ) : X ~*
{fc} <4 fe € X > {a} D X
X -» {a} n X X -~» 0 X -» 0 ,
where the responses after the first event for a particular trace are common. This is just (a -> fc -> Stop) n (b -» a -> Stop) , so the process is prepared to perform a and fe in either order. And when offered {a, b} it may choose a, b or be noncommittal. 5.10 Parallel: (a -» Stop) || (a -» fe -» Stop) {«} (a —» Stop) || (a —> fc —»• Stop) is a process that synchronises on the event a. The two {<>} component processes can only engage in a simultaneously so there is only one behaviour given by:
{) : X -» {a}nX (a) : X -» {fc} n X (ab) : X
-» 0 .
ft is an independent event, so one partner caa engage in it without the participation of its compatriot. So (a -» Stop) || (a -> ft -* Stop) = (a -* fc -» Stop) .
6 Alphabets and traces As above, there is an alphabet S of ordinary events: this will be large enough to include all the visible events of any process that we need to describe. To this we add the pseudo events / and X, writing S/x = S U {/,*}. Traces are sequences, empty or finite, of events drawn from E. () is the empty sequence. The set of all finite traces drawn from S is written as E* . The acceptance semantics here based on behaviours needs only finite traces. 7
Behaviours
We specify the meaning of a process by describing its responses after it has performed some trace of events. Traces are members of E* . Given such a trace, we then specify an acceptance function as {X ~» U}. The offer X is some subset of the alphabet E: that is X C E. And
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
29
a response U is a subset of I/*. So there is a partial function b : £* -+* (PE representing each possible behaviour. Thus we have a description which is a set BPof such behaviours: *PS / X ))
(3)
[[P J is the usual notation for the semantic function describing the behaviour of the process P, but BP is more intuitive and used here. The set of traces of the process is just the union of the traces of the behaviours: traces(P) = \J{traces(b) \ b e BP} , where traces(b) = dom b. We sometimes identify a process directly with its behaviours where the context warrants. And that leads to a simple matching normal form. 8
p-priority
CSPP extends CSP with extra operators including D and || . 8.1
Process Priority and external choice.
P = PI DP2 extends external choice and provides a semantics for PRI ALT in occam. The first event is selected in favour of PI. So Pa& = (a —>• Stop) D (b —> Stop), with a minimal alphabet S = {a, b} for simplicity, has the behaviour: <>: {):
(a): (b):
{«,&} - H {a} - {a}
X X
-» 0 -> 0
.
PI D PI is the symmetrical version. So the behaviour of P = (a —* Stop) D (b —> Stop]
is: {) : {}: (}••
{a,b} ~» {a, 6} {«} - {a} \b} - {*}
{} : (a) :
0 -» 0 X -» 0
(b):
X -0
5)
W
P! D P2 embodies the antithesis of p-priority in that it refuses to choose between a and b but rather treats them symmetrically. But it will always respect the p-priority of a parallel partner. 9 Fairness The presence of priority in CSPP provides a way to express degrees of fairness. Since infinite behaviour is also captured properly, this includes eventual fairness.
30
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
If A = up • a —»• p and B = up • b —» p, consider P\ = A \\\ B where there are no other processes involved so that both component processes are always ready. Then an implementation of PI can always favour A over B, that is A \ \ \ B I) PI , and no b is ever performed. <—»• Even PI = A \ \ \ B could also behave in just the same way. Although no behaviour of P2 favours a over b, but is neutral, the nondeterministic resolution of that neutrality, conceptually resolved by the environment in this case, might happen to be consistently unfair. Acceptance semantics based on behaviours allows us to define P0 — A \ \ \ B which is a process which cannot be consistently unfair. It guarantees that eventually there is an instance when a has priority, and likewise an instance when b has priority. To be precise, A \\\ B consists of those behaviours of A \\\ B which have at least one trace matching an acceptance giving priority to A and another giving priority to B. n
More precise control is given by A \\\B which ensures fairness in the sense above over every sequence of n events.
10 Abstracting from priority When two processes are combined, one or the other may have priority, or the result may be compliant. So if the first process accepts Xi and the other X2, the overall acceptance is determined by Xi if the first process has priority, by X2 when the second process has priority and by X\ U X2 in the compliant case. Each case corresponds to a particular variant of the joint process. But we also need a general version which abstracts from those details. We need it to model situations in which we lack full information, and for specification where we wish to leave the implementation choices open. This last is especially important in a programming language where we wish the compiler to generate the most efficient circuit or code. A more general and fairly extreme example is e : E —» Stop when £ is a large set. This is a process that may perform any event from E that may be offered. But if several events from E are offered simultaneously, we do not wish to constrain how the final choice of a single event is made. In a hardware implementation, the most efficient circuit is quite likely to be one with a definite priority hardwired. A biased implementation is acceptable. But a compliant implementation is just as acceptable, as is any intermediate sort that might be compliant with respect to {e1? e^} say, but biased in favour of {^3}.
11 Some definitions and notation The discussion so far has been introductory. There is no room in a conference paper for full technical detail. Here we lay out the foundations, in particular the axioms or 'health conditions' and simply give a taste of the full semantics by way of a small number of examples. Definition 11.1 When X C E, X' is used to denote X U { / } and X/x denotes X U { /. X}. #s is the length of s: #{) = 0 and #(r ~ (e)) = #t + 1. We extend this notation to behaviours: #6 = max{#.s | s € domfe} which is well defined when the lengths are bounded. Otherwise #b = oo.
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
31
12 Health Conditions (Axioms) A set of partial functions BP : P (£* -+> (PE -> P£ / A )) describing a process P, must satisfy the axioms or health conditions below. We usually write b(s) (X) as bsX following the usual conventions for curried functions. This is equivalent to writing b :: s : X ~» bsX. Each behaviour starts with a clean slate: HI:
\/b e BP •
{} € tmces(b)
traces(b) = domb above. The miraculous process has no behaviours: B T = 0. The traces of a behaviour are prefix closed, and extend while any event can be accepted:
s ~ ( x ) € t races (b) <£> s € traces (b) A ( 3 X C S « ; t e bsX) Notice that these closure conditions determine all behaviours when the acceptances are specified for a general trace. Every acceptance of an event is one of those offered. And there is a response to every offer because each b(s) is a total function: H3:
\/b£BP»Vs€traces(b)9VXCE9
bsX C X'*
If an offer can be refused, then so can any smaller offer. And if an event can be accepted, then no offer including that event can be refused. However the accepted event may differ from the original, perhaps because the second offer includes an event of higher priority:
H4>
VbeBP* Vs & traces(b) • VX, Y C E • bsX = 0 A Y C X =» bsY = 0 A bsX n Y ^ 0 =» bsY ^ 0
If an offer can be accepted, then smaller offers including accepted events can also be accepted: bsX n 7/x ^ 0 A Y C X
Combining conditions from H4 and H5 shows that all behaviours b & BP have acceptances which obey (Cl)
(C2) (C3)
bsX = 0
fex
A Y C X =» for = 0
fexny^0 => ^F^0 n y/x ^ 0 A F c x
when ^ is one of their traces and X and Y C S. These requirements for any individual behaviour & are collected in the following abbreviation.
32
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
Definition 12.1 behave (b) is an abbreviation for A {) € domfc
A Vs e dom(fe) • VX C E • bsX C X'* A
VseE* «VjceE« (x) € dom(6) <*• 5 € dom(fc) A (3X C E • x € foX) A
A r e x =^ => A K C X =*
foy
=0 foF^0
A A
13 Semantics There is only room here to give the precise semantics for some of the simpler cases and to highlight refinement and recursion. In particular, the definition of the parallel operators requires a little technical infrastructure which we omit. 13.1 Stop {} : X - 0
(6)
The only behaviour has a single empty trace and no events are accepted. This means BStop = {{{} -» {X - 0) | X C E}}} ,
(7)
so the only behaviour b has domain {{)} = traces(b) and b()X = 0 for every X C £. 13.2 Skip {):X~*{/} (8) Again there is only one behaviour, the only trace is empty, but the process always offers to terminate. BStip = {{()~ AX •{/}}} (9) 13.3 div
We define a simple div here. It is a process which immediately livelocks: 0 : X - {A}
There is a single behaviour {{} i-» AX • {X}}. 13.4 _L
J_ is the most unpredictable of processes: it includes every behaviour.
B_L = {fc | behave(fc)}
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
33
13.5 T
T is a miracle. And a fraud. It has no behaviours, and so is not implementable. But it refines every process. And ensures that every monotone recursion is well defined.
13.6
Prefix choice
Consider e : £ —> P(e] with E C E. In general there are many possible initial behaviours b:: {} : X ~ * 0 - « X n £ = 0 * U
where U C X n £ is not empty. So b() must satisfy b()X = 0 when X n £ = 0 and C X n £ otherwise; (£?:£-»/>()) =
behave(fc) b
A VX C E • b()X C X n E A (b()X = 0 =» X n £ = 0)
• (12)
A y £ £i JP" • 3 /? ^ f3 P\€] ^ V jS" €~ XI ^ \6?/
tS (E trCLC€,S\h) ~~^» fa( ( p )
SI ^^ /751
The behaviours represent all possible ways of assigning or refraining from assigning priority among the events off. They match one of those in BP(e) after the initial event. 13.7 Compliant Prefix choice The fully compliant refinement of prefix choice, e : E -+ P(e), is sometimes useful. An initial behaviour accepts anything from E: (} : X -^> XC\E. So B ( e : T->/>()) = behave(fc) A VX c E • b()X = xnE
}
> (13)
A
'e€E»3p£ BP(e] • Vs e E* • (e}^.y e tmces(b] => b((e)~s) = ps I
13.8
Non deterministic choice B(PlnP2} = BPluBP2
(14)
We extend the definition to sets of processes, writing B[r\{Pi}]=\J{BPi} \'-e/
/
/e/
(15)
34
A.E. Lawrence /Acceptances, Behaviours and Infinite Activity
13.9 Compliant external choice The process P = P\ D P2 abstracts away from implementation details. If given the choice, it may select between available initial events of PI and P2. If it always favours events from PI, that amounts to p-priority, but in general there are a large number of other possibilities. Among them is the case where the process abstains entirely from making a choice, and is completely symmetrical in its treatment of Pt and P2. It will always comply with the wishes of whatever mechanism makes the final decision:
behave(b) A
VX C S • b(}X = pl ()X U p2()X
(16)
A
W <= E • Vs <E S* • (e) 3X C E • « € j»i ()X A fc
traces(b)
V
13.10 p-prioritised external choice
behave(fe) A
3(Pl,P^eBPlxBP2» VX C E • b()X = piQX 4 pi ()X = 0 >Pl (}X A {e) € traces(b) • Vs € S* • (e) ~s € traces(b)
= 0A
(17) This simply says that the process always behaves like PI unless P! refuses in which case it behaves like P2. It allows PI to perform any event, terminate or diverge if it is capable of doing so. So if PI is active in any sense, it is let loose: ( ) : X ~ » l / if P! ::: () : X -» £/ apart from U = 0 P2 is only allowed to be active when PI is not. If PI shows any sign of life, even pathological life, it executes. 13. 1 1 External choice We define PI D P2 abstractly as (PiDP 2 ) n (Pl D P2) n (Pi DP2):
B(Pl np 2 )-5(P 1 DP 2 )uS(P 1 Hp 2 )u5(P 1 DP 2 )
(18)
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
35
13.12 Sequential Composition
behave(b) A 3(pi,p2) € BPl x BP2 • Vs e traces(b) • VX C E • 5 e traces(pi] A / 0 /?isX AfesX= p\sX V ,9 e traces^) A / € pi.vX A foX = ((/?i*X) \ {/}) U/> 2 V
, 5 2 ) € traces(p\) x traces(p-i) • 3 F C S • V / €pi5iK A s 2 / {) A5 2 (0) €piSiY f \ s = S!~s-2 /\ bsX = Notice that we cover processes like /j,p • (Skip H a -* p)%Q. And also that it is trivial to check that Skip ^ P — P 5 Skip = P. 13.13
Refinement
Refinement is as usual
PI 3 P-2 & PZ = P-2 n PI which simply maps onto set inclusion on the behaviours:
(20)
/*i3 PI & BPi ^&P2
(21)
13.13.1 TandJ_ The most nondeterministic process which has all possible behaviours is evidently below any other process in this order: it is the least element J_ of 3T has no behaviours, and is not iniplementable. Its contribution is to form the final brick in building a complete lattice. The structure then has the weaker property of being a complete partial order to which we can apply standard fixed point theorems to define recursion. 13.13.2 Meets and joins Obviously meets correspond to unions and joins to intersections of behaviours which always represent processes. Thus meet is just fl which is the motivation for the choice of symbol. For example, the meet of (a —> Stop} and (b —> Stop] is (a —* Stop) fl (b —>• Stop). Joins are written analogously as in (a —> Stop) U (b —>• Stop) = T. All meets and joins exist so the set of processes is a complete lattice under refinement. And so also a Complete Partial Order(CPO). 13.13.3
Unbounded nondeterminism
T handles cases of unbounded nondeterminism in a clean way. A standard example with an infinite E is: Example 13.1 Consider the set of processes pn = [~]{a, -»• Stop | i > n} for n €i N where all the a/ are distinct.
36
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
Here any finite set of the P, has an upper bound, namely Pm where m is the largest index. So we have a directed set. Yet for every n e N, an cannot be performed by members of {Pi | i > n}, so there can be no process behaviour which refines every element. But since T has no behaviours, it is the join: U{Pn \ n e N} = T. •[13.1] Another of the standard examples of an awkward directed set involves infinite behaviour even when S is finite. Example 13.2 Let S = {a, b} and P0 Pl P2 P3
= Hp^(b^ p) = a -> P0 = b^a^P0 = b^b^a^P0
(22) (23) (24) (25) (26)
so that Pn performs an a as the nth event in an otherwise unbroken stream ofb's. Write
Then D — {Dn \ n € N} is a directed set. A finite set of Dn is refined by a process that performs an a any time after the n'h b. It might be thought that a possible candidate for a process that refines every member of D is PQ. Yet is is clear that this is not true here for it consists of a single behaviour which is not present in any other Pn. Clearly the only candidate for an upper bound of D is given by intersection of the behaviours of members of D. That is obviously empty, so yields T. 13.14 Recursion fj,p • f(p) denotes a fixed point of the function/. This is often the least fixed point with respect to the refinement order in standard untimed CSP. Semantics based on Failures often have problems in this area: when unbounded non determinism or infinite traces are present, refinement is no longer a CPO. The Acceptance semantics based on behaviours presented here is the first denotational semantics that completely overcomes all those problems. It naturally includes infinite behaviour yet, as we have seen above, it is not only a CPO, but also a particularly simple sort of complete lattice under refinement. Standard theorems now ensure that every monotone function f has a least fixed point, and so up • f(p) is well defined. All the ordinary operators of CSPP are monotone with respect to the refinement order ~3: this follows from the fact that they are defined in terms of individual behaviours, so they all distribute over n. As noted earlier, a restriction metric can also be defined, and that is also complete. Contraction mappings then have unique fixed points: the corresponding recursions in CSP are known as constructive. The simplicity of establishing the uniqueness of the fixed points is often useful in proofs.
A.E. Lawrence / Acceptances, Behaviours and Infinite Activity
37
14 Conclusions Acceptances employing behaviours provides a simple satisfying powerful and intuitive denotational semantics for CSP and CSPP. It incorporates infinite behaviour naturally in contrast to other denotational theories for CSP which require awkward extensions to do the same. The standard operators distribute over fl which is related to refinement in the usual way. With the additional of a 'top' or 'miraculous' process, refinement yields a complete lattice. Thus all standard recursions are well defined and have fixed points. This intuitive result has been difficult to establish in other denotational semantics, but the present theory shows that the intuition was well founded. It extends CSP so that there is the option of refining to the level of detail required to describe priority, yet in an abstract way which does not require a fully timed theory. It forms a foundation for wider extensions of CSP needed for codesign, especially hardware compilation, but also for capturing a larger part of the semantics of occam-like languages than is traditional. These further developments will be reported elsewhere. 15 Acknowledgements Bill Roscoe noticed a deficiency in an early version of Acceptance semantics which did not include what are now called compliant processes. His prompting led to their formulation, but he is not to blame for the details. Support from Jeremy Martin when CSPP was just an experiment was crucial. The WoTUG community and discussions at CPA conferences have inspired many of the ideas underlying CSPP. References [1] C.A.R.Hoare. Communicating sequential processes. Communications of the ACM, 21:666—677, 1978. [2] Per Brinch Hansen, editor. The Origin of Concurrent Programming. Springer-Verlag, 2002. [3] C.A.R Hoare. Communicating Sequential Processes. Prentice Hall International, 1985. [41 A.W. Roscoe. The Theory and Practice of Concurrency. Prentice Hall, 1998. [5] Inmos Limited, occam 2 Reference Manual, 1988. Document 72 occ 45 01. [6] M.H. Goldsmith, A.W. Roscoe, and B.G.O. Scott. Denotational semantics for occam 2, part 1. Transputer Communications, 1:65-91, 1993. [7] M.H. Goldsmith, A.W. Roscoe, and B.G.O. Scott. Denotational semantics for occam 2, part 2. Transputer Communication-;, 2:25-67, 1994. [8] B.M.Cook and R.M.A.Peel. Occam on Field Programmable Gate Arrays - Steps towards the para-PC. In Barry Cook, editor, Proceedings ofWoTUG-22: Architectures, Languages and Techniques for Concurrent Systems, volume 57 of Concurrent Systems Engineering, pages 211-228, Amsterdam, April 1999. IOS Press. [9] R.M.A.Peel and B.M.Cook. Occam on Field Programmable Gate Arrays - Optimizing for Performance. In P.H.Welch and A.W.P.Bakkers, editors, Communicating Process Architectures, Proceedings of WoTUG 23, volume 58 of Concurrent Systems Engineering, pages 227-238. World occam and Transputer User Group (WoTUG), IOS Press, Netherlands, September 2000. [10] I Page and W Luk. Compiling occam into fpgas. In Will R Moore and Wayne Luk, editors, FPGAs. Abingdon EE&CS Books, 1991. [11] A.E.Lawrence. HARP (TRAMple) manual. Volume 1. User Manual for HARP1 and HARP2. Oxford University, 1992-95.
38
A.E. Lawrence I Acceptances, Behaviours and Infinite Activity
[12] Adrian Lawrence & Andrew Kay & Wayne Luk & Toshio Nomura & Ian Page. Using reconfigurable hardware to speed up product development and performance. In JFIT Conference. Oxford University, 1994. [13] A.E. Lawrence. Extending CSP. In P. H. Welch & A. P. Bakkers, editor. Proceedings ofWoTUG 21: Architectures, Languages and Patterns, volume 52 of Concurrent Systems Engineering, pages 111-131, Amsterdam, April 1998. WoTUG, IOS Press. [14] A. E. Lawrence. Hard and soft priority in CSP. In Barry M Cook, editor, Architectures, Languages and Techniques for Concurrent Systems., volume 57 of Concurrent Systems Engineering, pages 169-195, Amsterdam, Apr 1999. WoTUG, IOS Press. [15] A.E.Lawrence. CSPP and event priority. In Communicating Process Architectures - 2001, Concurrent Systems Engineering, pages 67-92, Amsterdam, Sept 2001. IOS Press. [16] A. E. Lawrence. Successes and Failures: Extending CSP. In Communicating Process Architectures 2001, Concurrent Systems Engineering, pages 49-65, Amsterdam, Sept 2001. IOS Press. [17] Oxford University Computing Laboratory. Two Papers on CSP, number PRG-67 in PRG Technical Monographs, July 1988. [18] A. E. Lawrence. HCSP, imperative state and true concurrency. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 39-55, Amsterdam, Sept 2002. IOS Press. [19] A.W. Roscoe. An alternative order for the failures model. In Two Papers on CSP [17]. [20] A.W. Roscoe. Unbounded nondeterminism in CSP. In Two Papers on CSP [17]. [21] C.J.Fidge. A formal definition of priority in CSP. ACM Transactions on Programming Languages and Systems, 15(4):681-705, September 1993. [22] Edmund M. Clarke, Jr., Orna Grumberg, and Doron A. Peled. Model Checking. MIT Press, 1999. [23] Formal Systems (Europe) Ltd, 3, Alfred Street, Oxford OX1 4EH UK. FDR2 User Manual May 2000. [24] Jeff Magee & Jeff Kramer. Concurrency: State Models & Java Programs. John Wiley, 1999. [25] Jeremy Malcolm Randolph Martin. The Design and Construction of Deadlock-Free Concurrent Systems. PhD thesis, University of Buckingham, 19%. [26] A.W. Roscoe, editor. A Classical Mind. Prentice Hall Series in Computer Science. Prentice Hall, 1994. Essays in Honour of C.A.R. Hoare. [27] Gavin Lowe. Prioritized and probabilistic models of Timed CSP. Technical Report PRG-TR-24-91, OUCL, 1991. [28] Gavin Lowe. Prioritized and probabilistic models of timed CSP. Theoretical Computer Science, 1994. Special Issue on Mathematical Foundations of Programming Semantics conference. [29] Carl A. Gunter. Semantics of Programming Languages. The MIT Press, 1992. [30] A.E.Lawrence. HCSP: Extending CSP for Codesign and Shared Memory. In P.H.Welch and A.P.Bakkers, editors, Proceedings ofWoTUG-21: Architectures, Languages and Patterns for Parallel and Distributed Applications, volume 52 of Concurrent Systems Engineering, pages 133-156, Amsterdam, April 1998. IOS Press. [31] A. E. Lawrence. Infinite traces, Acceptances and CSPP. In Communicating Process Architectures - 2001, Concurrent Systems Engineering, pages 93-102, Amsterdam, Sept 2001. IOS Press. [32] Andrew Butterfield and Jim Woodcock. Semantics of prialt in Handel-C. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 1-16, Amsterdam, Sept 2002. IOS Press.
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
39
HCSP: Imperative State and True Concurrency Adrian E. LAWRENCE Department of Computer Science, Loughborough University, Leicestershire, LEU 3TU UK A. E.Lawrence @ Iboro. ac. uk Abstract. HCSP is an extension of CSPP which captures the semantics of hardware compilation. Because it is a superset of CSPP, it can describe both hardware and software and so is useful for co-design. The extensions beyond CSPP include: true concurrency; new hardware constructors; and a simple and natural way to represent imperative state. Both CSPP and TiCSP were invented to cope with problems that arose while the author was trying to prove that the hardware that he had designed correctly implemented channels between a processor and an FPGA. Standard CSP did not capture priority, yet the circuits in the FPGA and the occam processes in the transputer both depended on priority for their correctness. The attempt to extend CSP rigorously to handle such problems of co-design has led to develoments that seem to have a much wider significance including a new way of unifying theories for imperative programming. This paper reports on the current state of TiCSP and focuses on handling imperative state and true concurrency. The acceptance denotational semantics is described briefly. Key words: CSP, CSPP, HCSP, true concurrency, denotational semantics, formal methods, hardware compilation, VLSI design, specification, parallel systems.
HCSP is an extension of CSP aimed at specifying and describing hardware: it is 'Hardware CSP'. There is a very short introduction to CSP in the companion paper [1]. One of the great merits of standard CSP is that it supports various levels of abstraction. One way is through hiding. And importantly there is an explicit refinement relation. If P 3 Q, then process Q is more abstract that P: the latter is more specific. So a CSP description can be relatively abstract, omitting detail and simplifying circuits to their essence. Or it can include considerable detail. Unfortunately, there are hard limits on how much detail. One such limit was encountered by the author when trying to prove that the interaction of a circuit generated by a hardware compilation system based on occam (Handel-AS) with a software occam process correctly implemented an array of CSP channels. An immediate problem was that the circuit depended for its correctness on priority: but CSP could not describe priority. So that was the first issue to be addressed. And it would be useful for software as well: priority in ordinary occam also had no formal semantics. A new denotational semantics was invented to define HCSP : acceptances. And the first extension CSPP defined, as described in the companion paper [1]: this differs from full HCSP in that it does not include true concurrency. When there is a limit to the detail which a theory can capture as with CSP and priority above, then one must change to some other theory to prove a design correct. In the case of circuits, one might consider VHDL or Verilog. However both of those hardware languages are complex, confusing and have not been designed around a proper mathematical theory. And they had no formal semantics when this investigation started. Even if proof was feasible there is still the need for a 'translation' layer to move between the high level rigorous description and the detailed circuit language. How much better if an extension of CSP could cope with fine circuit detail? Hardware is inherently parallel. CSP is the pre-eminent parallel language, occam had already been used to describe hardware, although not in a formal way. So an extension of CSP to capture circuit
40
A.E. Lawrence /'HCSP: Imperative State and True Concurrency.
detail was an obvious step. The ability to work at the highest level of abstraction is retained. Yet we can drop to full circuit detail and relate that to the higher level using refinement. This is the idea which has driven the development of JiCSP. occam was designed around CSP. TiCSP is intended to provide the same sort of foundation for hardware extensions of occam and for new rigorous hardware description languages. And just as CSP delivers simplicity, rigor and the ability to abstract away from all the inevitable complexities of a concrete language, its extension JiCSP can play the same role for a hardware designer. 1
Introduction
HCSP is an extension of CSPP which is in turn an extension of CSP. It aims to extend CSP to capture idioms useful for hardware compilation using an occam-like language. An exploratory version was introduced in [2]. The first step was to extend CSP to capture priority: that is achieved in CSPP, described in a companion paper [1]. The second step involved finding a way to capture the semantics of synchronous circuits where the state of many registers is updated simultaneously on a clock edge. Where such registers are independent that is modelled by standard CSP interleaving semantics, and occam usage rules ensure the independence. However, that is unnecessarily restrictive when true concurrency is present: occam-usage rules are there to eliminate undesirable non-determinism which arises only in interleaving. Thus:
is invalid in occam because an implementation would be entitled to perform the two communications sequentially in either order. So if AT is 3 before execution, and a value 4 is sent on channel d, the final value of x can be either of 3 or 4, and the value sent on channel c is also uncertain. Yet occam already included true concurrency in that it includes simultaneous assignments like 'x,y := y,x'. These assignments are not conventionally regarded as CSP events: they do not appear to involve communication or synchronisation with another process. Yet, as noted in [1], we can extend the idea of a CSP event to include assignment. Entertain processes that are 'always ready': this idea arises naturally in hardware compilation. Consider an environment which includes all variable names that can appear in a process together with their values. An assignment may need to read such values, and in general will modify a value: that is a communication. And this environment is 'always ready' and it engages in the communication which is also a synchronisation: the environment receives information that the assignment has occurred. Regardless of the underlying intuition, it is a valid interpretation of the CSPP and 11CSPsemantics to admit assignments as 'events'. The idea has been used informally by occam programmers reasoning about their programs, the real innovation here is to define the idea precisely using auras described below. The idea of an assignment as an event has a little in common with the notion of an 'action' in formalisms like the Temporal Logic of Actions [3]-
A.E. Lawrence /TiCSP : Imperative State and True Concurrency.
41
Every occam programmer is familiar with CHAN OF x.type c : PAR c ! e c ? x
The right hand side is a process composed of a pair engaging in a CSP event, although we may worry about how far that event is hidden outside the scope of the channel declaration. Thus the idea that the left hand side also represents (a process engaging in) an event is already implicit in the definition of occam. We will see that in HCSP the transformation above really amounts to attaching different labels to the same events: they may be called (x: = e), c. e, c! e, c?x or c. x. e, but these all represent a value determined by the expression e being passed into the variable x. In all cases there is a communication with an imperative aura1, an environment - it becomes an explicit parallel process below - maintaining a variable x and its value, if any. After the event, we have {x ^ v}, where v is the result of evaluating e. Classical CSP does not include explicit imperative state: it is normally taken to be a declarative language. One of the objectives of HCSP is to capture the semantics of occamlike languages easily and intuitively, so an imperative interpretation to stand along side the declarative understanding is very useful, and is essential when assignment is present. The use of one or more parallel processes, auras, to maintain the imperative state is of far wider significance, and provides a way of unifying theories of imperative languages in general, even purely sequential languages. The third step is to introduce new operators needed for hardware compilation. Many of these operators involve true concurrency so that synchronous circuits can be specified. This does not preclude the generation of asynchronous systems, perhaps mixed with synchronous components. Much of the power of the CSP family of languages comes from the high level of abstraction. But we also need to be able to specify fine detail in concrete languages including those designed for hardware compilation. It is sometimes asserted that CSP is not suitable for both tasks. The development of CSPP and HCSP shows that to be untrue. We maintain the highest level of abstraction: we can stay in the core CSP language if we so wish. But we also have refinements which can capture fine detail, and all within one rigorous framework. Thus HCSP is an extension of CSP which: • • • •
includes CSPP and so captures priority and is fully descriptive of infinite behaviour; extends CSP to handle true concurrency as well as interleaving; can support an imperative interpretation; and includes constructors useful in hardware compilation.
As with all variants of CSP, HCSP is intended for specification as well as concrete programming with the full rigour of a mathematical language. It first application is to define the formal semantics of codesign-oriented variants of occam. 2 Imperative and Declarative Views HCSP has two ways of treating variables: the usual declarative way and an imperative option. Consider (c?jt -> Skip) ? dlx -* Stop (I) 'Prototype auras were called halos in [21: the new name avoids any confusion with non-standard analysis.
42
A.E. Lawrence fHCSP: Imperative State and True Concurrency.
In the declarative view, x is not free: the input is just an external choice among the events | c | constituting the channel c. The only possible interpretation is D (c.x -» Skip | d.x -> Stop) = D (c.x ->
(2)
r-reW
which shows x to be bound by the quantifier D whose scope is the whole expression, x represents a value in a binding. In the declarative view, it is not even clear that the expression in equation (1) is well formed. ? would usually be taken as terminating the scope of the implied d constructor in clx. We return to this point below. In the imperative view of equation (1) x is the constant name of a variable, and the incoming value is stored in there. The input is still just an external choice among events in | c |, but those events all update a particular x. Notice that c?y —> d\y —> Stop involves a distinct set of events, those that update y rather than x. In the declarative view x and y are bound by an implied quantifier and so the two processes are the same. But there are two distinct processes when the free x ^ y in the imperative view. There is still an implied quantifier, but the external choice is over a restricted range of the events of the channel: our two imperative processes are each only prepared to engage in a subset of the possible events. The first allows only events of the sort clx €| c |, and the second only the distinct events cly €| c |. It is the latter understanding of equation (1) which is natural for an imperative programmer, especially for a hardware engineer who is mapping intuition about registers and memory arrays onto variable names. TiCSP is not intended to be esoteric, but accessible to the ordinary imperative programmer after the notation has been introduced. With P(x) = clx -» Skip and Q(x) = d\x -» Skip consider P(x) | Q(x). This represents quite different processes in the declarative and imperative views. In a declarative context, P(x) is normally interpreted as an external choice with a bound x. So the x in P(x] has no connection with the x in the body. But the x in Q(x) is the same x as that in its body. So the free variable x in the second component is the same as that in P(x) but they have no necessary connection with any input in the body of P. This is far from the natural interpretation. Equation (2) here is excluded by the implication that P(x) is a well defined process. In the imperative context, the P(x) is prepared to engage in any event of c that updates x. And the natural decomposition is permitted because x is free in both P(x) and Q(x). Now consider c!l —> Skip. In the imperative view this is a compliant external choice over all the events corresponding to the transmission of 1 over c. If the events of c are all of the type that update a variable from a set V, | c |= {c.n.v \ n € Z A v 6 V}, perhaps, then it has the form: ^ D c.l.v -» Skip c.\.ve\c\ Notice that this confirms that distinct events c.l.x and c.l.y occur in the imperative (c!l -» Skip) || (clx -» Skip)
and
(c!l -» Skip) \\ (cly -» Skip)
The distinction between the declarative and imperative views is only relevant when variables appear in expressions. If a is a constant event, a —> Stop has only one interpretation. The semantics of the constructors is the same in either interpretation. The imperative view is useful even when assignment is absent. Consider ((d?*! -* !?>>! -> Skip) H I (c2?:c2 -» J2?y2 -» Skip)) 5 out\(Xl + x2 + y, + y 2 )
(3)
In a declarative view, this is either malformed or if all the variables are free then either it is does not match the usual intuition or there are implied quantifiers and sorting out their
A.E. Lawrence / "HCSP : Imperative State and True Concurrency.
43
scope is tedious. The sequential constructor, is inside that scope. The process then does not decompose into two sequential processes. On the other hand the imperative interpretation matches a natural intuition: the particular values of xi,x<2,yi and yz communicated in the first component are passed across the sequential constructor to the second component. This is done by the use of the common free imperative variables and a parallel aura. Furthermore equation (3) represents a very common idiorn in occam programs: data is collected in parallel, and then some result subsequently computed. Yet this is decidedly awkward to express in the usual declarative version of CSP. The context of an TiCSP expression usually indicates whether it is necessary to distinguish between the imperative and declarative interpretations: that is whether variables are free or bound. Otherwise the phrases imperative TiCSP and declarative JiCSP can be used. 3
Events
Pure CSP does not require that an event involve exactly two processes. (a —> Skip || a —» Skip \ a —» Skip) = (a —> Skip \\ a —> Skip) = a —> Skip shows that it may not be possible to decide how many processes are associated with any particular event. In clasical CSP semantics where / is an event, albeit rather special, the process (a -> Skip || b -»• Skip) ? (c -> Stop)
involves 3 partners synchronising on /. Yet events are almost always regarded as involving precisely two processes in concrete situations at least conceptually and explicitly when channels are defined. This is a useful simplification apart from CSPP and JiCSP, not least in occam. Even in the case of SHARED channels, ordinary occam still deals with communication events involving precisely two processes. But when we model hardware in CSP we need to include 'events' which involve many processes. The most obvious case is a clock in a synchronous circuit: the clock edges are naturally regarded as events involving perhaps hundreds of processes. And if we regard assignment as an 'event', then it will involve only a single process in a concrete language. In all these concrete cases the aura represents the imperative state and is disregarded except when considering hiding and its relation to scope. T-LCSP puts that on a proper basis: assignments are events which update state. In that light, x, y : = y, x, which interchanges the values of x and y represents a pair of simultaneous events. It can be nothing else. It cannot be an interleaving, for execution in either order gives quite different results. Moreover, we need this idea of simultaneous events, or rather events 'tied' together, for other reasons when compiling hardware. That is particularly clear for synchronous hardware, especially when there may be several clock domains. So HCSP extends the conventional presentation of CSP in these respects: • events involving one or more processes are standard; • simultaneous events are introduced: true concurrency is present; and • atomic assignment is treated like other events. Assignment must be atomic as it is in occam and in many synchronous state circuits. Thus the change of state associated with an assignment event is uniquely determined by that event. This does not preclude hiding sets of events: that has the standard semantics. But when
44
A.E. Lawrence IJiCSP: Imperative State and True Concurrency.
modelling scope in a concrete language, a matching local aura is imported and also hidden. If the implementation of an event is extended in time and involves fine grained underlying actions, those actions must not be visible in any concurrently executing event: there must be no interference between overlapping event implementations. This is of course ensured by occam usage rules in the interleaving case. An assignment, or indeed any, event must behave as if it had an instantaneous implementation. 4 HCSP is compositional liCSP and CSPP are compositional: the properties of a process or circuit follow from those of its components. There can be no interference from the surroundings or other processes. That means that any interaction must be explicit as shared events. This allows understanding, proof of correctness and simplicity. JiCSP and CSPP also support separation of concerns partly through compositionality. One should not have to worry about too many aspects of a design at once. tiCSP supports and encourages the style in which one builds simple understandable units, transparently correct, preferably proved correct, and then assembles them into larger units which again can be seen, and hopefully proven, to be correct. Once the properties of the smaller units are established, they must not be undermined by assembling them into a larger system. One style of hardware compilation makes extensive use of simultaneous assignments as in:
Here SYNC is a constructor for true concurrency: it is a synchronous, truly concurrent, version of occam PAR. The left hand side may not appear to be compositional at first sight. However, remember that assignments here are events shared with an aura recording the value of variables. So the behaviour is determined by the interactions with parallel partners, and the system is compositional. Which is more obvious in the second and third ways of writing the process above. Abstraction is also a fundamental tool in compositional design: the minimum level of detail or specification should be used. This is very economical: the same process might be implemented in an asynchronous self-timed circuit, or in synchronous hardware with one or more clocks. But probably more important is that a human has less to worry about at any given point. The prospect of getting a system correct is much greater if one can dismiss detail irrelevant at a particular stage. Identifying what is relevant and handling the detail properly and rigorously when it becomes necessary, but not before, is central to compositional design with abstraction. These are the characteristics of co-design using JiCSP, which are largely missing from other methods not based on CSP. As a small aside, it is often required that real-time safety-critical systems be deterministic. While that may be a necessary requirement for systems designed with inadequate tools, it is misconceived. What is required is that a system must be proved to meet the timing constraints. Any further specification is an artificial and possibly dangerous additional constraint. It is normally far better and safer to have a nondeterministic system, if only so that it has the flexibility to work around partial system failure, work efficiently, and relieve the system designers of the burden of meeting a more complicated system specification. Of course,
A.E. Lawrence / 7-iCSP: Imperative State and True Concurrency.
45
languages that deal explicitly with time, like those based on HCSP, make these tasks much easier. 4.1
Examples of compositional design introducing new constructors.
To illustrate compositionality, suppose that we have a process which is to be compiled into a circuit: it could be an adder, for example. The process might be as simple as
Think of cl and c2 as two input buses to a circuit that places the result on out, the output bus. That can be written in imperative HCSP as (cl?* -> Skip) \\\ (c2?y -» Skip) § out\(x + y) -> Skip HCSP includes an extended form of 'interleaving', jjj, which allows two disjoint events to occur simultaneously. So jjj includes true concurrency as well as interleaving. Using interleaving as in standard CSP or CSPP in (cl?* -» Skip) ||| (c21y -> Skip) °9 out\(x+y) -+ Skip would only permit one input to occur at one time. But in synchronous circuits, such inputs will often be ready on the same clock edge, and be executed together. When clocks are explicit, we find that we require true concurrency to capture the behaviour of the circuit. In HCSP is extended to permit the possibility of two events happening together coincidentally. Example 4.1 fa -f Stop) HI (b -»• Stop) = (a o b -+ Stop) D (a -> b -> Stop) D (b -» a where aob represents the events a and b happening together. Although we still call the symbol ||| "interleaving", as example (4.1) shows, there is now the possibility of coincidental execution. In a context in which true concurrency is possible, it would be unnatural to exclude it. Of course one can still generate a process like (a -»• b -» Stop) D (6
Stop)
which is (a -» Stop) ||| (6 -> Stop) in CSP and GSP77. This could also be generated from the HCSP (a —*• Stop) \\\ (b —> Stop) by removing a o b from the alphabet, or by using a parallel partner imposing such a restriction. JH is like 11, but captures the frequent case where we desire efficiency: Example 4.2 (a -» Skip) |H (6 -> 5*i» = (a o £ -» Skip) D ((a -»• fc
D
46
A.E. Lawrence fHCSP: Imperative State and True Concurrency.
Example 4.2 matches the earlier process:
The programmer will expect a circuit that will perform the inputs on cl and c2 simultaneously whenever possible. Notice how priority arises naturally in capturing the semantics of circuits like this. We can require that the two inputs be simultaneous by writing: SEQ SYNC
— O
cl ? x c2 ? y out ! x + y
which is: (cl?jc -> Skip) O (c2?y -» Skip) ? out\(x + y) -> Skip in imperative HCSP. And if further it is to be clocked by Clock: TIE Clock — TIE is <j SEQ SYNC — The O constructor cl ? x c2 ? y out ! x + y
which represents the compositional TiCSP process: ((cl?* -+ Skip) O (c2?y -» Skip) ; out\(x + y) -» Skip) <j Clock Any events of the adder now happen only when the Clock process generates events. Presumably these will be standard clock edges determined by Clock. But the process above can only handle simultaneous inputs. Contrast: TIE Clock SEQ PAR cl ? x c2 ? y out ! x + y
> Skip) HI (c2?y -+ Skip) ; out\(x + y) -» Skip) <| Clock which will also complete a communication on a single input if it is ready on a clock edge. A clocked circuit cannot usually inhibit its controlling clock. The circuit may be inactive on some clock edges, while it awaits data from a microprocessor, perhaps. So while the circuit can only perform actions synchronously with the clock, the clock can continue ticking independently.
A.E. Lawrence /TiCSP : Imperative State and True Concurrency.
We have seen that O binds the events of processes even more closely. It should not be confused with the sync event used in the Kroc compiler. Notice that now we can write lx, y := y, x' as: SYNC x := y y := x
or: > Skip) O (y := x -> Skip) = (x := y) o (y := x) -»• Skip In passing we have introduced o, O and <| as well as \\\. o and O are closely related. o is used to 'glue' events together to form a compound event, and is the primary way that we capture true concurrency. This is placed on a proper basis later. O is a constructor that 'glues' processes together concurrently. <j is particularly useful in synchronous hardware and makes sure that the events of one processes can only occur in synchrony with those of another, typically a clock. The symbol for this 'tie' operation has been changed from that used in [2]: the new version suggests the way clock signals are usually drawn on schematics. It is the task of JiCSP to define precisely such entities. 5
Infinite behaviour
When is || useful? We have seen that f|j is usually preferred to specify an efficient circuit which eagerly does as much as possible in parallel. One potential application for [ | is in specifying clocks. Suppose that a system consists of two synchronous components driven by their own clocks C\ = p. C • ti —> C and C2 = p. C • t-2 —» C. These might be two chips each with their own local crystal oscillators: the clocks are unlikely to be correlated. How do we describe the overall trace, say of just the clock components? An extraordinarily cumbersome way would be to have an explicit composite clock like: t] -> t\ -^ti-+t2-*ti -»• f] -» ti -H. t'2 -+ fi -» ti O t-2 -»• . . .
which includes the very unlikely case of clock edges exactly coinciding. Then C\ and €2 could be extracted by hiding. This is so ugly and inelegant that it can be dismissed immediately for any normal use. 4— C\ j 11 C2 is close to what we need, but it includes too much: a refinement is C\ \ \ C2 = C\ which does not allow €2 to run at all when Ci is ready. C\ 111 C2 is almost right, but again one or other of the clocks could always be excluded: the non-determinism in selecting a component process at each step could always be resolved in favour of Ci, for example. Fortunately CSPP and HCSP include a form of interleaving which eventually admits both partners. This is a virtue of acceptance semantics based on behaviours, and permits the description and prescription of varying degrees of fairness. Here we want C\\\ €2- The behaviours of Ci j 1 1 C2 are those of C\\\\ C2, except that only behaviours which give priority to both C\ and to C2 at some stage in their evolution are admitted. Perhaps C\ is favoured on {), the acceptance is compliant for all traces of length between 1 and 10, and C2 has priority for any trace of length 11. n
More precise control can be given using the construction PI j 11 P2 which guarantees that any segment of any trace of length n is "fair": that is P} and P2 are given priority at least once
48
A.E. Lawrence /"HCSP: Imperative State and True Concurrency.
in such a segment. Clearly this only makes sense when n > 1: PI ||| P2 = T. So d \\\ C2 would alternate ticks from each clock if they were both always ready. These clock combinations are primarily of use in proving the correctness of a circuit: they permit an abstract modelling of the clocks. They would not normally be used in the compilation of crystal oscillator clocks. 6
Bags
This is a short digression to discuss bags or multisets. A bag is a set which may include repeated elements. At face value that statement is nonsense: it is inherent in the idea of a set that it can contain at most one instance of any particular element. A bag is really an integer valued function on a set: the bag containing two copies of an element a is identified with the partial function {a •-* 2}. We have to face the fraught topic of notation. One standard notation for the bag above is [{a, a|, but this overloads the symbols [[ and J. And it carries connotations of lists rather than sets. There is an alternative notation sometimes used in Z, but it is awkward to write neatly freehand: typeset symbols which are easily and quickly approximated by hand are preferred. Here we choose to write {]#, a|} with just the right suggestion of a set. {]a,a[} is a shorthand for \a »-+ 2[} where strictly speaking the bag brackets should be replaced by set brackets in the second case if we wished to be unhelpful. If a maps to 3, and b to 2, that can be written in many ways including fla.a, a, b, b§, §a,b,a. b. a(} and fla^3,fc.-»2[}. We write x € 0 to indicate that x is a member of the bag 0, although we sometimes overload € and just write x G 3 where the context warrants. Both mean x € dom 3: the codomain of a bag is NI . Similarly bag intersection and union are written as the similarly decorated symbols ffl and W. We write (3 5> 0', 0 ±> 3', 0 <± 0' and 3 ff & when 3 properly contains 3', contains 3' or is a proper sub bag of 3' or is any sub bag of 3'. We can write the empty bag as 0 without ambiguity but we may also write {] [}. 7
Merged events
If e\ and e^ are events, we write e\ o e Stop) O (b -* Stop) = a o b —> Stop In:
(a -H. Stop) ||t (b -> Stop)
(4)
a and b can happen together as a o b. As usual, the environment must offer a o b if the simultaneous pair is to occur. A compliant offer of {a, b, a o b}, for example, is an offer to perform one of three mutually exclusive alternatives: either of the lone events a and b, or both together as a o b. The response to the offer above will be {a o b}. The same event may happen concurrently in a synchronous circuit where there are multiple copies of a particular server. If there happens to be a demand for this service from several independent processes on a particular clock edge, then the servers may all engage in multiple copies of the same event simultaneously, for example a o a o a. Yet these events are not inherently tied together for they can happen independently. Nor do they represent synchronisation between the various servers: only with the clients using the servers. This is the natural extension of interleaving in conventional CSP and is modelled with ffj. In this version of the semantics, no distinction is drawn between events coinciding 'accidentally' in
A.E. Lawrence/HCSP : Imperative State and True Concurrency.
49
such ways, and those intentionally tied together: the aim is simplicity with a minimal set of primitives. In this light, we identify coincident events like a o b or a o a with bags of events which happen, accidentally or inherently, at the same instant. So for the process in equation (4) when a and b are distinct, acceptances include
61 61 62 63 61 61 61
62 63 ::{}: {a, 6, a 06} ::{}: >,*} ::{}: {« ,*>} ::{): {*,*} 62 63 ::{}: {«} 62 63 : : < > : {aob} {aob} ~> 62 63 :: {} :
m
(a o b} {a} {b} {a, 6} {«} {6} {«o6}
(5)
and so on. The interpretation of the first line: 61 62 63 :: () : {a,b,aob} -> {aob} is that a compliant environment offers to perform either the event a alone, or the event 6 alone, or both a and 6 simultaneously. All of the behaviours 61,62 and 63 respond by accepting the offer of a o 6. Notice that O can generate environments which offer only joint events, so an offer of {a o 6} in the last line makes sense: the environment may be prepared to do events jointly which it refuses to allow alone. This is an easy and minor modification of Acceptance semantics as described in [1]. The acceptances for a behaviour have the type PE —* PE^*, but now E consists of bags representing coincident events. It is not even necessary to modify the concept of traces in HCSP, although they have an underlying structure of sequences of bags of bare events. As usual, all processes have a trace {) before they have performed any events. But now a process like a -> (6 -»• c -»• Stop] HI (6 -> (c -* Stop \\\c-+ Stop)) has {{]a|}, "P> ^|}> ilc) c > cl}) which we normally write as (a, bob, cococ) among its traces. We write singleton bags as the corresponding event in the usual notation. So (|a|}) is generally abbreviated as (a). In JiCSP all events are really bags: singleton bags can be identified with the atomic events Es. Every event is a member of bag Es, the set of all nonempty bags formed from members of E$. If a, 6 € £5 are atomic events, then a is identified with {]«[}•, 6 with {|6[} and aob with {ja, 6[}. So o itself is really just bag union, W. It may sometimes be useful if E, the alphabet of events, is finite. If we required that £ be closed under bag formation, that would not be possible. So in general, the alphabet E is a collection of bags including all those that arise in modelling the systems of interest. Then £ can often be finite. More formally, E C bag £$, where bag £$ = ES -+•>• NI is the set of all bags containing atomic events from £5. Since acceptance semantics based on behaviours has no problems with infinite behaviour, E = bag ES is also acceptable. And as we have seen, o is an alias for the bag union operation tti. That is (b\ o b%)(e) = b'i(e) + b'2(e) for e € dom6i U dom62 and where b't — 6,; U {e >-> 0 | e £ dom6,}. What about the pseudo-events / and X? Since (a —> Skip) % (b —»• Stop) = a —»• b —* Stop we see that / cannot be tied to a clock. That also follows because / can never appear in a trace. If an implementation does take one or more clock cycles to terminate a process in hardware, and that process is tied to a clock, then an explicit 'termination' event can be added to the model if necessary to reflect the extra clock cycles. But most
50
A.E. Lawrence fHCSP: Imperative State and True Concurrency.
synchronous implementations will not require the extra cycles: in that sense, termination is free - it requires no additional cycles. Skip plays a special role in the termination of parallel processes: it acts as a sort of rendezvous. All partners must jointly offer / in order for the whole process to finish. The natural extension of this is to write: (a -»• Skip) O (fc -» Skip) = aob-+ Skip So might we expect that / O / = / if o can be applied to such terms. And: (a -> c -» Skip) O(b-* Skip) = a o b -> Stop because the processes cannot agree after the first action. Thus / cannot be merged with any other action. We must exclude a o / either by removing such pairs from dom o or by setting a o / = _L<>. The simplest approach is to exclude both / and t from the domain of o. A related example is: (a -+ Skip) If} (b -» c -» Skip) If environment offers aob initially, and then c, it seems clear that the natural meaning is that the event c can happen despite the fact that the first process offers no event or has terminated. Our semantics must permit the trace (a o b, c) in such cases. For the elementary processes: Skip O Skip = Skip
,
Skip O Stop = Stop
and Skip O div = div
Notice that: (a —> Stop) O (b —> c —* Stop) = a o b —> Stop because it is enough for just one of the processes to stop to prevent any further progress for . A models a process out of control, primarily one in an infinite internal loop. Consider P <| div. Since the clock never in fact emits any pulses, this is a process which can make no progress. And indeed, the overall process is itself locked into an internal loop, so clearly P <j div = div. Hence tot makes no sense in this context, div <j C — div follows for much the same reason, although now it is the tied process which does not respond to external clock pulses. Since the clock might continue, one might argue for a less extreme result: div <| C = div in effect 'stops' the clock which may well be driving other good circuits. But since one part of the overall circuit is broken, it is reasonable to regard the whole circuit as defective. Another option might be to write div <j C = _L where _L is the process which can do anything at all including diverge. For simplicity, we treat X like / with o, and do not include it in dom o. So o only produces bags of real events from S. 8 Acceptances An acceptance is a set of alternative actions of a process in a particular environment. Each 'action' consists of a bag of events. For example the compliant process: (a -> Stop) D (fc ^ Stop) accepts {a, b} = {fla[}, fl^fr} when both a and b are possible single events: at least when the environment is compliantly offering to perform either a or b but not both simultaneously. More exactly: (a -» Stop)S(b -» Stop) ::: {> : {a,b} ~> {a.b}
A.E. Lawrence / TiCSP : Imperative State and True Concurrency.
51
This indicates that the process is also willing to perform either a or b but not both simultaneously. Thereafter: (a) : X -* 0 (b) : X ~» 0
This matches standard interleaving CSPP semantics. They ensure that there is no possibility of simultaneous execution. And this is maintained in HCSP with true concurrency: a and b •5- Stop) will not perform a o b: it is only able to perform a or b, even if it delegates that choice to a partner: (a -» Stop)*S(b -> Stop) ::: {) : {a, ft, a oft} -» {a, 6} In contrast: (a — Stop) \\\ (b -> Stop) will respond with {a o ft} when it is available: (a -» Stop) ft} (ft -> 5toP) ::: () : {fl, M o fc) ~» {a o fc} so it will perform a and & simultaneously if the environment is willing to allow this. So one possible trace is (aob). For (a —> Stop) O (ft —•> Stop) we have:
with traces {{), (a o ft}}. A more interesting example is: (a -> Stop) O (ft -» Sfap) This engages in a oft, but then the second component attempts to synchronously terminate by executing Skip. But the first component stops, so the overall effect is Stop. Indeed (a —» a —»• Skip) O (ft —> Skip) also deadlocks after the first event for the same reason: one component tries to engage in a but the second wishes to terminate in a common action. Successful termination requires that both partners jointly engage in /. 9
Behaviours and Traces
Traces are sequences of bags drawn from bag S$ which is the same as sequences of elements drawn from E C bag S5. So this is similar to CSPP traces drawn from an alphabet E. The acceptances of a behaviour is a function recording the response to an offer of a set X e P E which is itself a set Y e P (S /lX ), written X ~+ Y. An individual behaviour ft is a partial function from traces to the acceptances: ft : E* -«• P S ->• P (S'**)
with tmces(b) = dom ft .
The behaviour of a process P is a set of behaviours: BP:P(£* -»PE-»P(E/'A)) with: traces(P) = |J{domft ft e 5 (P)}
(6)
52
A.E. Lawrence fHCSP : Imperative State and True Concurrency.
10 Auras and imperative state As we have noted, HCSP can also treat assignments as events. The target of an assignment is an imperative variable, so we introduce a set of Names and a set of Values. An aura is a process which keeps a record of the value of variables, that is the state of an aura is given by a partial function M : Names -++ Values. Consider (x :— 6) —> Skip. This is not an aura, but when run in parallel with one as in:
then the state of the aura A is {jn-> 6} © M after (x := 6) —> Skip has terminated. Thus an aura is a process that is prepared to engage in assignment events, and it keeps track of imperative state. It runs in parallel with the rest of a system and synchronises on all assignment events. Such events are communications with an aura. (x := x + y) —> Skip is a little more interesting. A natural implementation would involve inputting from an aura to discover the values of* and y, and then outputting the new value of x. Here we abstract all of that into a single atomic event which involves communication in both directions. Although that abstraction is a very good approximation to what happens in many synchronous circuits, that is by no means necessary. The abstraction is still valid even when implementations may require several communications over an extended period. If we run (x := x + y) —>• Skip in parallel with an aura A ({x i—» 4, y »-> 5}), then after the assignment, we have A ({x >—> 9,y *-> 5}). If we examine (x := x + y) —» Skip more closely, we realise that (x := x + y) really represents a compliant external choice, that is compliant prefixing, of events. So one offer is [x := x + y]6, that is the event involving collection of the values of x and y, and then sending the new value of 6. That would be refused by the aura above which will only accept Definition 10.1 [N :— e]v represents an assignment event which involves setting all the names in the set N C Names to the value v e Values. An aura will only engage in such an event if 1 . any variables that appear in e are in the domain of M, the current state of the aura; and 2. the expression e evaluates to v. Otherwise it refuses, so errors are signalled by a process stopping in the usual way. The definition (10.1) uses a set N to permit assignments like {bo, ^1,^2, £3} := 3 which may be useful in hardware when one driver is connected to several loads possibly on a bus. Usually N is a singleton, and then we usually write x := e radier than {x} := e. Now we can write (x:=x + y) -^ Skip =
D
[x := x + y]v -» Skip
vdValues
= e : {[x := x + y]v \ v € Values} -* Skip If M : Names -+* Values represents an imperative state, then we extend the notation M(exp) to all valid expressions exp. So among valid expressions are the names of the variables in dom M. There is no need to be prescriptive about which expressions are to be admitted: TiCSP is intended to have a wide range of applicability. When used to model Occam, the expressions would be those of that language. Nor do we specify here whether expressions
A.E. Lawrence / 1-iCSP: Imperative State and True Concurrency.
53
like x 4- x and 2x are to be identified. A strict reading of the definitions 10.3 and 10.4 below however would exclude [x := y - y]0 as a valid event when y & dom M. If an expression exp in [N := exp]v is not well formed, then the aura will refuse the event, so the overall effect will be Stop. This is a neat way to handle error in the spirit of occam. and it requires no new machinery. A similar issue arises with mutually contradictory assignments like (x := 1) o (x := 2). Again, an aura refuses such pathologies which might represent bus contention or worse in a circuit. Clearly we require that Names and Values be large enough to include all names and variables that appear in the alphabet of atomic events, Es. The set of all events which reference one of the variable names is E^/-: Definition 10.2 EA/- C E is the set of events which which contain at least one atomic assignment: S,v = [e £ E 3N C Names • 3 v C Values • 3 exp • [N := exp]v € e] Definition 10.3 The atomic assignment [N := exp]v is valid with respect to state M : Names •+» Values when all the free variables which appear in exp are in the domain of M, and the expression exp evaluates to v. Definition 10.4 An event e € bag £5 is valid with respect to the state M precisely when every atomic assignment in e is valid and all the assigned variables are distinct. Thus \p '.= 3]3 o \p := 4)4 is invalid. Definition 10.5 A(M) C E is the set of all valid events which contain at least one atomic assignment. With these definitions, it is easy to define an aura: Definition 10.6 An aura A is a function A : (Names -+-» Values) —» HCSP with
A(M) = where:
t ..
cf e -> A(M')
\ 0M
In this section, so far only assignments have been considered. What of the imperative: (c!42 -» Skip) || (clx -> Skip)
?
This too is run in parallel with an aura. Assume for now that the aura accepts the events clx and c!42. These are then events that involve three participants: the two processes above and an aura A(M). But, of course, clx and c!42 are just descriptions of the same event. That event updates the variable x with 42. In fact it is precisely [x := 42]42! Thus we see that we have already captured imperative channel communication. And we can write ((c!42 -+ Skip) | (clx -> Skip)) || A(M) = ((x := 42) -> Skip) \\ A(M)
(7)
which shows how the occam law identifying channel communication and assignment appears rigorously in TiCSP. We can just identify the channel events with assignment events.
54
A.E. Lawrence/'HCSP: Imperative State and True Concurrency.
Thus auras capture all aspects of imperative state in a very simple way without requiring any new concepts. They are also very flexible allowing multiple auras which might be useful in modelling distributed systems. And often an aura will have an initial empty state thereby STOPping an any attempt to read an uninitialised variable. Auras are run in parallel with imperative processes synchronising on the set E^: that is natural and intuitive. But it can be tedious to remember to specify the synchronisation set. An a-aura is an extension of an aura which synchronises vacuously also on the non-imperative events S - Ejy: Definition 10.7 An a-aura A is a process of the form A(M) = (e : A(M) -> A(M')} H (e : (E - EA-) -» A(M)\ where: M'= I
|J
\[X:
and M : Names -+» Values. Example 10.1 ((X • = y) _> Skip) || 4(0) = Stop
where A is an a-aura. The initial empty state has no value for y, so the assignment is invalid.
•[ 10.1 ]
Example 10.2 (((v := 1) -» Skip) 5 (jc := y) -» Skip) \\ A(0) = (\y := 1], -» [ x - . ^ y ] , -> (skip \\ A({x ~ l,y -> 1})) vv/tere A w an a-aura. •[10.2] Although HCSP avoids the full complexity of a complete concrete language, it is intended both as an aid in the design of such languages as well as capturing a substantial part of the semantics of existing occam-like languages. An approach to capturing imperative state based on auras seems to hold promise for simplifying the semantics presented in [4] and [5]. 11 Health Conditions In order for a function B P : P (E* -++ (P E —> P E /;f )) to be admissible as a description of a process, it must conform to some simple constraints. These "health conditions" are the same as for CSPP applied to an alphabet E C bag E$ of bags of atomic events. Thus TiCSP inherits the semantics of CSPP described in [1] in a remarkably simple way. This is a result of maintaining traces of events, but identifying those events with bags. It captures the semantics of true concurrency especially as encountered in synchronous circuits in a very satisfying way, maintaining abstraction from the full details of time.
A.E. Lawrence / HCSP : Imperative State and True Concurrency.
55
12 Summary and conclusions This paper has given a brief survey of the way in which "HCSP has evolved since its introduction in [2]. In particular, it shows how acceptance semantics based on behaviours is simply adapted to capture true concurrency by little more than extending events to include bags. The acceptance base allows both priority and infinite behaviour to be properly described. It has been shown how these arise naturally and inevitably in hardware compilation. A language lacking these features will be inadequate for specification and compilation. A novel and simple way to extend CSP to model imperative state at just the right level of abstraction has been described. This is needed since both occam and most hardware compilation languages are imperative and match a natural intuition about hardware. The treatment has been very substantially simplified since the original version of JiCSP, and at the same time made far more flexible and powerful. For the first time it has captured a rigorous model for the occam equivalence between communication and distributed assignment. Several of the constructors of HGSP have been introduced: full definitions have been omitted for lack of space. References [1] A. E. Lawrence. Acceptances, Behaviours and infinite activity in CSPP. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 17-38, Amsterdam, Sept 2002. IOS Press. [2] A.E.Lawrence. HCSP: Extending CSP for codesign and shared memory. In Proceedings of WoTUG 21: Architectures, Languages and Patterns, pages 133-156. WoTUG, 1998. [3] Leslie Lamport. The temporal logic of actions. ACM Transactions on Programming Languages and Systems, 16(3):872-923, May 1994. [4] M.H. Goldsmith, A.W. Roscoe, and B.G.O. Scott. Denotational semantics for occam 2, part 1. Transputer Communications, 1:65-91, 1993. [5] M.H. Goldsmith, A.W. Roscoe, and B.G.O. Scott. Denotational semantics for occam 2, part 2. Transputer Communications, 2:25-67, 1994. [6] Peter T. Breuer, Carlos Delgado Kloss, Andres Marin Lopez, and Natividad Martinez Madrid. A refinement calculus for the synthesis of verified hardware descriptions in VHDL. ACM Transactions on Programming Languages and Systems, 19(4): 586-616, July 1997. [7] A. E. Lawrence. Infinite traces, Acceptances and CSPP. In Communicating Process Architectures - 2007, Concurrent Systems Engineering, pages 93-102, Amsterdam, Sept 2001. IOS Press. [8] A.E. Lawrence. WoTUG.
Extending CSP - even further.
Communicating Process Architectures-2000, 2000.
[9] Andrew Butterfield and Jim Woodcock. Semantics of prialt in Handel-C. In Communicating Process Architectures - 2002, Concurrent Systems Engineering, pages 1-16, Amsterdam, Sept 2002. IOS Press.
56
This page intentionally left blank
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
57
James PASCOEf and Roger LOADERf }The University of Reading, Reading, UK. J . S . [email protected] .ac.uk Abstract The Agreement Problem Protocol Verification Environment ( APPROVE) has become a mature and robust platform for the automated verification of proposed solutions to agreement problems. APPROVE is based on the Spin model checker and leverages the literate programming tool noweb to provide Promela code supplied with MlgX documentation. The APPROVE discussion opened in Communicating Process Architectures 2001 and described the initial project phases and summarised some preliminary results. This paper presents a follow up, providing a canonical report on the development, application and insight gained from the project.
1
Introduction
Agreement problems are characterised by the need for a group of processes to 'agree' on a particular value and examples include: consensus, group membership and leader election schemes. As such, agreement algorithms are intrinsic to strong group communication systems. Strong group communication, differs from its semantically weaker counterparts by including the notion of membership, that is, each strong peer maintains a representation of the groups association termed a view. In [1], Chandra et al. proved that several important agreement problems, including consensus, are not solvable in asynchronous systems subject to even a single crash failure. As consensus underlies all agreement problems, the implications of this result are wide ranging. However, it should be noted that this impossibility result should be interpreted as 'not always possible' instead of 'never possible', and providing that the correctness of an agreement algorithm be determined rigorously, the proposed solution can circumvent this result. A range of techniques, at varying degrees of rigor, have been applied to accomplish this task. Birman adopted Temporal Logic to reason about the correctness of virtually synchronous group membership [2]. Lamport applied numerous formalisms in the study of consensus [3] as did Hadzilacos, Chandra and Toueg in their research on failure detectors [1]. Current work suggests that a verification technique termed Rigorous Argument has become a de facto standard and in a previous application, has successfully proved the correctness of the Collaborative Group Membership (CGM) algorithm [4]. During its application, it was observed that there exists an opportunity to extend the methodology to provide additional benefits. The Agreement Problem Protocol Verification Environment ( APPROVE) is a highly configurable automated verification environment that quickly and systematically verifies the correctness of a proposed solution to an agreement problem. Through the comparison of other proofs, it was observed that there exist central themes which are modeled repeatedly, albeit in different formalisms and so additional motivation was to exploit this potential for reuse. Thus, the APPROVE design philosophy is to provide a configurable, extensible, automated verification environment through which a catalogue of previously verified reusable components can be quickly composed to suit an application. In doing so, the researcher need only
58
J.
S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
model the algorithm or protocol under test and invoke the automated verifier to establish its correctness. The aim is to not only reduce the amount of effort and error associated with developing such proofs, but to instill a much higher degree of confidence in the process and demonstrate the effectiveness of formal tools in practical settings. This paper presents the development of APPROVE and its application to a suitable exemplar, which in this case is a revisitation of CGM. The next section presents background which includes an introduction and worked example of the techniques used to implement APPROVE. In section 4, the architecture of APPROVE is described and the design abstractions are justified. Section 8 discusses the application of APPROVE to CGM before section 10 concludes the paper. 2
Background
A number of formal techniques that reason about the development of strong group communication systems exist. Possibly the most notable is the application of the NuPrl theorem prover ('New' Proof/Program Refinement Logic) [5,6] to Ensemble. Ensemble [7] is a strong group communication system that develops network protocol support for secure fault-tolerant applications and is the successor to Horus [8]. An Ensemble developer composes the required system from a catalogue of micro protocols. Each micro protocol is coded in O'Caml and so has a formal semantic which can be translated into type theory, that is, the input language to NuPrl. Through NuPrl, the developer can prove correctness theorems or partially evaluate the type theory and so automatically perform some optimisations for common occurrences. This result is then reflected back in the original implementation. Although in this case, the NuPrl/Ensemble combination is a powerful mechanism for reasoning about micro protocols, NuPrl was not deemed to be a suitable basis for the realisation of APPROVE. This was primarily due to the level of user interaction that NuPrl and indeed most other theorem provers require. Since one of the primary project goals was to encourage a greater utilization of formal tools in more practical communities, it was beneficial that APPROVE should offer a 'press-on-the-button' approach. The methodology of automating verification is primarily embodied in a family of formal tools termed model checkers. Thus, APPROVE leverages this technology as its basis. 2.1 Model Checking Model checking is a technique for the automatic verification of software and reactive systems. Applying model checking to a design consists of several tasks [9]: 1. Modeling - the first task is to convert a design into a formalism accepted by a model checking tool. In many cases, this is simply a compilation task, however, where there are limitations on time and memory, the modeling phase requires the use of abstraction to eliminate unimportant details; 2. Invariant specification - before verification, it is necessary to state the properties that the design must satisfy. This specification is usually given using a temporal logic, which asserts how the behavior of the system evolves over time; 3. Verification - the verification is automatic resulting in either a positive result (in which case the invariants hold for the given model) or a negative result which provides a trace of the counter example. Scenarios where a result can not be determined usually occur because of a state space explosion. State space explosion occurs when the model is not sufficiently abstract and so verifications require more memory than is available. In this case either the model is amended or a partial search of the state space is conducted.
J. S. Pascoe and R. 7. Loader /The Agreement Problem Protocol Verification Environment
59
Several general purpose model checkers exist with the Symbolic Model Verifier (SMV) [9], Failures-Divergence Refinement (FDR) [10] and Simple Promela Interpreter (Spin) [11] being three of the more well known. Although SMV and FDR predate Spin, they are possibly more oriented towards formal methods experts than protocol engineers. For example, SMV only delivers a simple boolean result and does not simulate models whereas Spin provides full counter examples and advanced facilities for simulation. Furthermore, as Spin is stable, well documented and uses a C like syntax, it is considered suitable for protocol developers [12, 13, 14]. Also, as the XSpin interface is particularly usable, Spin was deemed the most suitable basis for APPROVE. Further motivation for using Spin came from the prior work of Ruys [15, 16]. In his thesis [15], Ruys provides insight into the modeling of weak multicast and broadcast protocols using Spin. Thus, APPROVE aims to supplement this work by investigating a strong model. Having concluded that Spin is the most suitable platform on which to base APPROVE, it is pertinent to expand upon its theory of operation. Thus, the subjects of automata, verification mechanisms and the state space explosion problem are discussed below. 2.2 Spin Model Checking In this section, the principles underlying the algorithms used for the Spin model checker are introduced. The discussion begins with a description of finite automata which is the method used for representing models suited to verification. Following this, the Spin automata approach to the model checking problem is given. This includes a more formal description of the state space explosion problem. A comprehensive source of additional model checking theory, including a description of the SMV algorithms is Be"rard et al. [17]. 2.2. /
Automata
A finite automaton is a machine which evolves from one state to another under the action of transitions. Overlooking second, a digital watch can be represented as 24 x 60 = 1440 states where each state represents an hour and a minute. Furthermore, each pair of states is linked by a single transition which represents times one minute apart. An arbitrary automaton is denoted using the symbol A and when graphically represented, it consists of circles (states) and arrows (transitions). An incoming arrow without origin identifies the initial state. The availability of such representations is one of the benefits of automata based formalisms and as such, Spin includes the capability to generate automata diagrams at the request of the user. 2.2.2
Automata Based Model Checking
The underlying Spin model checking algorithm is essentially due to Lichtenstein et al. [18]. For brevity, this treatment will summarise certain technical details and for a full treatment the reader is referred to [19]. For Linear Temporal Logic verifications it is not possible to deal with state formulas or the marking of automaton states. Instead Linear Temporal Logic uses path formulas, that is, sets of executions which are independent of a tree representation. Thus, a finite automaton will generally give rise to infinitely many executions, themselves often infinite in length. In this context, the adopted conventional perspective is language theory. Consider for example a Linear Temporal Logic formula 0 :? P which states that P is satisfied infinitely often and is the property under test. An execution 0 > 9b • • • satisfying <j> must contain infinitely many positions qni,Qn.2, • • • where P holds. Between each of these, there can be a finite number of states where -iP holds. Such an execution is said to be of
60
J.
S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
the form: ((->P)* .P)w. In a similar vein, an execution which does not satisfy <j> must, from a certain position onwards, only contain states satisfying ->P. Such an expression is of the form: (P + ~>P)*. (~>P)U1. These two expressions which define the form of an execution that satisfies and does not satisfy 0 respectively are o;-regular expressions and these extend the familiar notion of regular expressions to deal with languages of infinite words. For more information on this, see [17] page 43. The Spin model checker associates with each <£ an w-regular expression £$ describing the form imposed on an execution by its satisfaction of >. Thus, the 'is 0 satisfied (}=) by A' question reduces to 'are all executions of A of the form described by e^,'? In practice, the algorithm does not reason about the regular expressions, but on the automata themselves. Given a formula 0, Spin constructs an automaton B^ that recognises precisely the executions that do not satisfy <£. Spin then strongly synchronises the two automata such that A and B^ progress simultaneously to obtain a third automata: A ® B-^0, whose sole behaviors are the behaviors of A accepted by B^ or the executions of A which do not satisfy 0. Thus, the model checking problem A f= reduces to whether the language recognised by A <8> B^ is empty. 2.2.3
The State Space Explosion Problem
The main obstacle encountered by this and other model checking algorithms is the state space explosion problem. The algorithm presented above requires the explicit construction of the A and B^ automatons. In practice, the number of states in A rapidly becomes very large, particularly when A is divided into several components A\, - . . , An. In this case, the size of the result is in the order of \A\\ x ... x l^j where \A\ denotes the length of A. This is potentially exponential with respect to the system description. More frequently, such explosions arise when automata are interpreted, for example with automata ordering on state variables. When the automata (and possibly some atomic propositions) depend on the values of the state variables, the automaton A submitted to the model checker has to be the automaton of the configurations. For example, an automaton that has m — \Q\ control states and n boolean state variables results in m x 2" states. The discussion of the state space explosion problem concludes the theoretical introduction to the Spin model checking algorithms. The remainder of this section considers the pragmatics of realising APPROVE. Discussion with more practical researchers showed that documenting APPROVE (including its Promela source) was paramount to its usability. In an imperative language such as C, programs are often effectively documented through comments. However, the inherent power of formal notations can lead to scenarios where the verbosity and number of comments compromises the readability of the code. To address this, APPROVE was developed using the noweb literate programming tool [20]. noweb is described next. 2.3 Literate Programming Using noweb Literate programming was first proposed by Knuth [21] as a new programming methodology that primarily promoted two philosophies. Firstly, literate programming combines documentation and source into a fashion suitable for reading by humans, the underlying premise being that the experts insight is more efficiently conveyed if it is stored alongside the code to which it corresponds. Literate programming also aims to free the developer from ordering programs in a compiler specified manner, that is, when writing a program, the developer need not initially concern themselves with distracting side issues but instead focus on the problem in hand. In order to facilitate literate programming, Knuth provided a tool termed WEB [21] which produced both TgX documentation and Pascal code from a file written in the WEB
J. S. P-dscoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
61
notation. One of WEBs drawbacks, was that it was Pascal specific and so this was addressed by Ramsey who produced a language independent tool termed noweb. In terms of applying noweb to Promela, Ruys has contributed significant insight in [22, 15]. Thus, through noweb, the researcher is able to read the APPROVE source code and the corresponding ETgX documentation at an appropriate degree of combination. In the following sections, the noweb notation is introduced concurrently with Promela and the worked example. Thus, the paper now focuses on an introduction to literate verification. 3
Literate Verification Basics: A Brief Introduction
Promela is the modeling language that facilitates abstraction in Spin. A Promela model predominantly consists of processes, variables, message channels and verification constructs. Processes are global objects whereas message channels and variables are declared either globally or locally1. Processes specify behavior, channels and global variables define the environment in which the processes run. In this section a brief introduction to Promela is presented as a series of noweb chunks. Each noweb file must contain a root chunk which defines the order of subsequent chunks. The root chunk for the Promela tutorial is below: 61a (* Promela introduction 61a) = (Busy wait loop 61b) (Blocking statement 6lc) (Proctype: A 62a) (Smallest possible Promela specification 62b) (Longer init proctype 62c) (Channel: qname 62d) (Communication 62e) (Case selection 63a) (Repetition 63b) (Dijkstra Semaphore 64)
3.1
Executability
Central to the basis of synchronisation in Promela, is the notion ofexecutability. In Promela there is no distinction between conditions and statements, even totally isolated boolean conditions can be thought of as statements. Statements are either executable or blocked and processes wait for events to occur by waiting for statements to become executable. For example, instead of writing a busy wait loop: 61 b (Busy wait loop 61b}= while (a != b) skip /* 'skip' being a statement of no effect */
(6la)
the semantic equivalent in Promela is: 61c (Blocking statement 61c)= (a == b)
(6la)
A condition may only be passed (executed) when it holds. If the condition does not hold, execution blocks until it does. 3.2
Process Types and Variables
The state of variables or message channels can only be manipulated by processes. Processes are similar in concept to threads where the behavior of a process is defined in a proctype declaration. The following example declares a process with one local variable called state. Note that Promela does not support anything in between e.g. nested blocks.
62
J.
S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
62a (Proctype: A 62a)= proctype A ( ) { byte state; /* local variable */ state = 3 }
(6la)
Note that the semicolon is a statement separator and not a terminator (hence the absence of a semicolon on line 4 of the above example). In Promela another type of separator is permitted (->) and although the arrow is often used to imply a causal (->) relationship, it is equivalent to the semicolon. Furthermore, as the nature of Promela types is directly inferrable from their C counterparts, an explicit description is not given. Such a description can be found in [23]. 3.3 Process Instantiation A proctype definition only defines the behavior of a process i.e. it does not execute it. Initially, in a Promela model the only process that executes automatically is called init and this must be declared explicitly in every Promela specification2. Thus, the smallest possible Promela specification is: 62b (Smallestpossible Promela specification 62b)= init { skip }
(6la)
Furthermore, the init process can initialise global variables and instantiate other processes. Thus, an init declaration using chunk 62a could be: 62c (Longer init proctype 62c)= init { run A(); run A() /* two instances of X A' running concurrently */ }
(6 la)
where run is a unary operator that takes the name of a process and, providing it is executable, instantiates it. A process may not be instantiated if the number of processes running on the system exceeds a maximum threshold. Note that run may also pass parameters to processes. 3.4 Message Passing Channels are used to model the transfer of data from one process to another. They can be declared either locally or globally in the manner shown in the following example: 62d {Channel, qname 62d)= chan qname = [16] of { int
(61a) }
This declares an asynchronous channel since its buffer size is greater than 0. The corresponding send and receive operations exhibit the same notation as used in CSP. For example, the statements: 62e (Communication 62e)= qname ! expr,- qname ?msg
(6la)
would send the value of expr on the channel qname, receive that value from the same channel and store it in a variable called msg. If qname was full, a send statement will either block or discard the message depending on a selection made by the user. 2 In newer versions of Spin, a proctype declaration may be prefixed with the act i ve keyword to signify that it should be automatically instantiated at the start of each Spin run.
J. S. Fascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
63
3.5 Control Flow Implicitly, two notions of control flow have been introduced: concatenation of statements within a process and concurrent execution of processes. There are two other control flow constructs in Promela that require discussion and these are case selection and repetition. Promela also supports unconditional jumps through the infamous 'goto' statement, but due to the familiarity of this concept, it is not discussed further. Case selection is the simplest control flow structure. Given the relative values of two variables a and b, a choice between two options can be expressed as: 63a {Caseselection 63a) = if :: (a ! = b ) - > / * execute option 1 */ :: (a == b) -> /* execute option 2 */ fi
(6la)
In this instance, the guards are mutually exclusive but this is not mandatory. If more than one guard is executable, one of the corresponding outcomes is selected non-deterministically. If none of the guards are executable, the process will block until at least one can be selected. An extension of the selection construct is the repetition structure. In Promela, this is achieved through the do statement, which in the following example, will repeatedly increment or decrement the variable count and non-deterministically exit the loop when the value of count is 0. 63b (Repetition 63b) = byte count; /* global variable */
(61a)
proctype counter(} { do .- : count = count + 1 :: count = count - I :: (count == 0) -> break /* non-deterministically exit the loop */ od
3.6
Verification Constructs
When used as a verification language, Promela facilitates assertions to be made about the model's behavior. As in C, Promela supports the assert statement which allows developers to insert propositions into the Promela code. In Promela, assert statements are always executable and providing they hold, have no effect. If however, the condition does not hold, the statement produces an error report and aborts the run. Alternatively, if a Promela specification is to be checked for the presence of deadlocks, the verifier must be able to distinguish a normal end state from an abnormal one. A typical example of a normal end state is when all Promela processes have reached the end of their execution and there are no outstanding messages. However, not all Promela processes are designed to reach the end of their program body e.g. protocols often remain in an 'idle' state, or they may wait in a loop ready to engage in some action when new input arrives. Thus, end states are denoted in Promela by the prefix end followed by an individual label. Combinations such as end-protocol _1 and end-state jnumber-2 are valid end state labels.
64
J.
S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
As a full example of the concepts introduced this far, consider the model of a Dijkstra binary semaphore: 64 (Dijkstra Semaphore 64)== ^define p 0 /* Spin uses the C pre-processor */ ttdefine v 1 /* so statements such as these are valid */ chan sema = [0] of { bit
(6 la)
}; /* synchronous channel */
proctype dijkstraO { byte count = 1; end : do :: (count == 1) -> semalp; count = 0 : : (count = = 0 ) -> sema?v; count = 1 od }
proctype user ( ) { do : : sema?p -> /* critical section */ semalv -> /* non critical section */ od
init { run d i j k s t r a O ; run u s e r ( ) ; run user() }
It is not an error for process di j kst ra to have not reached its closing brace at the end of an execution sequence. Note that such a state could still be part of a deadlock, but if so, the deadlock is not caused by this process. The final notion introduced in this overview is the progress state label which marks a state that must be executed for the protocol to make progress. Any infinite cycles in the protocol execution that do not pass through at least one progress state are potentially starvation loops. As with end state labels, progress labels are prefixed with the progress keyword. The introduction of the progress state label concludes this introduction to the basic features of Promela. More detailed tutorials are available in [24, 23, 1 1]. As a consolidation of these ideas, a detailed worked example of a literate verification is presented in [15]. 4 The APPROVE Architecture Revisited At the highest level, the architecture of APPROVE consists of three components (see Figure 1). Each of these is discussed below with the exception of the test protocol, that is, a Promela model of the proposed algorithm. Although inherently this component can not be provided, APPROVE offers a template and guidance for its construction. As one of the primary benefits of APPROVE lies in its reconfigurability, extensive investigation of a test protocol is made simple. For example, it is trivial to reconfigure the model to examine the consequences of employing a different failure detector. Using APPROVE, this is a matter of toggling a boolean flag, whereas manual methods would require extensive alterations.
J. S. Pascoe and R . J. Loader/ The Agreement Problem Protocol VerificationEnvironment
65
Figure 1: The High Level APPROVE Architecture
4.1 The Environment
The environment is the collective term for the entities required to support the simulation and verification of the test protocol. At this level, global channels facilitate message passing for the various sub-entities are declared as are a suite of options which can be used to configure APPROVE for a specific scenario. Possibly the most complex component is the Group Membership Service (GMS). The GMS in APPROVE is virtually synchronous and so it is generally accepted that it has at least the following responsibilities [25]: 1 . Providing an interface for group membership changes - the GMS furnishes processes with a means to create or remove process groups and to join or leave process groups;
2. Notifying members of view changes - all members of a process group are notified when a view change occurs, that is, all processes are informed when hosts join and leave the group or are evicted because they have failed. As with the GMS, client processes exhibit a specific behavior in relation to their operation. Before being admitted to the group, a client must send a join request message to the GMS. Recall that in a virtually synchronous system, view change operations are dealt with differently than in weak group communication systems. In order to guarantee virtual synchrony, messages transmitted in one view must be delivered in the same view, so the GMS responds to a view change operation by broadcasting a flush message. On reception of a flush message, each client consumes its outstanding message queues before signaling the GMS of the flush protocols completion. On receiving an acknowledgment from all of the group members, the GMS adds the joining process to the membership and the new view is broadcast. Once part of the group, an APPROVE client is free to transmit an arbitrary number of messages to other clients. In APPROVE, two group communication primitives are modeled: reliable FIFO multicast and atomic multicast. The motivation for selecting this pair of primitives is that FlFO multicast forms the basis of the CGM quiescent failure detector and atomic ordering is frequently used to transmit the results of agreement algorithms. For now, causal ordering is left as an avenue for future work. Also, at any time a client may request to leave the group (by performing a protocol symmetric to the join) or it can fail. In some systems, a further responsibility of the GMS is to provide failure detection and implicitly, a model of failure. In APPROVE, the conventional fail-stop model of failure is adopted and processes fail by either halting prematurely or being irrecoverably partitioned away. APPROVE models three failure detection mechanisms, two in an independent heartbeat process and the third as part of the client. Moving failure detection from the GMS proctype and into the client not only reduces complexity, but is also more realistic.
66
J. S.Pascoeand R. J. Loader/ The Agreement Problem Protocol VerificationEnvironment
Figure 2: The APPROVE Concept of a Group
5 Modeling APPROVE :Phase 1 (An Ideal System)
The initial APPROVE modeling phase developed an ideal model, that is, nothing was pennitted to fail. The first phase developed models for the global aspects, the GMS and the client entities. This section describes each of these presenting select Fragments of Promela code as the following noweb chunks: 66 (* APPROVE: Modeling phase I 66) (Globalchannel definitions 67) (Modeling the view 68a) (Joinpmtocol 68b) (Flushprotocol 69)
5.I
Global Considerations: Channel and Message Definitions
Based on the conclusions of Ruys [ 151, APPROVE uses a matrix of nine channels to model communication between the various entities. Each client process indexes into the channel matrix by using an identification number assigned to it at instantiation by the init process. Individually, each of the APPROVE channels can be classified into one of the following three categories: 1. General channels - to facilitate communication amongst the group entities;
2. Message guarantees - channels that model delivery ordering semantics; 3. Failure channels - for coordinating failure resolution.
Channels of the first group conform to the labeling convention entity '2' entity, where an entity can be one of c 1i = client, gms = group membership service, hf d = heartbeat failure detector, eh = error handler and e m = error master. An error handler is a process that embodies an instance of the protocol under test. The error master is a term used to address the coordinator of a protocol, viz. the process which collates and determines the algorithm's result'. 'I)rpically,this is then sent to the GMS (using the emagms channel) which evicts any failures and distributes the new view. 31n the interests of consistency, the term error master is used here, however, as APPROVE is suitable for any agreement algorithm, it may be pertinent to opt for a more general term in future releases.
J. S. Pascoe and jR. J. Loader/The Agreement Problem Protocol Verification Environment
67
The second group of channels form the basis of the delivery ordering guarantees. Note that the gbcast channel is synchronous and only carries a single byte. In practice, the only messages to be sent using the globally ordered message passing primitive is the instruction to flush and symmetrically, the acknowledgment from a client that it has completed the protocol. Thus, in APPROVE, the gbcast channel is only permitted to carry the FLUSH and FLUSH JYCK messages. Conversely, the failure channels provide facilities for announcing failures and serve as a modeling interface to the developer. Other channels in this category deal with communication between the error handlers and provide the error master with a means of informing the GMS of those processes deemed to have failed. All of the APPROVE channels are defined with a maximum of three fields where the first is the message type (e.g. JOIN) and the others are values. Thus, the message exchange cli2gtns [2] ! JOIN would correspond to a request from client 2 to join the group. 67 (Global channel definitions 67) = chan cli2gms[NUM_CLIENTS] = [BUFFER_SIZE] of { byte } chan gms2cli[NUM_CLIENTS] = [BUFFER_SIZE] of { byte, int } /* Channels for communicating primarily outside of the group */
(66)
chan cli2hfd[NUM_CLIENTS] = [BUFFER_SIZE] of { byte } chan hfd2cli[NUM_CLIENTS] = [BUFFER_SIZE] of { byte, int } /* Channels for querying the heartbeat failure detector */ chan fbcast[NUM_CLIENTS] = [BUFFER_SIZE] of { byte, byte, byte } chan abcast[NUM_CLIENTS] = [BUFFER_SIZE] of { byte, byte, byte } chan gbcast[NUM^CLIENTS] = [0] of { byte } /* Delivery ordering channels */ chan fail = [BUFFER_SIZE] of { byte, int } chan eh2eh[NUM_CLIENTS] = [NUM_CLIENTS] of { byte, int, int } chan em2gms = [BUFFER_SIZE] of { byte, int } /* Failure channels */
5.1.1
The APPROVE Message Types
In conjunction with the COM exemplar, APPROVE defines sixteen messages which are again split into several sub-groups: 1. Membership messages - voluntary membership operations; 2. Data messages - quiescent reliable failure detection; 3. Failure messages - querying heartbeat failure detectors and announcing failures to the error handler; 4. CGM messages - coordinating CGM. As the role of each message is inferrable, the topic is not discussed further. 5.2
The Group Membership Service
Apart from the roles discussed above, the GMS is also responsible for modeling the view. As Spin opts to convert bit arrays into arrays of bytes, APPROVE models the view using the more efficient bit vector representation.
68
/. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
Through the unsigned keyword, the size of each value can be set to the number of clients and so an optimal amount of memory is used. Manipulation is performed using bitwise operators wrapped in macros. 68a (Modeling the view 68a}= unsigned view:NUM_CLIENTS=0;
(66)
The CMS executes continuously and is instantiated by the init process. In its idle state, the GMS waits for a message to arrive on one of its input channels. Regardless of whether a client wishes to join, leave or the error master is reporting evictions, the GMS behaves in essentially the same manner. On reception of a message, the GMS initially updates its internal view. Then, the FLUSH message is sent to all operational clients instructing them to execute the flush protocol (see chunk 69). Each client delivers all of its outstanding messages before returning a FLUSH-ACK to the GMS. Once all of the clients have completed the flush, the GMS distributes the new view and the operation is complete. Note that priority is given to dealing with membership changes originating from the error master, that is, the GMS explicitly checks for messages on die em2gms channel before dealing with voluntary membership operations. As the length of the GMS model prevents its inclusion as a chunk here, a Promela Pseudo Code outline is given in Algorithm 1. Algorithm (Promela Pseudo Code) 1 Group Membership Service Outline Initially view = 0, i - 0 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.
if :: nempty(em2gms) -> em2gmslmessage -> atomic { update view }; i = 0; do :: ((i < NUM_CLIENTS) && (view & (1 «i))) -> gms2cli[i]!FLUSH; i++ :: (i = NUM_CLIENTS) ->• i = 0; break :: else -»i++ od; Collect the FLUSH.ACK messages -» atomic { broadcast the new view } :: else —> skip fi; do :: (i < NUM.CLIENTS) -»• repeat lines 1-10, but substitute cli2gms[i]/or em2gms :: (i = NUM_CLIENTS) -»• i = 0; goto line 1 od
5.3 The Client Process The APPROVE client process is intended to model a generic group participant. Each client process executes a join protocol and once admitted to the group, is free to exchange an arbitrary number of messages with the other group members or leave the session and terminate its execution. The simplistic join protocol is shown in chunk 68b: 68b (Join protocol 68b)= cli2gtns[id] ! JOIN -> gms2cli [id] ?eval (VIEW) , view -> printf("APPROVE (client % d ) : view received % d . \ n " , i d , v i e w ) ;
(66)
J. S . Pascoe and R. J. Loader/ The Agreement Problem Protocol VerificationEnvironment
69
Recall that after requesting admission to the group, the GMS broadcasts the instruction to FLUSH. If the session is empty, a view containing only the joiner is immediately returned. Otherwise, each client executes the flush protocol: 69 (Flush protocol 69)
(66)
if gbcast [id]?eval (FLUSH) - > do
::
f b c a s t [ i d l ? r e c e i v e r ~ s e t , f r o m , m s g- >
::
if (msg ! = ACK) - > fbcast [from]!from,id,ACK else - > skip fi / * do the same for the abcast channel * / : : gbcast [id1 ?: : gms2cli [id]?-,: : (empty(fbcast[id] && empty(abcast [id] && empty(gbcast [id]1 && empty (gms2cli[id]) ) - > gbcast [id]! FLUSH-ACK; gms2cli [id]?eval (VIEW),view; break od ; printf ("APPROVE (client %d) : flush completed.\n",id) : : empty(gbcast [id] - > skip fi; ::
::
Figure 3: A Message Sequence Chart Depicting Virtually Synchronous Operation
Once part of the group, each client is free to either leave the session or exchange an arbitrary number of f /abcast messages with other processes.
70
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
As it is not meaningful to model message delivery ordering semantics in a failure free environment, the topic will be addressed in the second modeling stage. Thus, at this point, APPROVE was tested and debugged before the second phase commenced. Figure 3 shows a typical Spin simulation using the failure free version of APPROVE. The message sequence chart depicts a typical interaction where client process 2 sends a join request to the CMS at time step 25. As client process 2 is the first to join, the GMS immediately returns a view. Subsequently, client process 3 sends a join request to the GMS and so the flush protocol is executed before the new joiner is admitted. 6
Modeling APPROVE : Phase 2 (Introducing Failure)
The notion of failure in conventional group communication systems is twofold. In the first instance, APPROVE must incorporate at least one failure model, that is, a description of exactly how a process behaves when it fails. The second concept is detection, that is, by what means are process failures identified. As before, select Promela fragments will be presented as chunks: 70a <* APPROVE: Modeling phase 2 70a)= (Fail-stop model of failure 70b) (Selecting a random receiver set 7la) (FIFO delivery and quiescent failure detection 71b) (Atomic delivery and quiescent failure detection 7 Ic) 6.1
The Fail-Stop Model of Failure
Modeling fail-stop failures in APPROVE means adding a further non-deterministic clause to the main do loop in the client process. Clients are only permitted to fail when no other operation is in progress. The reason for this abstraction is to eliminate failure events that would not be handled by the protocol under test, for example, a failure during admission would be dealt with by the join protocol and not the error handler. Intuitively, a fail-stop failure could be modeled as a simple termination event. However, as Promela abstracts away from the low level details of a process' execution status, some form of external 'announcement' is required as an interface to the failure detectors. This was incorporated as a global bit vector mask (termed \htfailedjnembersjnosk) which operates in the same manner as the view, but denotes failure rather than membership. Thus, we have: 70b (Failure model 70b)= /* main client do loop (other non-deterministic clauses) */ :: (FAIL_MODEL == FAIL_STOP) -> atomic { failed_members_mask = failed_members_mask | (l«id) ; } printf ("APPROVE (client: %d) : failed. \n", id) ,- break 6.2
(70a)
Quiescent Failure Detection and Delivery Ordering Primitives
A quiescent reliable failure detector considers reliable messages as probes of the sessions liveness. The main advantage being the absence of processing overhead in a failure free environment and a drawback is the arbitrary detection intervals. In order to model an arbitrary message exchange, each client must be furnished with the ability to select a destination set at random. In APPROVE, this is achieved using a rationalised random number generated by the inline random definition suggested by Ruys [15].
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment 7la (Selecting a random receiver set 71a)EE random(receiver_set,(2"NUM_CLIENTS)-1}; /* Select a random value between 0 and (2~NUM_CLIENTS)-1 */
71 (70a)
receiver_set = receiver_set & view; if
:: (receiver_set & (l«id) ) -> receiver_set = receiver_set " (l«id) : : else -> skip fi
/* rationalise the value into a valid destination set */ Modeling the reliable FIFO primitive is achieved by selecting a random receiver set and iteratively inspecting each of its members. If a host is a member of both the receiver set and the failed members mask, then a new failure has been detected and is announced over the f ailchannel. Note that duplicate failure reports are ignored by the error master. If a recipient has not failed, then a DATA message is exchanged for an acknowledgment. 71 b (FIFO delivery and quiescent failure detection 71 b) = (70a) i = 0 -> do
:: ((i < NUM__CLIENTS) && (receiver_set & (l«i)) -> if
:: (failed_members_mask & (l«i) ) -> fail!FAIL,i; i++ /* announce the failure */ : : else -> fbcast[i]1DATA,id,receiver_set; fbeast[i]?eval(ACK),_,_; i++ fi
-. : (i == NUM_CLIENTS) -> break : : else -> i++ od
The difference between the model for the reliable FIFO primitive and the atomic algorithm is that the latter will initially check that none of the recipients have failed. If this is the case, then an atomic message exchange is executed. Conversely, the operation is aborted, and the failures are reported. 71 c (Atomic delivery and quiescent failure detection 71c}= (70a) i = 0 -> atomic { if :: (receiver_set & failed_members_mask) -> do :: ({i < NUM_CLIENTS) && (receiver_set & (l«i) ) && (f ailed_members_mask & (l«i) ) ) -> f ail! FAIL, i; i + + :: (i == NUM_CLIENTS) -> break : : else ~ > i + + od
: .- else -> i = 0 -> do
:: ((i < NUM_CLIENTS) && (receiver_set (l«i) ) ) -> abcast [i] 1DATA, id, receiver__set; abcast [i] ?eval (ACK) ,_,_; i + + :: (i == NUM_CLIENTS) -> break :: else -> i++ od fi }
6.3
Heartbeat Failure Detection
Heartbeat failure detectors (HFDs) are used in many systems (though not in CGM) and so were deemed essential to the APPROVE catalogue. Heartbeat failure detection differs from
72
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
quiescent mechanisms in one important aspect, namely, HFDs are triggered periodically. In Spin, the notion of time is implicit, that is, it is not possible to reason about specific durations and so the Promela model of an HFD has to abstract away from its traditional implementation. Note that the key distinction preserved by APPROVE is that heartbeat failure detection is independent of the pattern of communication, that is, an HFD detects failures on the basis of a loop, whereas quiescent mechanisms detect failures arbitrarily. In a similar vein, the mechanism by which HFDs detect failure is also modeled differently from conventional implementations. HFDs can be categorised into two groups: ping (or explicit acknowledgment) and 'I'm alive'. When using a ping HFD, each group member will periodically broadcast a message to all others before awaiting a series of acknowledgments. If after waiting 8t units of time an acknowledgment has not been received, then the host it refers to is suspected of failure. Similarly, a process using an Tm alive' HFD will periodically broadcast a message announcing its continued presence to the group. If such a message is not received in 6t units of time, then again the corresponding host becomes a failure suspect. In terms of triggering the HFD, it is not possible to effectively reason about a specific timeout duration (5t). In APPROVE, two heartbeat failure detectors are modeled. The first is a general model which polls the value of the f ailedjnembers-mask for changes, whereas, the latter pends on the failure event (see Algorithm 2). This triggering abstraction results in a significant decrease in interleaving and so reduces complexity. Algorithm (Promela Pseudo Code) 2 Pending Heartbeat Failure Detector Initially old .failed .members jnask = failedjnembers jnask, i = 0 1. 2.
if :: (old Jailed _members_mask != failed-members _mask) —>•
3.
i = 0;
4. 5. 6. 7. 8. 9. 10. fi
do :: (failed-membersjnask & (1«i)) && (! (old Jailed -members _mask & (l«i)))-^fail!FAIL,i;i++ :: else —* i++ od; old_failed_membersjnask = failed jnembers _mask; goto line J
7 Invariant Specification The development of the heartbeat failure detectors concluded the APPROVE modeling phases. Subsequently, the instrumentation of the model for the purposes of verification was considered. One of the beneficial aspects of the APPROVE project was that the termination, agreement, validity and irrecoverability invariants were known from its inception. Termination is tested through the introduction of end state labels in combination with an explicit idle state in the error handler. Thus, if a verification terminates and any of the error handlers are not idling, then Spin detects and reports the violation. Conversely, the other invariants are interrelated; thus, these are discussed in combination. Note, that the noweb chunks referred to throughout the next section are listed below: 72 (* APPROVE. Verification phase 12)= (Assigning a new result 73a) (Checking validity 73b)
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
7.1
73
Irrecoverability, Validity and Agreement
The main distinction between termination and the other invariants is that irrecoverability, validity and agreement are only in question when a new result is assigned, that is, when the error handler master receives a result from a client and wishes to store it. In this case, testing for irrecoverability is the same as verifying that a result has not been previously received (and so set) for a particular client. This is achieved using an assert statement in conjunction with a bit vector of flags. Thus, the assignment of a new result implies verifying that a result has not been set previously, before storing the clients input and resultant views in global arrays. This is encapsulated in the following inline definition which comprises part of the APPROVE user template model: 73a (Inline: result assignment 73a} = inline ASSIGN_RESULT(input_view,result,id) { atomic { assert ( ! (verif ication_result_set & (l«id) ) ) ; verification_input_view[id] = input_view; verification_output_view[id] = result; verification_result_set = verif ication_result_set | (l«id) ; CHECK_VALIDITY() CHECK_AGREEMENT() /* use inlines for validity and agreement */ } )
7.1.1
(72)
Inlines vs. Never Claims To Guarantee Validity and Agreement
Note the use of the inline statements CHECK-VALIDITY and CHECK.AGREEMENT in chunk 73a above. Intuitively, the validity and agreement invariants lend themselves to expression by a never claim; indeed a significant amount of effort was invested in pursuing this idea. Spin never claims apply invariants to the global space of the model whereas in this case, the validity and agreement invariants only apply to the client processes. It is possible for never claims to inspect variables local to processes suggesting the idea of using a bounded do loop inside a never claim to cycle through each client in turn checking the invariants. However, due to the assignment of the counting index, Spin objects warning that the never claim contains side effects. Although the side effect is known to be safe, it is arguable that the approach contravenes the philosophy of the never claim and so the alternative method of using inlines was adopted. Actually verifying validity and agreement is again based on a bounded do loop. Due to their similarity, only the validity chunk is presented: 73b (Checking validity 73b)= (72) inline CHECK__VALIDITY ( ) { i = 0 -> do :: ((i < NUM_CLIENTS) && (verification_result_set & (l«i))) -> j = 0 -> do :: ( (j < NUM_CLIENTS) && (verification_result_set & (l«j)) && (i!=j) && (verification_input_view[i] == verification_input_view[j])) -> assert(verification_output_view[i] == verification_output_view[j]); j++; :: (j == NUM_CLIENTS) -> i++; break : : else - > j +-f od :: (i == NUM^CLIENTS) -> break od; }
The introduction of the verification constructs concluded the development phase. Next, consider the application of APPROVE to an exemplar agreement problem, which in this case is a revisitation of COM.
74
8
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
Collaborative Group Membership Revisited
The Collaborative Group Membership algorithm provides fault-tolerant group membership in collaborative systems [4]. At a high level, COM consists of two complementary entities: the error monitor which provides first level processing of transport layer fault reports and the error handler which coordinates two distributed elections to resolve membership. The CGM model abstracts away from the error monitor and the session (second) election, focusing on the membership removal (first) election. Recall that when triggered, the error handler broadcasts an EL_START message followed by an EL.CALL. This informs the other group members that an election is pending and that they should refresh their suspect lists. Using the quiescent failure detector, each client reliably broadcasts an EL_PROBE message to its peers. The underlying reasoning being that this will generate new fault reports for failures that were not detected previously. Client views based on heartbeat failure detectors can be refreshed by consulting the failure detector directly. This returns a list of suspects in the form of a bit vector. Based on its refreshed view, each client votes for the removal of the members it deems to have failed and sends a digest of the result to the error master via a point-to-point EL_RETURN message. Having received all of the votes, the error master determines the outcome and instructs the GMS to evict any agreed failures. 8.1
Results
Verifying CGM with APPROVE raised a number of interesting points, though as the CGM design had already been refined during implementation, no new substantial design errors were found. However, APPROVE was particularly beneficial in evaluating the semantic implications of eliminating the probe mechanism which was previously identified as the quadratic component of the algorithm's message complexity [4]. A series of experiments was conducted in which this was investigated. In addition, it was decided that these experiments would quantitatively investigate APPROVE's performance in relation to larger groups (i.e. between 4 and 10 clients). The expected result was that there exists a small linear increase in overhead for each additional client process. A session consisting of two client processes is an exception to this hypothesis since the system is effectively point-to-point and so has a significantly simpler interaction. The experimental environment consisted of one Pentium III desktop machine using a 600 MHz processor and 768 Mb of RAM. In terms of software, the PC is running Linux Mandrake 6.5 and Spin version 3.4.10. Each experiment was conducted using the same compiler and run-time options. To repeat these experiments, for compilation define: -D_POSIX_SOURCE -DBITSTATE -DSAFETY -DNOCLAIM -DXUSAFE -DVECTORSZ=4000
For the arguments to pan, define: -X -n>3000000 -w29 -cl
Note the use of the partial search. As is discussed in the APPROVE critique, exhaustive verifications are possible, but are limited to a smaller number of clients. Thus, studying APPROVE in the context of larger groups necessitates the use of the partial search. The results from the experiments are shown in Table 1. When the model was reconfigured to not probe, Spin did not detect any invariant violations, suggesting that the probe could be removed from the protocol. In addition, a significant decrease in the overhead required to complete the verification (particularly for the larger groups) was observed. The cause for the sudden increase in verification time (at 8 clients) has not been determined, but it is conjectured that it is due to a specific pattern of interaction
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
75
Probe enabled / NUTVLCLIENTS
sv
4
5 1108 1388202 2.26861 2:32:17
6
7
8
9
2708 1440 1808 2216 1227178 1819674 1639334 1244615 2.22499 2.12071 3.16569 2.70916 6:28:39 5:56:39 2:36:55 2:56:02 Probe disabled / NU1VLCLIENTS
10 3200 1454759 2.23208 4:50:41
T
820 1288425 2.10856 1:40:50
4
5
6
7
8
9
10
SV DR SS T
724 1288748 1.89953 1:22:52
928 1464371 2.07028 1:40:50
1152 1933316 2.12812 1:58:37
1388 1895244 2.18374 2:19:03
1640 1312446 2.25818 2:39:49
1952 1415283 2.85684 4:23:59
2240 1356739 2.46325 3:42:05
DR
ss
Table 1: Quantitative Effect of Toggling the CGM Probe in APPROVE. Note the following key: SV is the state vector measured in bytes, DR is the depth reached, SS is the number of states stored (the decimals are e+08) and T is the averaged elapsed time (hours:minutes:seconds). All of the experiments used 219.02 Mb of memory and no errors were reported at any stage.
that is less amenable to optimisation. However, this anomaly poses only verification performance implications. During Spin simulations, it was that further failures would cause the CGM algorithm to restart. Based on these results, it was confirmed that the probe would not be removed completely, but its use would be restricted to environments where failures occur in rapid succession. Normally, the algorithm operates without the probe yielding a linear message complexity and less overhead in the Promela model, but, where failures occur in rapid succession, the probe can be employed to avoid multiple iterations of CGM. 9 Future Work: Extending APPROVE The next phase of the work considers the issue of heterogeneity in the context of strong group communication and as a precursor to this study it was pertinent to formally investigate the effect of employing strong group communications in a wireless environment. Through the recent maturation of related technologies, strong wireless group communication (particularly in the mobile sense) is being suggested as one of the next paradigms in the field. Thus, in this section, APPROVE is extended to include a wireless model of failure. The main distinction between the fail-stop and wireless models is the notion of intermittent connectivity. More formally, a wireless model of failure states that in addition to the fail-stop model, a host may move into a trouble spot, that is, an area of poor (or no) communication before emerging after an arbitrary period of time. Furthermore, it is possible that whilst in a trouble spot, a host may fail. Thus, a wireless model of failure can be modeled as follows (where the root chunk is omitted for brevity): 75
(wirelessfailure 75)== ::
(FAIL_MODEL == WIRELESS)
(75) ~ > atomic {
failed_members_mask = failed_members_mask (1 « id); do : : failed_members_mask = failed_members_mask " (1 « id) ; goto end_client_top /* Recovered */ :: printf("APPROVE (client: %d): failed, failed members mask: %d.\n" , id, f ailed_members_mask) ,- goto end_client_done; /* Failed */ od
76
J.
S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
Through simulations with the wireless model of failure, it was observed that the system encountered difficulties, that is, the membership algorithm was erroneously triggered (leading to the removal of essentially live hosts) and in some cases, the delivery ordering/virtual synchrony semantics were invalidated. To an extent, this is intuitively expected, however, the APPROVE simulations neatly distiled two future problems namely: developing a more insightful model of failure and providing virtual synchrony in the wireless domain. In pointto-point scenarios, trouble spots may manifest as slow links, stalled web page etc. but, in the context of strong group communication this perceived joining and leaving compromises the overall usability of the system. Note that this is not simply an extension of the network partition problem. In providing truly heterogeneous strong group communication trouble spots and their effect on the system's semantics must be reconciled to a much greater extent. 10 Conclusion and Critique In this paper, the Agreement Problem Protocol Verification Environment (APPROVE) has been presented. The discussion began with a tutorial on the techniques used to realise APPROVE and this facilitated a description of the development process. Later, APPROVE was instrumented for verification and applied to an exemplar (COM). APPROVE did not detect any substantial design errors which is possibly because the CGM design had already been refined through implementation. However, APPROVE was particularly useful in determining the semantic implications of removing the probe mechanism. Evidence from a series of experiments suggests that the probe can be safely removed, and so probe optimisation strategies can be safely investigated. The final consideration was the extension of APPROVE to include a wireless model of failure. Through this it was suggested that one of the fundamental problems in directly deploying strong group communication systems in a mobile wireless environment is the perceived pattern of membership (which is due to trouble spots). This is discussed in greater depth in [26], however, before closing the discussion on APPROVE, the approach is critiqued. 10.1 APPROVE Critique It is tentatively suggested that APPROVE fulfilled all of its design goals and contributed a compact versatile framework to the community. The reaction from a range of researchers has been encouraging with APPROVE enjoying moderate usage. However, the approach embodies several caveats: • Partial verifications - the state space explosion problem prevents exhaustive verification of APPROVE models using machines with memories smaller than 2 Gb. This stems from an inversely proportional relationship between abstraction and realism, that is, as the model becomes more abstract (and so has a smaller search space), it also becomes more difficult for a strong group communications expert to understand and interpret APPROVE's operation. However, it should be noted that Spin's partial searching techniques are particularly extensive and it is postulated that a partial search offers a much greater degree of confidence than the conventional manual alternatives; • Bounded verifications - in its present form APPROVE verifications are conducted with respect to an explicit number of client processes, which for pragmatic reasons, reduces the size of the state space. In practice the scalability of virtual synchrony is limited, which means that systems of this nature rarely operate with more than 20 processes (see Birman [27]). From the results in Table 1, it is clear that APPROVE is capable of performing verifications at this level. However, one advantage of manual methods is
J. S. Pascoe and R. J. Loader/ The Agreement Problem Protocol Verification Environment
77
that they verify correctness in the absolute sense, that is where NUM-CLIENTS = #(G). In order to achieve this in APPROVE, it would be necessary to adopt a fundamentally different technology. A review of the field has suggested that current emphasis is on scalability. However the APPROVE work moves orthogonally, investigating the provision for heterogeneity in strong group communication systems. This is not only in terms of wireless environments, but also hybrid and multi-characteristic wired networks. References [1] Tushar Deepak Chandra and Sam Toueg. Unreliable Failure Detectors for Reliable Distributed Systems. Journal of the ACM, 43(2), March 1996. [2] K. P. Birman. Building Secure and Reliable Network Applications. Manning, Greenwich, 1996. ISBN: 1-884777-29-5. Also available as: World Wide Web page: http://www.cs.comell.edu/ken/. [3] L. Lamport, R. Shostak, and M. Pease. Byzantine Generals Problem. ACM Transactions Programming Languages and Systems, 4(3):382-401,1982. [4] J. S. Pascoe, R. J. Loader, and V. S. Sunderam. Collaborative Group Membership. The Journal of Supercomputing, 22(l):55-68, May 2002. ISSN: 0920-8542. [5] S. F. Allen, R. L. Constable, R. Eaton, C. Kreitz, and L. Lorigo. The NuPRL Open Logical Environment. In D. McAllester, editor, Proc. 17th International Conference on Automated Deduction, number 1831 in Lecture Notes in Artificial Inteligence, pages 170-176. Springer-Verlag, Berlin, Germany, 2000. [6] Mark Bickford, Christoph Kreitz, Robbert van Renesse, and Xiaoming Liu. Proving Hybrid Protocols Correct. In Proc. 14th International Conference on Theorem Proving in Higher Order Logics (TPHOLs 2001), volume 2152 of Lecture Notes in Artificial Inteligence, pages 105-120, Edinburgh, Scotland, September 2001. Springer-Verlag, Berlin, Germany. [7] Ohad Rodeh, Kenneth P. Birman, and Danny Dolev. The Architecture and Performance of Security Protocols in the Ensemble Group Communication System. ACM Transactions on Information and System Security (TISSEC), 2001. Accepted: 17th October 2000. Also available as: technical report: TR20001822, Department of Computer Science, Cornell University. [8] Robbert van Renesse, Kenneth P. Birman, Roy Friedman, Mark Hayden, and David A. Karr. A Framework for Protocol Composition in Horns. In Symposium on Principles of Distributed Computing, pages 80-89, Ottawa, Ontario, Canada, August 1995. ACM Press, New York. [9] Edmund M. Clarke Jr., Orna Grumberg, and Doron A. Peled. Model Checking. The MIT Press, Cambridge, MA & London, England, 1999. ISBN: 0-262-03270-8. [10] A. W. Roscoe. The Theory and Practice of Concurrency. Prentice Hall, 1997. [11] Gerard J. Holzmann. Design and Validation of Computer Protocols. Prentice Hall Software Series. Prentice Hall, Upper Saddle River, New Jersey, 1991. ISBN: 0-13-539925-4. Also available as: World Wide Web page: http://cm.beU-labs.com/cm/cs/what/spin/Doc/Book91.html. [12] J. S. Pascoe, R. J. Loader, and V. S. Sunderam. The Agreement Problem Protocol Verification Environment. In Model Checking Software. Proc. of The 9th International Spin Workshop on Model Checking of Software. In association with ACM SIGPLAN, volume 2318 of Lecture Notes in Computer Science, pages 148-169, Grenoble, France, April 2002. Springer-Verlag, Berlin, Germany. ISBN: 3-540-43477-1. [13] J. S. Pascoe, R. J. Loader, and V. S. Sunderam. Working Towards the Agreement Problem Protocol Verification Environment. In Alan Chalmers, Majid Mirmehdi, and Henk Muller, editors, Communicating Process Architectures 2001, volume 59 of Concurrent Systems Engineering, pages 213-229, Bristol, UK, September 2001. IOS Press. ISBN: 1 -58603-202-X.
78
J. S. Pascoe and 1?. /. Loader/ The Agreement Problem Protocol Verification Environment
[14] J. S. Pascoe, R. J. Loader, and V. S. Sunderam. An Election Based Approach to Fault-Tolerant Group Membership in Collaborative Environments. In Proc. of The 25th IEEE Anniversary Annual International Computer Software and Applications Conference (COMPSAC), pages 196-201, Chicago, IL, October 2001. IEEE Press. ISBN: 0-7695-1372-7. [15] Theo C. Ruys. Toward Effective Model Checking. PhD thesis. University of Twente, March 2001. ISBN: 90-365-1564-5. [16] Theo C. Ruys. Low-Fat Recipes for Spin. In Proc. 7th Spin Workshop, number 1885 in Lecture Notes in Computer Science, pages 287-321, Stanford University, California, September 2000. Springer-Verlag, Berlin, Germany. [17] B. Berard, M. Bidoit, A. Finkel, F. Laroussinie, A. Petit an L. Petrucci, Ph. Schnoebelen, and P. McKenzie. Systems and Software Verification: Model Cheking Techniques and Tools. Springer-Verlag, Berlin, Germany, 1999. ISBN: 3-540-41523-8. [18] O. Lichtenstein and A. Pnueli. Checking That Finite State Concurrent Programs Satisfy Their Linear Specification. In Proc. 12th ACM Symposium on Principles of Programming Languages (POPL'85), pages 97-107, New Orleans, LA, January 1985. ACM Press, New York. [19] M. Vardi and P. Wolper. An Automata-Theoretic Approach To Automatic Program Verification. In Proc. 1st IEEE Symposium Logic in Computer Program Verification (LICS'86), pages 332-244, Cambridge, MA, June 1986. IEEE Press. [20] N. Ramsey. Literate Programming Simplified. IEEE Software, 11:95-105,1994. [21] D. E. Knuth. Literate Programming. Center for the Study of Language and Information, 1992. [22] Theo C. Ruys and Ed Brinksma. Experience with Literate Programming in the Modeling and Validation of Systems. In Bernhard Steffen, editor, Proc. 4th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'98), number 1384 in Lecture Notes in Computer Science, pages 393-408, Lisbon, Portugal, April 1998. Springer-Verlag, Berlin, Germany. [23] Rob Gerth. Concise Promela Reference, June 1997. Available from: http://cm.bell-labs.com/cm/cs/what/spin/Man/Quick.html. [24] G. J. Holzmann. Basic Spin Manual, August 1997. Available from: http://cm.bell-labs.com/cm/cs/what/spin/Man/Manual.html. [25] G. Coulouris, J. Dollimore, and T. Kindberg. Distributed Systems Concepts and Design. Addison-Wesley, third edition, 2001. See chapter 11 for Coordination and Agreement problems. [26] J. S. Pascoe, V. S. Sunderam, U. Varshney, and R. J. Loader. Middleware Enhancements for Metropolitan Area Wireless Internet Access. Future Generation Computer Systems, 18(5):721-735, April 2002. ISSN: 0617-739X. [27] Ken Birman, Bob Constable, Mark Hayden, Jason Hickey, Christoph Kreitz, Robber van Renesse. Ohad Rodeh, and Werner Vogels. The Horus and Ensemble Projects; Accomplishments and Limitations. In Proc. DARPA Information Survivability Conference & Exposition, Hilton Head, South Carolina, January 2000. IEEE Press.
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
On the Complexity of Buffer Allocation in Message Passing Systems Alex BRODSKY, Jan Bskgaard PEDERSEN and Alan WAGNER Department of Computer Science, University of British Columbia Vancouver, EC, Canada, V6T1Z4
(abrodsky,matt,wagner}@cs.ubc.ca Abstract. In modern cluster systems, message passing functionality is often offloaded to the network interface card for efficiency reasons. However, this limits the amount of memory available for message buffers. Unfortunately, buffer insufficiency can cause an otherwise correct program to deadlock, or at least slow down. Hence, given a program trace from an execution in an unrestricted environment, determining the minimum number of buffers needed for a safe execution is an important problem. We present three related problems, all concerned with buffer allocation for safe and efficient execution. We prove intractability results for the first two problems and present a polynomial time algorithm for the third.
1
Introduction
For efficiency reasons, most modern clusters off-load message passing functionality to the network interface card (NIC) [1] to facilitate a greater overlap of computation and communication. Unfortunately, most NICs have two orders of magnitude less memory than the average host, which makes message buffers a limited resource. Thus, programs that use asynchronous message passing and execute correctly otherwise, might dead-lock when executing on a system where parts of the message passing system have been off-loaded to the NIC; such issues have been investigated in [2, 3,4]. Ideally, we want to detect the possibility of dead-lock and prevent it from happening. The MPI message passing standard [4] defines a "safe" program as one that requires no buffering and uses synchronous communication. Since better performance can be obtained by taking advantage of system or network buffers to reduce unnecessary synchronizations— asynchronous communication—the notion of a "k-safe" (asynchronous) program arises. A "k-safe" program requires k buffers to guarantee completion [1, 5, 6]. Unfortunately the value of k is usually not known a priori. In this paper we investigate the complexity of determining a minimum value of k for programs that use asynchronous buffer communication and have static communication patterns. Under these assumptions we show that the problem of determining k, the minimum number of buffers for deadlock free execution, is NP-hard. Furthermore, we show that the simpler task of verifying whether a buffer assignment is sufficient to avoid deadlock is coNP-complete. Finally, we give a polynomial time algorithm for determining the exact number of buffers needed to ensure nonblocking sends, achieving efficient program execution. Interestingly, the nonblocking requirement makes the problem tractable! Motivations and precise definitions of the system and the problems considered are given in the following sections.
80
2
A. Brodsky et a/. / The Complexity of Buffer Allocation
Background
Determining the k-safety of a program is important for several reasons. First, the semantics of asynchronous communication depend on the availability of buffers, and change—in an implementation dependent fashion—when buffer space becomes exhausted. Since buffer space is typically limited, or pre-allocated during application initialization, a priori knowledge of the application's buffer requirements ensures that communication semantics do not change part way through the execution. Second, to improve performance, many systems use zero-copy techniques: messages are transferred directly from the NIC to application buffers. Since these buffers must be pre-allocated by the application, the application must know the number required. Third, one of the most common purposes of parallelization is to enable the solutions of bigger problems on larger data sets. Typically, host memory is the limiting resource and, if the number of buffers needed is known, memory utilization can be improved. Finally, communication libraries like MPI allow the application to manage the buffer space; this can be optimized if the number required is known in advance. The most natural primitives for asynchronous buffered communication are "nonblocking sends" and "blocking receives"; these are also the standard communication primitives in MPI [4] and PVM [7]. Cypher and Leu formally define the former as a POST-SEND immediately followed by a WAIT-FOR-BUFFER-RELEASE and the latter as a POST-RECEIVE immediately followed by a WA1T-FOR-RECEIVE-TO-BE-MATCHED [8, 9]. Informally, the send blocks until the message is copied out of the send buffer and the receive blocks until the message has been copied into the receive buffer. We only consider point-to-point communication similar to the model used in [10], since multicast and broadcast communication can be simulated with point-to-point communication. A multiprocess system 5" is a set of simultaneously executing independent asynchronous processes that perform a computation by interspersing local computation and point-to-point message passing between processes; these are referred to as A-computations in [10]. Such a system is equivalent to the system with three different events like the one defined by Lamport [11]: send events, receive events and internal events. Send events cause messages to be sent, receive events causes messages to be received and internal events represent computation on internal state. The system uses nonblocking sends and blocking receives; processes synchronize by communicating and no assumptions can be made about the computation time of any process. When a process performs a send, the message may either be sent to the receiving process where it is either received or buffered, or the sending process blocks. If the receiving process is ready to receive (i.e., has issued a receive request for the incoming message), or the message passing system has free buffers available, the sending process does not block. If no buffers are available and the receiving process is not ready to receive, the sending process blocks until one of the conditions changes. These communication primitives are comparable to the default send/receive primitives found in PVM and MPI: blocking receives and nonblocking sends when buffers are available and blocking when not. In such a model each process has a pool of buffers for incoming messages used on a first come first served basis; we call these receive buffers. Buffers are used when a message is not ready to be received by the receiving process. Buffers are returned to the receiving process's buffer pool when the message has been received (i.e., the corresponding receive request has completed). Other models that use only send buffers, a combination of send and receive buffers, or buffers that are allocated on a per channel basis, are possible. While the focus of this paper is on the receive buffer model, our results are applicable to other models as well. In this paper we consider programs that are repeatable [8, 9] when executed in an unrestricted environment, i.e., programs with static communication patterns. While this narrows
A. Brodsky et al. / The Complexity of Buffer Allocation
81
the class of programs we consider, the class of applications with static communication patters is still considerable, including: grid computations, linear system computations, and pipe line computations. Since the communication pattern must be static we do not consider the case where a receive can specify a wild card, i.e., a receive specifies exactly one sender; regardless of this restriction the problems we consider remain intractable. A message history M for the execution of a system S is a set of messages of the form m — (i, 3j, j, Sj) where i and j are process identifiers; i represents the sender and j represents the receiver. The sequence numbers Sj and Sj are local to the sending and receiving processes, respectively. For process i the first send or receive has sequence number 1, the second has number 2, and so forth. We use a communication graph G(S), described in the next section, to encode the message history; sends and receives are encoded as vertices, and the sequence numbers are represented by the topological order of the vertices in G(S). Even though the communication pattern is static, the arrival time of messages varies from execution to execution. The following example shows this effect and how the allocation of buffers can result in deadlock. Consider the three process system depicted in Figure 1 where Pi and pa each have zero buffers available and process pa has one buffer available. Two different scenarios exist. In the first scenario, message 1 arrives before message 3 and takes up the buffer. Now processes P2 and p$ can never progress as one additional buffer is needed to break the deadlock between them. In the second scenario, message 3 arrives first and takes up the buffer. Process pa can proceed to its receive call and block until message 2 arrives. Consequently, process p2 will send message 2, receive message 1 followed by message 3. Thus, all processes finish and no deadlock occurs. For example, if process ps had one buffer rather than process p2, the system depicted in Figure 1 is guaranteed to complete for all executions.
Figure 1: Order of execution can cause deadlock.
As demonstrated, the order in which messages arrive at a process determines whether a system deadlocks. We call a system that exhibits such behaviours unsafe. This situation can be rectified by an appropriate allocation of buffers to processes. To conserve memory, we want to allocate only as many buffers as needed. For programs with static communication patterns, we would like to determine a minimum buffer assignment or to determine whether a given buffer assignment prevents deadlock. As we will show, both of these problems are intractable. If system S is to execute efficiently, ideally, no send operation blocks. In addition to safety, we like to determine the minimum buffer assignment necessary to prevent blocking sends. Interestingly, the stricter constraint makes the corresponding buffer allocation problem tractable. Determining whether a system is buffer independent—the system is 0-safe—was investigated in [8, 9]. In our model the interesting systems are buffer-dependent, and require an unknown number of buffers to avoid deadlock. To determine the minimum number of buffers, the execution of a system can be modeled using a (coloured) Petri net [12]. In order
82
A. Brodsky et al. / The Complexity of Buffer Allocation
to determine if the system can reach a state of deadlock, the Petri net occurrence graph [13] is constructed, and a search for dead markings is performed. However, the size of the occurrence graph is exponential in the size of the original Petri net. A variation of this problem has been investigated by the operations research community [14, 15, 16]. In these models, events or products are buffered between various stations in the production process, however, the arrival of these events is governed by probability distributions, which are specified a priori. In our model, since processes are asynchronous, the time for a message to arrive is non-deterministic, i.e., a message may take an arbitrarily long time to arrive and a process may take an arbitrarily long time to perform a send or a receive.
3
Definitions
Let S be a multiprocess system with n processes and Ej communication events occurring in process i; a communication event is either a send or a receive. A communication graph of 5" is a directed acyclic graph G(S) = (V, A). The set of vertices V = {viiC \ 1 < i < n. 0 < c < (E,; + 1)} corresponds to the communication events. The arc set A consists of two disjoint arc sets: the computation arc set P and the communication arc set C. Each vertex represents an event in the system: vertex u,.0 represents the start of process i, vertex ^z.o 1 ^ c < Ei, represents either a send or a receive event, and vertex v,.(E,+i) represents the end of a process. An arc (v,;.c, Vi. c +i) € P, 0 < c < E{, represents a computation within process i and an arc (vi<s, Vj_t) € C represents a communication between different processes, 1 and j, where vt_s is a send vertex, and Vj_t is a receive vertex (e.g. Figure 2). Note, the process arcs are drawn without orientation for clarity; they are always oriented downwards. In general vertices are labelled with double indices, representing the process label and the sequence number of the corresponding event. The former is dropped when the process label is given by the context. Communication graphs are comparable to the time-space diagrams— without internal events—noted in [11].
PI
P2
o---S ^2,2
8 Figure 2: An example of a communication graph with a 2-ring.
A multiprocess system S is unsafe if a deadlock can occur due to an insufficient number of available buffers; if S is not unsafe, then S is said to be safe. Figures 1 and 3 are examples of unsafe systems. Note: the numbers above the graph in Figure 3 represents the buffer assignment. A buffer assignment is an n-tuple B = (6^ 62,..., 6 n ) of non-negative integers mat represent the number of buffers that can be allocated by each process. Since buffers use up memory, which may be needed by the application, ideally, as few buffers as possible should be allocated. However, allocating too few buffers can result in an unsafe system. Two natural decision problems arise from this optimization problem. Given a communication graph G(S) and a non-negative integer k, the Buffer Allocation Problem (BAP) is to decide if there exists a buffer assignment B = (&i,62, • •-, M such that S is safe and Zr=i h < k. In order to solve this problem we need to solve a simpler one. Suppose we are
A. Brodsky et al. / The Complexity of Buffer Allocation
0
1
0
83
1
Figure 3: Order of buffer selection can cause deadlock.
given a buffer assignment B = (61,6 2 ,..., &„) and a communication graph G(S), the Buffer Sufficiency Problem (BSP) is to decide if the assignment is sufficient to make S safe. Additionally, we can require that no process in system S should ever block on a send. Given a communication graph G(S) and a non-negative integer fc, the Nonblocking Buffer Allocation Problem (NBAP) is to decide if there is a buffer assignment B = (61? 6 2 ,..., bn), such that no send in S ever blocks, and £"=1 &i < ^;- Next, we introduce the terminology used throughout the paper. 3.1
Terminology
The ith process component Gi(S) of G(S) is the subgraph Gt(S) = (K, A) where V% = {viiC € V 0 < c < (Ei + 1)} and A, = {(vilC, Vt )C +i) € A | 0 < c < £J. The process component corresponds to a process in S1. We construct communication graphs by connecting process components with arcs. Hence, it is more intuitive to treat a process component as a chain of send and receive vertices bound by a start and an end vertex. A t-ring is a subgraph of a communication graph G(S), consisting of t > 1 process components such that in each of the t process components there is a send vertex s^. and a receive vertex ri}4j, c3 < djt I < j < t such that the arcs (s fl , c ,, rit,dt) and (s^, ltC.+l, ri},d.), 1 < j < t, are in A This definition is equivalent to the definition of a "crown" in [10]. A t-ring is a circular dependence of alternating send and receive events; an example of a tring is illustrated in Figure 4. As the shaded arcs in Figure 4 show, each receive event depends on the preceding send event and each send event depends on the corresponding receive event. Thus, without an available buffer, there is a circular dependency that results in the system deadlocking. Pi
Pi
Pi.
Pi
Figure 4: Dependency cycle in G(S).
Therefore, a system 5* whose communication graph G(S] contains a t-ring will deadlock unless one of the processes in the t-ring has an available buffer. Since G(S) must be a DAG, no cycle will ever occur in G(S). Figure 2 shows an example of a 2-ring. If G(S) contains a t-ring, then we say that system S also contains a t-ring. In this paper we use G(S) rather than the dependency graph because the order in which received messages are allocated to buffers depends on the execution history. In order to
84
A. Brodsky et al. I The Complexity of Buffer Allocation
model the execution of a system we define a colouring game that simulates the execution of the system with respect to G(S). 3.2
Colouring the Communication Graph
Given a communication graph G(S), an execution of a corresponding system 5 is represented by a colouring game where the goal is to colour all vertices green; a green vertex corresponds to the completion of an event. We use three colours to denote the state of each event in the system: a red vertex indicates that the corresponding event has not started, a yellow vertex indicates that the corresponding event has started but not completed, and a green vertex indicates that the corresponding event has completed. Hence, a red vertex must first be coloured yellow before it can be coloured green; this corresponds to a traffic lights changing from red, to yellow, to green '. To represent buffer allocations we use tokens. For each process with a number of allocated buffers, we associate an equal size pool of tokens with the corresponding process component. To represent a buffer allocation, tokens are removed from the process component's token pool and placed on the receive vertices. The colouring game represents an execution via the following rules. Initially, the start vertices of G(S) are coloured green and all remaining vertices are coloured red; this is called the initial colouring. 1. A red send vertex may be coloured yellow if the preceding vertex is green (i.e., the send is ready). 2. A red receive vertex may be coloured yellow if the corresponding send vertex is yellow, and (a) the preceding vertex (in the same process component) is green (i.e., both the send and the receive are ready), or (b) a token from the corresponding buffer pool is moved on to the vertex (i.e., the send is ready and a buffer is available). 3. A yellow send vertex may be coloured green if the corresponding receive vertex is coloured yellow (i.e., the communication has completed from the senders perspective). 4. A yellow receive vertex may be coloured green if both of its preceding vertices are green. If the vertex has a token, the token is returned to the process component's pool (i.e., a receive completes once the send completes and the preceding computation completes). 5. A red end vertex may be coloured yellow if the preceding vertex is green. 6. A yellow end vertex may be coloured green if the preceding vertex is green. Buffer utilization is represented by placing a token from the token pool of the process component on the selected vertex, and colouring it yellow. If no tokens are available, the rule cannot be invoked. A valid colouring of G, denoted XG> is a colour assignment to all vertices that can be obtained by repeatedly applying the colouring rules, starting from the initial colouring. A colouring sequence S = (x1, X 2 , •••) is a sequence of valid colourings such that each colouring is derived from the preceding one by a single application of one of the colouring rules. An execution of a multiprocess system S with buffer assignment B is represented by a colouring sequence on G(S). We say that a colouring sequence completes if and only if the last colouring in the sequence comprises only green vertices. A colouring sequence deadlocks if and only if the last colouring in the sequence has one or more non-green vertices and the sequence 'Naturally, we refer to a European traffic light.
A. Brodsky et al. / The Complexity of Buffer Allocation
85
cannot be extended via the application of the colouring rules. Furthermore, each transition, from one colouring to the next, within a colouring sequence, corresponds to a change of state of an event in the corresponding execution. Assuming that all events in the system are ordered, there is a correspondence between the colouring sequences on G(S) and the executions of system S. A system S is safe if and only if every colouring sequence on the graph G(S) completes. Furthermore, sends in S never block if and only if every partial colouring sequence on G(S) that ends with a colouring containing a yellow send vertex and a corresponding red receive vertex can be extended by applying rule 2 to the red receive vertex. The choice of when to apply rule 2b affects future choices. For example, in Figure 3, applying the rule to the receive vertex corresponding to the send from process p\_, before the send vertex in process p$ is coloured green, results in a deadlocked colouring sequence.
4 The Buffer Allocation Problem In order to prevent deadlock in distributed applications the underlying system needs to allocate a sufficient number of buffers. Ideally, it should be the minimum number required. Unfortunately, the corresponding decision problem, BAP is intractable: given a communication graph G(S) and a positive integer A;, determine whether there exists a buffer assignment of at most k buffers such that S is safe. We show that BAP is NP-hard by a reduction of the well known 3-SAT problem[17] to BAP. Recall the definition of 3-SAT: determine if there exists a satisfying assignment to A"=i («; V 6j V c,), where a^, 6j, and q are boolean literals in { x \ _ , x i , x - 2 , x2, • . • , xn, xn}. We first need the following lemma. Lemma 4.1 (The t-Ring Lemma) Let G be a communication graph comprising a single tring. No colouring sequence on G can complete without invoking rule 2b at least once. Proof: Assume by contradiction that there exists a complete colouring sequence E that does not make use of rule 2b. Consider the first colouring in S where one of the sending vertices is green; call the vertex Sj. Let r,- be the corresponding receive vertex. By rule 3, the vertex rj must be yellow. Since rule 2b has not been applied, rale 2a must have been invoked earlier in the sequence. By the definition of a t-ring, the send vertex Sj must be the predecessor of TJ. Since the rule 2a was applied to r,, Sj must be green. Hence, there is an earlier colouring in S with a green send vertex. This is a contradiction. • Theorem 4.2 The Buffer Allocation Problem (BAP) is NP-hard. Proof: We prove this by reduction of 3-SAT to BAP. For any 3-SAT instance F we construct a corresponding system S and the corresponding communication graph G(S), both which are polynomial in the size of F. The system has 2n + 1 processes, where n is the number of variables. Each process contains c + 1 epochs where c is the number of clauses in F. An epoch is a consecutive sequence of one or more events in a process. An epoch terminates on a send to a barrier process, or when the process terminates. An epoch begins on a receive from a barrier process or when the process starts. A barrier process is used to synchronize all processes at the end of its epoch. Each process performs a send to the barrier process at the end of their epoch, and waits for a response from the barrier process. The barrier process sends the response to every process only after it has received a message from each process. An epoch of a process component is correspondingly defined. Epochs are used to prevent unwanted interaction between processes. For each literal Xi and Xj, let the system S contain processes pXi and pSi. In addition, let Pbarrier denote the barrier process in S.
86
A. Brodsky et al. I The Complexity of Buffer Allocation
Epoch 0 is used to fix a buffer assignment corresponding to a variable assignment in 3SAT. In epoch 0 add a 2-ring between processes plt and pit. This corresponds to fixing an assignment, because by Lemma 4.1, we have to assign a buffer to either plt or pit to prevent deadlock (see Figure 5). Next, every process plt and p*., 1 < i < n, performs a send to process pbarrier- After ^barrier receives from all processes, it performs a send to all processes, allowing them to proceed into the next epoch. P_X, x,,0
PXj s x,,0 Xi ,o
Figure 5: The choice widget.
The jth epoch of each process corresponds to the jth clause of F and is a 3-ring on the processes paj, Pb}, and pc. that correspond to the literals a,-, &,, and Cj (see Figure 6). By Lemma 4.1, in order to avoid deadlock, at least one of the three processes, paj, p^, or po, must have a buffer. Finally, at the end of the epoch, all processes perform a send to the process ^barrier and wait for a reply. This is formalized in the following Lemma.
Figure 6: The clause widget.
Lemma 43 There exists a token assignment of size n such that all colouring sequences on G(S) complete if and only if formula F is satisfiable. Proof: Let sXi$, s^. 0, rXi$, and rii-0 be the send and receive vertices in epoch 0. By Lemma 4.1, to colour vertices sli-0 and siii0 green, a free token must be available at either process component pXi or p£i. Hence, we need at least n tokens. Note, at the end of each epoch, each token is released back into its token pool. As long as all vertices in each epoch of each process component can be coloured green, the barrier component can also be coloured green. A colouring sequence will only deadlock in the barrier if the corresponding send vertex in one of the process components can not be coloured yellow. If F has a satisfying assignment, then at least one literal in every clause will be true. A corresponding token assignment will ensure that each 3-ring has at least one process component with a token (one corresponding to a true literal). Hence, by Lemma 4.1 none of colouring sequences will deadlock on any of the the 3-rings. Hence, any colouring sequence on G(S) will complete. If F does not have a satisfying assignment, then for any assignment there exists at least one clause comprising false literals. The corresponding token assignment will not assign any tokens to the process components in the corresponding 3-ring. Thus, by Lemma 4.1 all colouring sequences will deadlock in that 3-ring. Further, none of the colouring sequences on G(S) will complete. Hence, any colouring sequence on G(S) will complete if and only if the corresponding assignment satisfies F. •
A. Brodsky et al. / The Complexity of Buffer Allocation
87
Hence, there exists a buffer assignment of size n such that S is safe if and only if F is satisfiable. Thus, BAP is NP-hard. • Theorem 4.4 The Buffer Allocation Problem (BAP) is in E2P. Proof: By Theorem 5.1, verifying that a token assignment is sufficient to prevent deadlock (BSP) is coNP-complete. Since we can non-deterministically guess a sufficient token assignment, the result follows. • The Buffer Allocation Problem remains intractable for systems with send buffers only, and for systems with a combination of both send and receive buffers. In the latter case, the problem remains in E2P because the class of systems with receive buffers only is a subclass of systems with both receive and send buffers. In the former case, we conjecture that the problem is NP-complete. The NP-hardness follows from the observation that each t-ring in the system has to have a buffer assigned to one of its processes in order for the system to progress. It does not matter if it is a send or a receive buffer. Hence, the reduction used in Theorem 4.2 can be applied with no modification.
5
The Buffer Sufficiency Problem
We now turn our attention to the possibly simpler problem of verifying whether a given buffer assignment is sufficient to prevent deadlock. However, this turns out to be an intractable problem as well; we show that BSP is coNP-complete by a reduction from the TAUTOLOGY problem [18, Page 261] to BSP. Given an instance of a formula in disjunctive normal form (DNF), V"=o Aj'=o Lij where L ZJ e [x\, x\,..., xn, xn}, the formula is a tautology if it is true on all assignments. An assignment for which the formula is not true is a concise proof that the formula is not a tautology. Theorem 5.1 The Buffer Sufficiency Problem (BSP) is coNP-complete. Proof: We first observe that if S is unsafe, then there exists a concise certificate of this fact comprising a colouring sequence on G(S) that does not complete. Since the size of a colouring sequence is at most twice the number of vertices in G(S), the certificate is polynomial in size and hence, BSP is in coNP. Let F be a DNF formula with t terms where the ith term has li literals. For any formula F, we construct a system S and show that there exists a deadlocking colouring sequence on G(S) if and only if the formula is not a tautology. We represent the disjunction of t terms by a subsystem containing a t-ring on t processes where each term is a subsystem consisting of a single process, called a 'term' process. The communication graphs corresponding to the term and the disjunction are illustrated in Figures 7 and 8 respectively. Following the t-ring, one 'term' process performs a send - called a 'done' send - to signal that the t-ring did not deadlock. Finally, the ith 'term' process performs /, receives, corresponding to the literals of the ith term; the latter we call 'literal' receives. The first receive in the 'term' process is called a 't-ring' receive. When a message to a 'term' process arrives before any send within the t-ring begins, the message is buffered, using up the one available buffer and preventing the process from buffering any additional messages until it leaves the t-ring. This corresponds to the falsification of the corresponding literal and term. To represent a variable assignment, we use a select subsystem consisting of three processes pXi, p±i, and qi\ the first two processes correspond to the truth values of xt and are called 'select' processes. The third process, called the 'arbiter', fixes the truth value of x,.
A. Brodsky et a/. / The Complexity of Buffer Allocation
1
1
1
1
•u •u done send Figure 7: Disjunction subsystem.
cycle send
Figure 8: Each term is a process.
Processes pXi and pXi each perform a send to the third 'arbiter' process, but the 'arbiter' process can receive neither message until it receives a 'done' message. Since the 'arbiter' has only one buffer, only one of the sends from the processes pXt and px> will be buffered; the other send will cause the sending process to block until the 'done' message arrives.
done recv -»•< -^tOXi
Figure 9: Select subsystem.
The process that did not block performs a nonblocking send, via a disperser (described below), to each 'term' process whose corresponding term contains the complement of the fixed variable (i.e., 'select' process pXi sends a message to all 'term' processes whose corresponding terms contain the literal Zj). Once the 'arbiter' process receives the 'done' signal, all three processes, pXi,pXi, and
A. Brodsky et al. / The Complexity of Buffer Allocation 0
0
0
0
0
disperse. receive
S, 82 83 Sd Figure 10: Disperser subsystem.
tion subsystem performs the 'done' send to a disperser, which broadcasts it to the select subsystems. Finally, each select subsystem broadcasts its selection, via a disperser, to the corresponding 'literal' receives of the 'term' processes; Figure 11 depicts the corresponding communication graph G(S) of a composition. Intuitively, if each of the 'term' processes receive a message from a select subsystem sent via a disperser - before any of the 'term' processes initiate a send event, the buffers on each of the 'term' processes will be used up. By Lemma 4.1 this will prevent the disjunction subsystem from advancing past the t-ring. If the formula cannot be falsified, then at least one 'term' process will not receive any messages before the disjunction subsystem advances past the t-ring. This is formalized in the following lemma. Lemma 5.2 There exists a deadlocking colouring sequence on the communication graph G(S) if and only if the formula F is not a tautology. Proof: If F is not a tautology, then there exists an assignment for which every term in the disjunction is false. In this case we show that a deadlocking colouring sequence exists. Let vXifl and vXif) be the start vertices of process components pXi and pXi respectively. Similarly, let the send vertices sXiti and sXiti be adjacent to the receive vertices rqitXi and rg.tXi in the process component <&, and let sXi$ and s£i)2 be the send vertices of the arcs that are incident on the disperser components. Let Z C V be the set of start vertices corresponding to the falsifying assignment of F; Z will contain one of vXi$ or vXito for each i. First, let EZ be the longest colouring sequence constructed by applying the colouring rules only to vertices that have ancestors in Z. Since each <& has one free token, for every vXi$ € Z or VX^Q 6 Z, vertex rqitXi (respectively rqi,Xi), will be assigned a token and coloured yellow via rule 2b; later in the sequence the vertices sXi$ (respectively 5^,2) can be coloured yellow and then green. The corresponding receive vertex in the disperser component can thus be coloured green. Since the dispersers are adjacent to the 'literal' receive vertices in the term process components, the corresponding send vertex of each of the receive vertices can be coloured yellow, implying that rule 2b can be applied to the 'literal' receive vertices. Thus, if Z contains vXito or vXi)o, and a term process component contains a 'literal' receive vertex corresponding to Xi (respectively x^), then the token in the corresponding process component will be used up. Second, extend E^ by allowing all valid colourings. To avoid deadlock, Lemma 4.1 requires at least one of the term process components in the disjunction to have a free token. Since F is not a tautology and we used a falsifying assignment, none of the process components has a free token, and the colouring sequence will deadlock. If F is a tautology, then regardless of the assignment, one of the terms will be true (i.e., all the literals in the term will be true). Hence, at least one term process component will have no 'literal' receive vertices that are descendants of Z, and will have a token available
A. Brodsky et al. I The Complexity of Buffer Allocation
90
select subsystems,
> (>
disjunction ~~~--^' N \^ r-, ^
>—
-** \
( 1
-M 1
4
i—
-fcH
, 1
'
4
y. K
V/ \>^
disperse^ / P ^'
hv
^/k\ ^V^ ^l \^\ ^^K i > ^^ ^^K
^ ^ >, +A \l ^ i• <1 i
•
4
Figure 11: The composite widget for the formula (xi h x2 /\ x3) V (xi A x3) v (xi hx?).
to prevent the colouring sequence from deadlocking on the t-ring. Observe that the selection component allows at most one of r9jil. or r,,.^ to be coloured yellow, via rule 2b. Provided that the 'done' receive vertex of process component q, is not coloured green, at least one term component will not have any 'literal' receive vertices to which rule 2b can be applied. Thus, the token belonging to that process component will be available to prevent the colouring sequence from deadlocking on the t-ring. On the other hand, if the 'done' receive vertex is coloured green, then the 'done' send vertex in the term component is coloured green. This means that at least one 't-ring' receive vertex is coloured green, and hence the colouring sequence will not deadlock in the t-ring. Once the 'done' send vertex is coloured green, the colouring sequence can always complete. Hence, the system S is safe. • Hence, system S is unsafe if and only if the formula is not a tautology. Since BSP is in coNP and is coNP-hard via a reduction from TAUTOLOGY, BSP is coNP-complete. • This result also holds for systems that use a combination of both send and receive buffers. The coNP-hardness follows from the fact that systems with receive buffers only are also systems that use combinations of send and receive buffers. Since a colouring sequence also serves as a deadlock certificate for combination systems, the coNP-completeness result follows. In the case of systems with send buffers, we conjecture that the corresponding BSP is in P. Unlike communication in systems with receive buffers only, the order of the sends implies an order on the allocation of buffers. Hence, we believe that the computation of sufficiency is similar to the nonblocking buffer allocation problem and hence is in P.
A. Brodsky et al. / The Complexity of Buffer Allocation
6
91
The Nonblocking Buffer Allocation Problem
We now turn to the last of the three problems we consider in this paper. In addition to the system being safe, we can require that no sending process ever blocks due to insufficient buffers on the receiving process. The Nonblocking Buffer Allocation Problem (NBAP) is to determine the minimum number of buffers needed to achieve nonblocking sends. Given G(S), the following algorithm computes the number of buffers needed to assure that none of the sends will ever block. Given two vertices, Vi_c+k and v^c, in G(S), k > 0, vertex vif+k is communication dependent on vertex v^c if vif is the start vertex or if there exist a vertex Vj^ j / i such that there exists a path from vi
r P
i
j
v
i.c+k
Figure 12: t>i.c+j.. is communication dependent on Vi
1. For each receive vertex vitC determine its terminal communication dependency, vertex v^t, where t < c. 2. Set IiiC = [t, c] to be the interval between vertex v^t and vertex vif. 3. For each process component Gi(S), compute 6j, the maximum overlap over all intervals 7i]C. 4. B — {&!,&2 ; ..., bn} is the optimal nonblocking buffer assignment.
Figure 13: Algorithm for computing an optimal nonblocking buffer assignment.
The time between when a message can arrive and when it is received at receive vertex vijC is represented by the interval /jiC. Each interval must have a buffer to ensure nonblocking sends. Hence, the minimum number of buffers, bi, is the maximum overlap over all intervals of a process component Gi(S}. Computing the terminal communication dependencies for G(S) can be done via dynamic programming in 0(| V|ra) time, where V is the vertex set of G(S) and n is the number of processes. If there exists a path from vertex vif to Vjtd, then there exists a path from v^c to all vertices Vjtd+k, k > 0. Associate with each vertex vic to Vj^, and thus to Vj^+k, k > Q. The vector a,)C is computed by taking the elementwise minimums over the vectors of the adjacent vertices VitC', this is simply a depth first traversal of G(S). Since the number of arcs is at most 3| V /2 and the pairwise comparison takes n steps, the traversal takes O(| V|n) time. Next, computing the O(|V|) intervals, /,,c, requires one table lookup per interval. To compute the maximum overlap we sort the intervals and perform a sweep, keeping track of
92
A. Brodsky etaJ./The Complexity of Buffer Allocation
the current and maximum overlap; this takes O(\V\ log \V\) time. Thus, the total complexity is O(|V|n + |V| log |V|) time. In the worst case where n « |V|, this algorithm is quadratic. However, in practice n is usuallyfixed,in which case the | V| log | V| term dominates. 6.1
Proof of Correctness of the Nonlocking Buffer Allocation Algorithm
In terms of the colouring game, a system 5 will not block on any send if for any valid colouring on G(S) containing a yellow send vertex 5 and a corresponding red receive vertex r, vertex r can be coloured yellow by applying rule 2b. This corresponds to guaranteeing buffer availability for every send in the system. Lemma 6.1 Given a multiprocess system S, let G(S) be the corresponding communication graph. For all vertices Vi,s,Vj i € G(S), ifv},t is a send vertex and there exists a path from the vertex v ZiS to vertex Vj,t, then vertex Vj_t cannot be coloured yellow until vertex i>,.s is coloured green. Proof: By rule 1, the predecessor of Vj,t must first be coloured green before Vj.t can be coloured yellow. Since rules 3 and 4 imply that the predecessors of a green vertex must be green, the result follows. • Corollary 6.2 Let S, G(S), vifS, and vjit be as in Lemma 6.1 and let Vj,r be the receive vertex corresponding to the send vertex Vj^ Rule 2b will never be applied to vertex vt> before vertex vlfS is coloured green. The preceding corollary implies that a buffer for the receive event corresponding to vertex Vi,r need not be available until the completion of the send event corresponding to the vertex, on which v,,T is terminally communication dependent. Hence, it is sufficient to allocate the buffer just before the completion of the respective send event. Finally, we argue that this is also necessary. Theorem 63 Given S and G(S), let v^s be a send vertex and vir be a receive vertex that is terminally communication dependent on vertex Vi,s. A token for the application of Rule 2b on vertex v^r must be available before vertex t\.s is coloured green. Proof: Let v,-.t be the send vertex corresponding to the receive vertex v,-.r and let Q = {VI,Q | s < q < r} be the set of vertices that are predecessors of viir, but on which t\.r is not communication dependent. Since v,-ir is not communication dependent on the vertices in Q, we can construct a colouring sequence on G(S) that fixes the vertices in Q to be red, and colours vertex v^t yellow, making the application of rule 2b possible in the next step. Since no progress is made in the zth process component after colouring vertex v^8 green, the state of the associated token pool does not change until the application of rule 2b to vertex v^r. Hence, when vertex v,_s is coloured green, the token pool must have a token destined for vertex v,-.r. • Thus, if receive event r is terminally communication dependent on send event s, then it is necessary and sufficient that a buffer to be used for receive event r must be allocated before the send event s completes. The start event may be thought of as a special send event. Since a buffer is required for each receive over a corresponding interval, computing the maximum overlap of intervals yields the number of buffers required.
93
A. Brodsky et al. / The Complexity of Buffer Allocation
6.2
Other Models
For systems with only send buffers the problem remains in P. The problem can be solved by first reversing all arcs in the communication graph, swapping the start and end vertices, and then running the algorithm described above. For systems with both send and receive buffers, we conjecture that the nonblocking buffer allocation problem is NP-hard. This follows from the observation that we have a choice of either allocating a buffer on the sending or the receiving side, each time a buffer is needed. 6.3 Approximating BAP with NBAP The NBAP algorithm may be useful for determining a buffer assignment that prevents deadlock (BAP). Since a non-blocking execution is guaranteed not to deadlock, any buffer assignment determined by the NBAP algorithm ensures a safe execution. However, the buffer assignment may be far from optimal. A simple example of this phenomena is a two process producer-consumer system comprising of n messages sent by the producer and received by the consumer in the respective order. Such a system requires zero buffers to execute safely, but requires n buffers to execute without blocking. Thus, the aforementioned buffer assignment may entail infinitely more buffers than required. 6.4
Implementation of the NBAP algorithm in Millipede
We implemented the NBAP algorithm and added it to the Millipede parallel debugging system, which is a a multi level parallel debugger for message passing programs [19, 20, 21]. Millipede logs all messages between processes in a parallel system; these message histories are then used to analyze program execution and locate bugs. Determining the number of buffers required for block free execution is one such analysis. To demonstrate the NBAP algorithm we ran Millipede on a program that implements the pipe-and-roll parallel matrix multiplication algorithm [22]. The program has one control process and a number of worker processes arranged in a 2 dimensional mesh. We ran the NBAP algorithm on meshes of size 2 x 2, 3 x 3 and 4 x 4 . The communication graph for the smallest example, comprising four workers ordered in a 2 x 2 mesh is depicted in Figure 14. The corresponding optimal buffer assignment is, listed in the second column of table 1. Proc.
Max overlap
0 1 2 3 4
4 3 3 3 3
Overlap for intervals Ij 0 2 3 3 2
0 1 2 2 1
0 2 1 1 2
0 3 2 2
4 2 1 1 7
3 1 1 1 1
2 1 1 1 *
1 0 0 0 0
0 0 0 0 0
Table 1: The result of NBAP algorithm on the 2 x 2 example,
In this example process 0 is the control process and processes 1 through 4 are the workers. The control process needs 4 buffers and the workers each need 3 to execute without blocking. The results obtained when executing the NBAP algorithm on a 3 x 3 worker system is 9 buffers for the control process and between 4 and 5 buffers for the worker processes. For the 4 x 4 system the numbers are 16 for the control process and between 5 and 7 buffers for the workers.
94
A. Brodsky et al. I The Complexity of Buffer Allocation
Figure 14: The communication system for a 2 x 2 worker process mesh.
7 Conclusion As more and more functionality of message passing systems is off-loaded to the network interface card, limited buffer space becomes an increasingly important issue. Hence, the problem of determining fc-safety plays an increasingly important role. Unfortunately this problem is intractable. We have shown that in the receive buffer model, determining the number of buffers needed to assure safe execution of a program is NP-hard, and that even verifying whether a number of assigned buffers is sufficient is coNP-complete. On the positive side, if we require that no send blocks, we provide a polynomial time algorithm for computing the minimal number of buffers. By allocating this number of buffers, safe execution is guaranteed! There are several strategies that a programmer can use to reduce the likelihood of deadlock when only a few buffers are available. To decrease the risk of deadlock the programmer can introduce epochs that are separated by barrier synchronizations. If each epoch only needs a small number of buffers, the risk of deadlock due to buffer insufficiency is reduced. For systems with only send buffers we conjecture mat the Buffer Sufficiency Problem can be solved in polynomial time because the order of the sends in each process is fixed. This would imply that the buffer allocation problem for systems with only send buffers is NPcomplete. For systems with both send and receive buffers we conjecture that the nonblocking buffer allocation problem is NP-hard. Unlike in the other two models, a buffer can be assigned either on the send side or on the receive side, dramatically increasing the size of the search space. The results (conjectures) are summarized below. Problem || Receive Buffers Send Buffers Send/Receive Buffers NP-hard NP-hard NP-hard BAP coNP-complete coNP-complete BSP (P) P P (NP-hard) NBAP
A. Brodsky et al. / The Complexity of Buffer Allocation
95
References [1] C. Keppitiyagama and A. Wagner. Asynchronous MPI messaging on myrinet. In Proceedings 15th International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2001. [2] D. Frye, R. Bryant, H. Ho, R. Lawrence, and M. Snir. An external user interface for scalable parallel systems. Technical report, IBM highly parallel supercomputing systems laboratory, November 1992. [3] J. Dongarra, R. Hempel, A. Hey, and D. Walker. A proposal for a user-level, message-passing interface in a distributed memory environment. Technical Report TM-12231, ORNL, June 1993. [4] J. Dongarra. MPI: A message passing interface standard. The International Journal of Supercomputers and High Performance Computing, 8:165-184, 1994. [5J J. Bruck, D. Dolev, C. Ho, M. Rosu, and R. Strong. Efficient Message Passing Interface (MPI) for Parallel Computing on Clusters of Workstations. In 7th Annual ACM Symposium on Parallel Algorithms and Architectures, pages 64 - 73, Santa Barbara, California, July 1995. [6] G. Burns and R. Daoud. Robust MPI Message Delivery with Guaranteed Resources. MPI Developers Conference at the University of Notre Dame, June 1995. [7] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam. PVM: Parallel Virtual Machine: A Users' Guide and Tutorial for Networked Parallel Computing. Scientific and engineering computation. MIT Press, 1994. [8] R. Cypher and E. Leu. Repeatable and portable message-passing programs. In Proc. of The Symposium on the Principles of Distributed Computing (PODC), pages 22-31, 1994. [9] R. Cypher and E. Leu. The semantics of blocking and nonblocking send and receive primitives. In Proceedings of 8th IEEE International parallel processing symposium (IPPS), pages 729-735, 1994. [10] B. Charron-Bost, F. Mattern, and G. Tel. Synchronous, asynchronous, and causally ordered communication. Journal of Distributed Computing, 9(4): 173-191, 1996. [11] L. Lamport. Time, clocks and the orderings of events in a distributed system. Communications of the ACM, 21:558-565, 1978. [12] K. Jensen. Coloured Petri nets. Basic Concepts, Analysis Methods and Practical use, volume 1. Springer, 1992. [13] P. Huber, A. M. Jensen, L. O. Jepsen, and K. Jensen. Reachability trees for hihg-level Petri nets. Theoretical Computer Science, 45:261-292, 1985. [14] V. Anantharm. The optimal buffer allocation problem. IEEE Transactions on Information Theory, 35(4):72l-725, 1989. [15] M. Retman. The optimal buffer allocation problem in light traffic. In IEEE Conference on Decision and Control, 1987. [16] T. Sheskin. Allocation of interstage storage along an automatic production line. AIEE Transactions, 8(1), 1975. [17] S. A. Cook. The complexity of theorem-proving procedures. In Proceedings of the 3rd Annual ACM Symposium on the Theory of Computing, pages 151-158, 1971. [18] M. R. Garey and D. S. Johnson. Computers andlntractibility: A Guide to the Theory ofNP-Completeness. W. H. Freeman and Company, New York. 1979. [19] J. Baekgaard Pedersen and A. Wagner. Sequential Debugging of Parallel Programs. In Proceedings of the international conference on communications in computing, CIC'2000. CSREA Press, June 2000. [20] J. Baekgaard Pedersen and A. Wagner. Correcting errors in message passing systems. In Frank Mueller, editor, High-Level Parallel Programming Models and Sypportive Environments, 6th international workshop, HIPS 2001 San Francisco, CA, USA, volume 2026 of Lecture Notes in Computer Science, pages 122-137. Springer Verlag, April 2001.
%
A. Brodsky et al. I The Complexity of Buffer Allocation
[21] J. Pedersen and A. Wagner. Protocol Verification in Millipede. In Communicating Process Architectures 2001. IOS Press, September 2001. [22] G. Fox, M. Johnson, G. Lyzenga, S. Otto J. Salmon, and D. Walker. Solving problems on concurrent processors. General techniques and regular problems, volume 1. Prentice-Hall, Inc., 1988.
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
97
A Structured Distributed Shared Memory System Kei Simon PEDERSEN and Brian VINTER Department of Computer Science, University of Southern Denmark Abstract. The architecture and performance of a Java implementation of a structured distributed shared memory system, PastSet, is described. The PastSet abstraction allows programmers to write applications that run efficiently on different architectures, from clusters to widely distributed systems. PastSet is a tuple-based three-dimensional structured distributed shared memory system, which provides the programmer with operations to perform causally ordered reads and writes of tuples to a virtual structured memory space called the PastSet. It has been shown that the original, native code, PastSet was able to outperform MPI and PVM when running real applications and we show that the design translates into Java so that Java PastSet is a qualified competitor to other cluster application programming interfaces for Java.
1
Introduction
Structured distributed shared memory, structured DSM, was first introduced in Linda [1], but the interest into structured DSM has since been low. The general consensus is that it is not possible to achieve performance comparable to that of simpler systems. We started the work on PastSet with the idea that bad performance is not inherent to the concept of structured DSM, and that good performance can be achieved by paying close attention to design and implementation. The increasing popularity of Java have also spawned an interest in 'easy-to-use' application programming interfaces, APIs, for parallel and distributed applications, which in turn has sparked a renewed interest in tuple space based DSM since this model is truly simple to use. The result of various ventures into tuple space has resulted in a variety of Linda-like systems and two major, widespread, systems; JavaSpaces[2] and TSpaces[3]. JavaSpaces is the result of a collaboration between the original Linda team and SUN, while TSpaces is a product from IBM that seeks to address some of the semantic shortcomings found in JavaSpaces. The Java PastSet, JPS, project seems like an obvious experiment based on the high performance we achieved with the original, native code, PastSet, and one of the primary goals of the JPS system were to test if the performance of PastSet were due to correct design or simply a result of very good programming. A secondary goal were naturally to provide yet-another tuple space DSM system for Java, hopefully far more efficient than the existing ones. The remainder of this paper is organized as follows; in section 2 we describe the PastSet model in some detail, and section 3 explains the design of JPS. In section 4 we report on the basic performance of the JPS operations and in section 5 we show the application level performance of JPS. Finally we draw our conclusions in section 6.
98
2
K. Pedersen and B. Vinter / Java PastSet
PastSet
PastSet was first introduced as an interprocess communication paradigm in [4]. The paradigm resembles that of Linda[l], but with some significant differences. Tuples are generated dynamically based on tuple templates that may also be generated dynamically. A tuple template is an ordered set of types. Tuples based on a particular template has an ordered set of variables matching the types in the template. Each type in a tuple template has an associated value-space, or dimension, describing the set of all possible values for that type in this template. Taken together, the types in a template spawns a space encompassing all conceivable type-value combinations for tuples based on that template. A tuple with all singular values represents a singular point in this space. As with Linda, PastSet supports writing (called move) tuples into tuple-space and reading (called observe) tuples that reside in tuple space. Contrary to Linda's in operation, PastSet observe does not remove tuples from tuple space, the tuple that is observed is marked as observed but remains in the PastSet so that it can be read again if specified so directly. No mechanism is provided to remove individual tuples from PastSet. All PastSet operations return only when the operation has completed, or an error has been detected, no asynchronous calls exist. In PastSet, each set of tuples based on identical templates is denoted an element of PastSet. An element may be seen as representing a trace of interprocess communications in the multidimensional space spawned by the tuple template. PastSet preserves the causal order among all operations on tuples based on the same or identical templates. There is no ordering between tuples of different elements. Tuples that match the same type, but which the programmer does not wish to place in the same element can be differentiated by an initial flag. In effect, PastSet keeps a causally ordered log of all tuples of the same or identical templates that has existed in the system. This also allows the processes to re-read previously read tuples. It is the intention that the added semantics of PastSet will allow programmers to more easily create parallel programs that are not of the traditional 'bag of tasks' type. Two pointers First and Last are associated with each element in PastSet. First refers to the elements oldest unobserved tuple. Last refers to the tuple most recently moved into the element. A parameter, DeltaValue, associated with each element in PastSet defines the maximum number of tuples allowed between First and Last for that element. A process may change DeltaValue at any time. The move and observe operators update First and Last, and obey the restrictions imposed by DeltaValue for each element in PastSet. Functionality is provided to truncate PastSet on a per element basis, permanently removing all tuples that are older than a given tuple. To address some of the performance problems for DSM systems, we have introduced the concept of "User Redefinable Memory Semantics", URMS[5], into PastSet. The principle behind URMS is to offer users the opportunity to redefine the semantics of any or all memory operations for memory areas that are specified by the user. The redefined semantics are specified by providing code that should be executed instead of the memory operation. The specification of the redefinition may also include initialization code that is applied once to initialize the specified memory area. In effect, the redefined memory operation semantics will be applied for memory operations on the specified areas only. For example, a memory location may be redefined to accumulate a global sum of partial sums that are produced by independent processes. For that location, the Move operation would be redefined accordingly to code that stores the aggregate sum, and keeps track of a completion criterion that may be realized as an access count, process list, or by other means. The Observe operation is redefined to return the aggregate sum only after the reduction has
K. Pedersen and B. Vinter/Java PastSet
Application
Application
JPS Client
JPS Client
JPS Kernel
Figure 1: Layout of a node that supports PastSet
completed, in effect blocking until the termination criterion used for the Move operation is satisfied. This approach makes it very simple for programmers to overlap communication and calculation if there is any work that may be done between the partial sum is ready and the global sum is needed.
3 Java PastSet 3.1 Architecture The Java version of PastSet is focused on achieving high performance, while preserving as much of the original PastSet semantics in an OO-environment. To minimize overhead the architecture comprises two parallel paths, one that executes PastSet operations that may be completed locally and one for operations that requires remote operations. Since the nature of PastSet focus on the elements of PastSet, the distribution also focus on the elements. Each element exists on one node only and is not replicated. An operation on elements that are located on the same node as the process that performs an operation is called a local operation. Operations on elements that are located on other nodes are called remote operations. Thus PastSet supports remote observes and remote moves, similar to remote read and remote write on other DSM systems. Remote operations are wrapped and communicated to a JPS Server on the remote node using TCP/IP. 3.2 Data Location To achieve low overhead on operations it is imperative that applications can easily determine, at run time, whether an operation should execute locally or remotely, and in the latter case, which node should perform the remote operation. To this end each element descriptor object includes the identity of the server that holds the element. In the case of a local element the JPS Client calls the PastSet operation directly. For a remote element, the call performs a procedure, which sends the operation to the respective server and then reads and returns the answer. 3.3
The JPS Server
Each node in the cluster that holds parts of PastSet runs a JPS Server that services remotely issued PastSet operations on local elements. As the PastSet operations move and observe are potentially blocking, the server must be able to serve several blocking operations simultaneously. It is easily seen why simply performing one operation at a time and queuing other
100
K. Pedersen and B. Vmter/Java PastSet
operations would allow false deadlocks to occur. This problem is solved by making the IPS Server multi-threaded, so that once a thread blocks, another thread is activated. The JPS Server starts a thread for each remote client who needs access to elements on the local kernel. Each thread is dedicated exclusively to one client and runs until the client is shutdown by the user application.
3.4
User Redefinable Memory Semantics
The URMS functionality in JPS is implemented as an URMS interface which in turn may be implemented as a class that is loaded at run time. This approach is far more elegant than the native-code URMS where any URMS function must be available to the PastSet system at compile-time, and since the native version runs at kernel level it does not allow user-defined URMS functions. In JPS a programmer can easily provide his/her own URMS code.
3.5 Distribution A central part of the distributed PastSet is the distribution of elements. Elements are placed on a server the first time a process enters the dimensions that specify the element. To this end every application that use JPS is connected to a central name-server that keeps track of all existing elements. When an application issues an EnterDimension operation, the JPS Client first contacts the name-server to find out where the element is or should be located, and then it sends the EnterDimension operation to the corresponding server. This way different datadistribution models can easily be implemented by changes in the name-server only. Currently we have implemented three distribution models: Central-Server, Round-Robin Server and First-Touch. The Central-Server model does not distribute data at all, but centralizes PastSet on one machine. This model does not scale well, but serves as a reference for the other models. The Round-Robin Server tries to balance the load of the elements by letting each node hold every n'th element. This way the load on the servers is equally distributed, assuming the elements are used in a uniform way. In the First-Touch model an element is created on the node that runs the process that first enters the element. Under the assumption that not all elements are accessed uniformly by all processes, the programmer can take advantage of the First-Touch policy by having an element placed at the node where the most activity of the element will be initiated, thereby improving both local and global performance by reducing unnecessary network traffic. An in-depth coverage of the distribution models and their impact on application performance may be found in [6].
4 Basic Performance Good performance is vital to all programming paradigms, and cluster programming APIs even more so. To measure the performance of JPS we have chosen to compare to the best known tuple-space system in Java, TSpaces. All experiments are run on a cluster of 16 nodes, each having two 450MHz Pentium-Ill CPUs and 256MB main-memory. The nodes in the cluster are connected via a lOOMb/sec Fast Ethernet connected to a switch.
101
K. Pedersen andB. Vinter/Java PastSet
4.1
Latency
In DSM systems it turns out that the latency of communication, i.e. the time it takes from a package is sent until it is received, is of significance to the task grain size that can be supported efficiently by the system. The higher the communication latency is the coarser the grain size must be for good performance. A lower latency allows finer grained parallelism to perform efficiently. For instance, the Internet project of solving the original RSA cryptography challenge communicated via an e-mail system over a global network. In this case latency was extremely high, but the parallelism was trivial, e.g. no communication required during execution. On a system like a SMP machine where the communication latency is lower than access to main memory, i.e. cache to cache, parallelism can be very fine grained. observe(); //Observe the initial token get_start_time(); //Get timestamp for(i=0; i<1000; i++){ move(); //Send token to right neighbor observe(); //Get token form left neighbor get_stop_time();
Figure 2: Pseudo-code for the ring latency test
To determine the latency of PastSet operations we look at two basic cases, one where two processes passes a token of varying size back and forth, and one where an increasing number of nodes form a ring that passes a token around. The first experiment will be executed in three scenarios. Firstly where the elements used for the communication and the two processes are placed on the same node. Secondly, where one of the processes is on a remote node. Thirdly, where the elements and the processes are all placed at different nodes. Pseudo code for the ring experiment core is shown in figure 2.
+•— x— *— B— »— s—
16
32
JPS Local/Local JPS Local/Remote JPS Remote/Remote TS Local/Local TS Local/Remote TS Remote/Remote
64 128 256 512 1024 2048 4096 8192 1638432768 Tuple Size (bytes)
Figure 3: Round trip latency for two process synchronization.
Figure 3 shows the two way latency for process synchronization using JPS and TSpaces. The most noticeable difference between the two models is the inability of TSpaces to efficiently utilize access to a local tuplespace when synchronizing two processes on the same node. In addition TSpaces is more than an order of magnitude slower than JPS, and only
102
K. Pedersen andB. Vmter/Java PastSet
when the data size grows above 4KB is this difference narrowed from the impact of the network propagation delay. At 32K data size IPS is still a factor 5 faster than TSpaces.
Number at Processors
Figure 4: Latency of passing a package in a process ring.
Repeating the experiment for more than two processes, as shown in figure 4, passing the tuple in a ring amongst the processes, the same picture presents itself. Two points are worth while observing in this experiment; first of all that the latency in TSpaces grows far more rapidly on growing number of processes than IPS. And secondly that TSpaces seems to be more sensitive to the size of the tuples that are passed around than IPS is. 512 bytes 32768 bytes
Number of Processors
Figure 5: Latency of passing a package in a process ring, IPS only.
In figure 5 we have removed the TSpaces performance to investigate the performance of IPS in further depth. The most important conclusions one may draw from this graph is that the passing of tuples between processes on the same node is very efficient. While the tuples are small the overhead almost hides the effect of the underlying hardware architecture, but once we use 32KB tuples we clearly see the staircase that tells us that we have a cluster of dual-processors.
103
K. Pedersen and B. Vinter/ Java PastSct
4.2
Bandwidth
The purpose of testing the communication bandwidth of the PastSet system is to define the limits for writing applications that communicate large amounts of data. Applications that require high bandwidth are frequently found in scientific processing and data-mining. The bandwidth experiments are similar to the latency setup, although somewhat simpler. A process first writes, and then reads varying block sizes. To eliminate noise, and partly to test the behavior when the element grows, each packet size is written and read 1000 times (figure 6). get_start_time ( ) ; for(i=0; i<1000; i++) move ( ) ; //Write data-block to PastSet get_stop_time () ; get_start_time ( ) ; for(i=0; i<1000; i++) observe (); //Read a data-block from PastSet get_stop_time ( ) ,Figure 6: Pseudo code for the bandwidth test.
JPS Remote Mov — JPS Remote Obs -- TS Remote Move — TS Remote Obse
*~
T-
CO
t
Tuple Size (bytes)
Figure 7: Bandwidth of PastSet local and remote operations.
From the latency-figures (figure 7), we would expect to see a significantly better bandwidth with JPS than with TSpaces and we see just that. At tuple-size 1 byte the bandwidth of JPS is 3.4 times better than TSpaces and at 32KB JPS reads a factor 1.9 faster than TSpaces and writes a factor 2.1 faster.
5 Applications To test the performance of JPS in real use we have implemented three applications using JPS. In order to compare the performance to the alternatives we have also implemented the three applications in TSpaces and mpiJava[7]. The applications we have chosen are a virtual windtunnel, a Lattice Gas Automaton, which use only nearest neighbor communication, an n-body
104
K. Pedersen aadB. Vinter/Java PastSet
simulation of stellar bodies, which use all-to-all communication and a Raytracer which uses pipelined communication. 5.1
Lattice Gas Automaton
The Lattice Gas Automaton, LGA, is a complete application that models air-flow in a twodimensional wind tunnel. The flow is modeled as particles rather than using gas dynamics. The model is both simple and efficient, and is to some degree easy to parallelize. The idea is to model the two-dimensional space using several large bit-matrices. The bit-value one indicates the presence of a gas particle moving in a direction associated with each matrix. The bit-value zero indicates the absence of a particle at this particular location and direction represented by the matrix. The simulation runs for a predetermined number of time steps. The simulated space is divided into bands, and a worker is spawned to service each band. Each iteration consists of three distinct phases. First all points in the space are checked to see if more than one particle occupies this space, indicating a collision of two or more particles. A set of collision-rules determines the implications of collisions on the particles involved. This first step accounts for most of the time spent in each iteration. Following the collision-test, all gas particles are moved in the matrices. Finally new particles are injected into the system. Each worker starts an iteration by doing collision-detection, it then moves the particles that should be migrated to another partition, moves the particles inside its own block and finally observes and inserts the particles that are migrated into its block. The LGA application uses fairly small messages' and may run in phase-lock and thus should have only little contention on the messaging systems. Each worker writes it's border values to one given element. This element is the one most frequently used by the process, and because of this the application is written so that the worker is guarantied to be the first to touch that element which is an advantage when the First Touch model is used. JPS TS mpUava
5
10
15 20 Number o» Processors
25
Figure 8: Performance of the LGA application.
The execution times for the LGA application are shown in figure 8. The TSpaces version, which of course differs only in syntax from the JPS version, consistently crashed when running with 32 workers and thus we did not manage to get performance numbers for 32 workers 1
One bit per point length-wise
105
K. Pedersen andB. Vinter/Java PastSet
using TSpaces2. It was the TSpaces server that crashed and not the application. The trendline for TSpaces is clear enough though and it does not scale at all. JPS however achieves very good performance and gets a speed-up of 25 using 32 CPUs. The mpiJava version runs stable enough but the performance of mpiJava in this application is quite disappointing! Using 32 CPUs the mpiJava version achieves a speedup of only 13, a CPU utilization of less than 0.5 where JPS achieves an utilization of 0.8! 5.2
N-Body
The chosen N-Body algorithm is a trivial O(n 2 ) complexity simulation of celestial bodies. The JPS implementation works by dividing the responsibility for the bodies amongst the workers. Since updating one body requires knowledge about all the other bodies the JPS implementation exchanges the complete set of bodies in each iteration. To this end we use an URMS function that assembles the bodies from all workers into one single tuple, much like an MPUVllgather operation. In this way all workers write their local bodies to the PastSet and reads the complete set of bodies. — JPS — TS — mpiJava
0
5
10
15 20 Number of Processors
25
30
35
Figure 9: Performance of the N-Body simulation.
Figure 9 shows that JPS achieves close to linear speedup until 16 CPUs while at 32 CPUs the CPU utilization drops to 0.63. It is obvious that collecting results from 32 workers is too much work for the server, and the application looses performance. To address this we need to incorporate the PATHS [8] work in the Java version of PastSet also. In the TSpaces version however each worker need to write its own bodies to the tuple-space and then read n-1 tuples from the tuple-space, one from each other worker. The mpiJava implementation achieves fair speedup up until 16 CPUs but only gains another speedup of two by adding 16 more CPUs. 5.3 Ray tracer Ray-tracing is another classic target for parallelization which remains popular for two reasons; the need for speedup in ray-tracing is very real; and parallelizing ray tracers is seemingly straightforward. 2 TSpaces also crashed frequently with 16 workers but managed to terminate correctly a few times so that we could get the performance numbers
106
K. Pedersen and B. Vinter/Java PastSet loop:
Search through all objects to find the first one the ray hits If ray hits no objects Return ray Modify ray to match the hit of the object Increase ray-age by one If ray-age > ray-max-age Return ray Goto loop
Figure 10: Pseudo-code for the raytracer
The pseudo-algorithm for tracing one point is very simple, as shown in figure 10. This is then done for each pixel in the picture that should be rendered. The intuitive way to parallelize this is to divide the area that is rendered into blocks or stripes and let each worker render one such block. This seems natural since it requires no communication between the workers and thus in effect becomes an embarrassingly parallel application. There are two pitfalls however. First of all the individual blocks won't require equal time to render and thus the parallel execution may become highly unbalanced. This problem could be partially hidden by applying cyclic-striping instead. The second, and more frequently visited, pitfall is that in the embarrassingly parallel version of ray-tracing each worker needs a copy of all the objects in the model to render the portion of the frame that it is assigned to. All workers then need to search exhaustively through all objects for each pixel they render. If the problem is parallelized by distributing the objects along the workers instead one gets the benefit of improved cache locality which can result in super-linear speed-ups for this particular problem, thus we apply the latter approach in this work. JPS TS mpiJava
Number ol Processore
Figure 11: Performance of the raytracer.
The performance of the raytracer is shown in figure 11. Once again we were unable to correctly execute the TSpaces version using 32 CPUs, again it was the TSpaces server that crashed. However, in this scenario TSpaces performs better that JPS using 2 and 4 CPUs. We are not quite sure why this is, but suspect that the TSpaces operations are so slow that the raytracer can successfully run the garbage-collector during an operation in tuple-space, and this may be an advantage in this application. Once the raytracer is run with eight or
K . Pedersen and B. Vinter/ Java PastSet
107
Figure 12: Result of the raytracer.
more CPUs the TSpaces version levels out while JPS takes off and end up with an amazing speedup of 145 using 32 CPUs! The result is very good, but on par with what one might expect with raytracing. The mpiJava implementation turned out to cause us lots of problems as it crashed at various points in the execution. It turned out that mpiJava has problems when an application uses the dynamic memory model heavily, which the raytracer does. The work around we applied was to call the garbage-collector before every send or receive operation. This is clearly visible in the performance of the mpiJava version of the raytracer, which is only super-linear from 4 through 16 CPUs and has a final speedup of 3 1 using 32 CPUs.
6
Conclusions
Java PastSet, JPS, is an attempt to provide a Java version of PastSet with as few differences as possible. We believe that the design of JPS is sound, and furthermore that the performance we have shown verifies this claim. Compared to the best existing tuple-based DSM system, TSpaces, JPS is as much as two orders of magnitude faster and it provides a communication bandwidth that is 3.4 times faster than TSpaces. At application level JPS consistently outperforms both TSpaces and mpiJava. Java PastSet achieves speed-ups of between 19 and 145 which are quite fair considering that we are working with Java. We believe that JPS provides programmers with all of the convenience of the shared memory paradigm while providing usable performance. References [ 11 N. Carrier0 and D. Gelernter. Linda in context. Commun. ACM, 32(4):444-458, April 1989.
[2] Eric Freeman, Susanne Hupfer, and Ken Arnold. JavaSpaces(TA4) Principles, Patterns and Practice. SUN MicroSystems, 1999. [ 3 1 IBM. http://www.almaden.ibm.com/cs/tspaces/.
108 [4] O. J. Anshus and Tore Larsen. Macroscope: The abstractions of a distributed operating system. Norsk Informatikk Konferanse, October 1992. [5] B. Vinter, Anshus, T. Larsen, and J. Bj0rndalen. Extending the applicability of software dsm by adding user redefinable memory semantics. In Parallel Computing 2001, ParCo2001, September 2001. [6] B. Vinter, Anshus, and T. Larsen. Data distribution models for a structured distributed shared memory system. In International Conference on Parallel and Distributed Processing Techniques and Applications, July 1999. [7] Mark Baker, Bryan Carpenter, Geoffrey Fox, Sung Hoon Ko, and Sang Lim. mpijava: An object-oriented Java interface to mpi. In Presented at the International Workshop on Java for Parallel and Distributed Computing, IPPS/SPDP1999, San Juan, Puerto Rico, April 1999,1999. [8] John Markus Bj0rndalen, Otto Anshus, Tore Larsen, and Brian Vinter. Paths - integrating the principles of method-combination and r emote procedure calls for run-time configuration and tuning of highperformance distributed application. Norsk Informatikk Konferanse, pages 164-175, November 2001.
Communicating Process Architectures 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
109
Synchronous Active Objects Introduce CSP's Primitives in Java Claude PETITPIERRE EPFL, Laboratoire de Teleinformatique, 1015 Lausanne, Switzerland
Abstract. This paper presents a proposal of a language, based on synchronous active objects that introduces the CSP primitives into Java. The proposal does not use channels to realise the inter-process communications, but is shown to offer the same expressive power as the channel based solutions. The paper also shows that the rendezvous defined by CSP or our synchronous objects are very well adapted to the industrial development of event driven applications, handling simultaneously GUIs and accesses to remote applications.
1 Introduction Concurrent programs are simpler if programmed in the right way, and in this regard CSP's rendezvous is an invaluable model that allows a programmer to avoid most problems linked to Java's approach to concurrency. In this paper, we present an approach based on the concept of rendezvous, implemented directly at the level of the method calls without resorting to any kind of channels (neither logical as in CSP, nor physical as in occam). We then discuss the compatibility of our approach with the channel-based approaches and show that it is very relevant to the development of interactive and distributed applications, which make up the bulk of today's developments. 2
Synchronous Active Objects
2.1 The Concept of an Active Object Java's threads are often considered to be orthogonal to objects. In this paper we claim that threads are too delicate to be handled at the application level. They must be tightly bound to objects, from which they should not be distinguished more than the movement of a watch is distinguished from the watch itself. This view defines a new kind of objects, named active objects, that can be contrasted with the passive (i.e., usual) objects. Passive objects are acted upon, whereas active objects may act on their own without being called. The behaviour of active objects is defined in their run method, which is executed on a thread attached to the object when it is instantiated. An active object is conceived as such, and we consider that a method of a passive object cannot be run on a bare thread without complicating an application's structure. We propose thus to define a way of creating active objects (by decorating their class with the keyword active) that does not require programmers to start threads directly. The active keyword tells the compiler that it must start a thread at the end of the execution of the constructor and synchronise all the calls made to its methods as described in the following.
110
C Petitpierre / Synchronous Active Objects
2.2 The Concept of a Synchronous Active Object A communication between two active objects is naturally made with method calls. However, as active objects introduce concurrency, these calls must be synchronized. This synchronization can be observed under two angles. Under the first angle, shown in the following frame, the focus is on the method call: the object on the left calls a method in object b, defined on the right. As b is an active object, this call blocks and is only executed when object b accepts that this specific method be called. // active object b
// active object a run () {
b.myMethodO ;
public void myMethodO { run() {
•accept myMethod;
Under the second angle, the focus is on the communication. As the accept statement in object b is also blocking, and is executed only when a call to method myMethod has been executed, a communication between two synchronous active objects also expresses the rendezvous shown below. The first thread must be at the call statement and the second one at the accept statement for the rendezvous to occur. The method is executed during the rendezvous, while both objects are blocked. // active object b
// active object a
run () {
b.myMethodO ; -4
r
public void myMethodO { run() {
accept myMethod;
}
Depending on the situation, one view or the other is preferred. When one studies the internal functioning of an object, the important fact is that a method has been executed, whereas when one analyses the global behaviour of a multi-active object application, the important aspect is the interleaving of the rendezvous. 2.3 The Select Statement Simple synchronous calls are of course too constraining. We have thus introduced a statement that implements CSP's alternation, allowing a program to wait for as many calls, accepts and delays as needed. We have also introduced a guarded statement, shown in the second case of the select described below. Guards are optional, but they can be added to the three kinds of cases.
C. Petitpierre /Synchronous Active Objects
public void run {) { for (;;) { select { case y = obj .method (x) ,-
when (guard) accept myMethod; waituntil (currentTimeMillis ()+1000) ,-
The first statement of each case must be a (possibly guarded) trigger, namely a call to another active object, an accept, or a waituntil. The cases that have a false guard when the program enters the select are discarded for that execution of the select. The statements placed in the body of a case are executed after the rendezvous that triggers the case has taken place. They may contain passive or active calls as well as new select statements. Our select statement is very close to Ada's. The main difference is that ours may contain any number of synchronous calls, accepts and waituntil. 2.4 n x m Communications Our communication concept does not involve any channel, a method in an object being the sole identification used by a caller to address the receiver. Actually, it is commonly admitted in object-oriented programming that calling a method in an object is equivalent to passing a message to the object. The method thus plays the role of a channel. However, with channels, it is possible to implement an n x m communication pattern (Figure 1, left), whereas simple method calls only implement the nx 1 communication pattern (Figure 1, middle). We have thus devised an extra statement to cope with that aspect.
n x1
Figure 1. Channels versus synchronous calls.
The statement below implements the last pattern of Figure 1, which is equivalent to the first pattern.
112
C. Petitpierre I Synchronous Active Objects
select { cases (i
The cases statement represents a set of N cases. The code is not replicated - the kernel just returns the value of / that corresponds to the selected case and this value is used during the execution of the body of the case. The handling of this kind of communication is a little bit more complex than with the use of a channel, but to pass from the situation n x 1 to the situation n x m, the programmer only needs to replace an object by an array of objects and to indicate how many objects are involved in the call. An advantage over channels is that the receiver automatically gets an identification of the sender (the / value). Note also that all receivers need not be of the same class. They just have to implement the same interface, namely the one that comprises the called method. 2.5 The CSP Primitives SEQ and PAR The CSP primitives SEQ and PAR have not been explicitly introduced in our approach. Sequentiality appears only at the statement level, not at the object level. The sequential execution of two objects must be organized with an explicit synchronization by synchronous calls. As far as the PAR statement is concerned, each active object that is instantiated runs in parallel with the other objects, but the junction of the execution of two objects must be explicitly programmed. 3
Synchronous Calls, Channels and Select Statements
3.1 Buffered Channels A buffered channel can easily be implemented with synchronous calls (as well as with zeroslot channels) and actually, we cannot imagine a better specification of a buffered channel than the following one: a buffered channel is an active object that defines a write method and a read method with the signatures and the scheduling "specified" below: active class Channel { int N; constructor delivers N (the buffer size) List list = new L i s t ( O ) ; // signature void write (Object obj) { l i s t . a d d ( o b j ) ; } Object read () { return list.remove(0); } // scheduling public void run () { for
(;;)
{
select { case when ( ! l i s t . i s E m p t y ( ) ) accept read; case when (list.sizeO < N) accept write,-
C. Petitpierre /Synchronous Active Objects
113
This "specification " clearly identifies the data that can be passed to/from the channel, as well as the conditions under which the read and the write methods are executable. Note that the signature is clearly separated from the scheduling of the operations. In order to create an unlimited channel, one just has to drop the limitation - i.e. the second when. This channel uses a thread, but an optimiser could detect that there are no call statements in the run method and somehow integrate the code executed on the thread at the end of the methods. 3.2 Zero-buffer Channel In this paragraph, we show that a zero-buffer channel (namely, a logical channel, used for identification purposes only) can also be described with synchronous calls, although it is obviously better to stick to the simple method call when developing applications. active class SynchroChannel { Object x = null; void write (Object obj) { x = obj; accept read; Object readf) n{ return x; public void run () { for (;;) accept write;
This implementation is not as clean as the previous one. Firstly, the accept read statement must be placed in method write. Secondly, the write cannot be called from a select statement, because it would be selected even if the rendezvous with the receiver is not ready. This example highlights both the flexibility and the limitations of the approach. 3.3 Characteristics of Synchronous Calls The synchronous method calls actually integrate the read method in the receiver, which allows the programmer to replace the symbol write by significant names and to pass any set of parameters, some of which being specifiers indicating the type, the size or other characteristics of the passed data. The methods can return indications on the success of a call, which is not the case with channels; another channel is needed for that purpose. The methods store data directly under the form expected by the receiver, in collections, databases, or whatever data structure the receiver requires. This eliminates an intermediate storage and the packing/unpacking of the data into/from transmission messages. The synchronous calls make it possible to use object-oriented features such as overloading. Of course, these features could be integrated in the channels, but the programmer should then have a means to redefine the read and the write primitives. The synchronous active objects can be integrated in libraries and called like the passive objects. They are very useful for creating synchronous as well as asynchronous APIs that avoid the need to poll to determine when their operations are finished (see paragraph 4.2). These calls extend the call-channel concept defined by JCSP [1], by allowing » x m communications, as well as the placement of both ends of the communications within select statements. The details of the implementations are otherwise very similar.
114
C. Petitpierre / Synchronous Active Objects
3.4 Calls in Select Statements Often, a potential deadlock can be avoided either by using unlimited channels or by inserting a call - or the writing of data into a channel - within a select statement. hi Figure 2, two clients send messages to a server, which broadcasts them back to the clients, for example in order to manage a distributed blackboard.
Figure 2. Broadcast server
Figure 3. Protocol stack
If the client and the server are implemented with synchronous calls, zero-slot buffers or bounded buffers, and without select statements, it may happen that one client tries to send a message while the server is itself busy, trying to send a message to that client through a synchronous call, a zero-slot buffer or a bounded buffer that has reached its upper limit. In that case, the system enters a deadlock state. This situation can be avoided either by using unlimited buffers, in which case, the senders can never block, or by having the clients accept all incoming calls or messages in parallel with their attempts to send messages, namely having the clients accept and send messages within select statements. Note that if the select statements cannot contain calls with the accepts statements, the solution becomes much more complex. Another example is represented in Figure 3. This figure depicts a protocol stack as defined by the OSI model, in which each layer is an active object. As it is not possible to use select statements that contain only accepts in two consecutive layers (one of them must indeed contain a caller), either a select statement with calls and accepts or unlimited channels must again be used to avoid potential deadlocks. Note that channels would establish an auxiliary protocol between the layers used to create the protocol stack and multiply the number of states that must be examined to analyse the protocol stack. 4
Communications Over a Network
Synchronous calls can be considered as the most basic object-oriented communication means within a single processor because they just add synchronization together with the calls. Accesses to remote objects however are definitely different. In the following, we present our approach to remote method calls, which allows a seamless integration of the current concepts (RMI, CORBA, SOAP) in a CSP-based environment. The main aspect we have considered is the possibility of handling remote accesses from select statements. 4.1 Blocking Call Method remoteMeth shown below is obviously blocking, and if it is called from a select statement, the case that begins with this call is immediately triggered and thus the caller of remoteMeth is blocked while waiting for the returned data, which typically produces the kind of deadlock situations described above.
C. Petitpierre /Synchronous Active Objects
15
class Proxy { // client side TCPSocket socket; result remoteMeth (type param) { string = encode (param); send (string, socket); string = receive (socket); result = decode (string);
4.2 Asynchronous Accesses We have thus decomposed the call into two synchronous calls and executed the message transmission on the run method of the proxy. After the post_ has been accepted, the client must execute a ready_ before it can execute a new post'_. // asynchronous version active class Proxy { // client side TCPSocket socket; void post_remoteMeth (type param) { string = encode (param); result ready remoteMeth { ) { result = decode (string); void run () { for (;;) { select { // there may be other cases case // in this select accept post_remoteMeth; send (string, socket); string = receive (socket); accept ready_remoteMeth;
The following code shows how to use those stubs in a situation where the end-user is given the possibility to cancel the call in case s/he is getting impatient. The proxy should of course contain a cancel method, but these details go beyond the scope of this article. void run () {
// client
proxy.post_remoteMeth(); select { case proxy.ready_remoteMeth(x); case cancel.pressed();
116
C. Petitpierre I Synchronous Active Objects
We have developed a generator that creates stubs and skeletons from an interface. The resulting stubs contain the three methods, m, postjn and ready m, for each method m defined in the interface. The program may thus decide at runtime which kind of call it uses. The skeletons generated only handle the normal methods, as the same message is sent in all cases. The methods of our stubs can thus access the normal CORBA servers. 4.3 Modeling a Remote Connection One can argue that a language such as the one we propose should offer channels to implement remote communications. However, the developer must have a tight control on the way this "channel" is implemented. In some situations, the call is just made to get some information; in which case, there is no problem to cancel it and to perform the port method again later if needed. However, the application may require that if the network breaks after the message has arrived at the destination and before the result has returned to the client, the client and the server either agree that the operation has been executed or that it has not started at all. This is a complex task that must be programmed at the application level, as it must be solved for example by resorting to a two-phase commitment. The fact that the client can back off the ready jnethod in the stub proposed above allows it to get the processor control back and to secure the negative result of the operation on the disk, in order to be capable to transmit it to the server when the connection is reestablished. Simple remote communications such as the connections between distant objects implemented by means of RMI or CORBA stubs, skeletons and TCP sockets can be modeled easily with CSP, considering the fact that two remote calls are executed sequentially, and that a new call can only be performed if the previous one has been completed, hi the following, we model a connection accessed by our post and ready methods, keeping in mind that the former two methods placed in sequence can model a simple method. stub = stub.post m ? x —> server! x —* server ? y —> stub, ready_m ly—> stub
This model allows the analysis of distributed applications composed of synchronous active objects and based on standard environments (RMI, CORBA). 5 Examples of GUI Applications The few statements defined in the previous paragraphs can be used across the whole spectrum of concurrent problems. We have written thousands of lines of code for interactive and distributed applications, without the need of any semaphore, monitor or any kind of channels, hi the following, we present a GUI application and its coding with synchronous calls, then with channels and finally with listeners (see the complete sources at [7]). 5.1 Specification of the Problem We assume we want to read a username and then a password in the same field (in a dialog box similar to the one shown in Figure 4). The entry of the username may take any amount of time, but once a username has been typed in, either: • • •
a password is typed in or the cancel button is pressed (and the demand is cancelled) or nothing happens for a predefined amount of time (and the demand is cancelled).
117
C. Petitpierre /Synchronous Active Objects
Figure 4. A dialog box.
5.2 Design with Synchronous Calls The solution based on synchronous active objects is very close to the definition of the problem. The highlighted words in the source below can be mapped directly into the words of the problem statement. Note however that the clicks made when the program does not read the corresponding device must be ignored. This program can readily be compiled with our compiler and run; one just needs to add the imports and a main. public active class Dialog { Jdialog dialog = new JDialogO; Jlabel label = new JLabel ( "username: ) ; AtextField name = new ATextField(20); Abutton cancel = new AButton(llCancelll) ; Container pane; String passwd, username; public void run 0 { pane = dialog.getContentPane0; pane.add(labe1,BorderLayout.WEST); pane.add(name,BorderLayout.CENTER); pane.add(cance1,BorderLayout.SOUTH); dialog.pack ( 1 ; dialog.setvisible (true); username = name. read ( 1 ; label.setText ( llpassword: If ) ; select { case passwd = name.read0; P: System.out .println("U: + username + case waituntil(System.currentTimeMillis( + 5 0 0 0 ) ; System.out.println("Cancelled! 1 1 ) ; case cancel.pressed0; System.out.println("Cancelled ! I t ) ; } System.exit ( 0 );
+ passwd) ;
1 1 The first statements instantiate the GUI elements and pack them in a dialog box. As the GUI elements AButton and ATextField extend Jbutton and JTextField respectively, they can be handled and integrated in a window exactly like the original elements. The way this program handles non-determinism is worth noticing. The select statement contains several calls, which allows a program to instantiate the GUI elements and then to call them like it would do with passive objects instantiated from a library (and unlike what must be done with the listeners shown in paragraph 5.4 !). This is a very important point, as it avoids the program control inversion introduced by the listeners.
118
C. Petitpierre / Synchronous Active Objects
5.3 The GUI Application with Channels The following code shows how the above select statement would be coded with channels in a JCSP alternation. Like with synchronous calls, the events produced by clicks not taken care of overwrite the previous events. The fact that one channel must be generated for each GUI element makes things a bit more complicated than the solution based on synchronous calls, but the two solutions are otherwise very similar.
AltingChannellnput eventB = Any2OneChannel.create (new OverWriteOldestBuffer (10)); AltingChannellnput eventT = Any2OneChannel.create (new OverWriteOldestBuffer (10)); CSTimer tim = new CSTimer (); ActiveButton button = new ActiveButton (null, eventB, label); ActiveTextField activeText = new ActiveTextField (null, eventT, string) ; Guard[] guards = {eventB, eventT, tim}; final BUTTON = 0, TEXT = 1, TIMEOUT = 2;
// guard indices ... // optional, but nice Alternative alt = new Alternative (guards); tim.setAlarm (tim.read () + timeout); switch (alt.readO) { case BUTTON: String s = (String) eventB.read System.out.printIn ("Button " + break; case TEXT: String s = (String) eventT.read System.out.println ("Text field break; case TIMEOUT: System.out.println("Timeout"); break;
(); s + " pressed");; (); returned " + s) ;
5.4 The Same Application with Listeners Listeners are often considered a very nice concept. However, the implementation of problems such as the one stated in the previous paragraphs with listeners leads to programs as bad as programs written before structured programming was recognized. The source code below presents the part of the implementation that handles the text field.
1 textField.addActionListener (new ActionListener () 2 boolean readUsername = true; 3 public void actionPerfortned (ActionEvent e) { 4 if (readUsername) { 5 names [0] = textField.getText (),6 label.setText ("password: "); 7 readUsername = false;
C. Petitpierre / Synchronous Active Objects 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 }
cancelButton.addActionListener ( new ActionListener () { public void actionPerformed (ActionEvent e) { timer, stop () ,de.continuation (names); } }) ; timer.start (); // code executed somewhere else } else { names [1] = textField.getText ( ); timer.stop (); de.continuation (names); }
} }) ;
The order in which these lines are executed does not correspond at all to the lexical order, which makes their analysis, debugging and maintenance a nightmare and may even explain why the developments of event-driven applications often end up with delays and unstable results. • •
• • •
•
Lines 1 and 2 are executed at the initialisation of the program. Lines 3 to 9 and 15 are executed when the text field is filled in for the first time. The Boolean readUsername had to be added to indicate if the method is called to provide the username or the password, although it did not appear in the specification. Line 9 actually modifies the program at runtime. Lines 16 to 20 are executed after the second time the text field has been filled in. Lines 10 to 13 are executed when the button cancel has been clicked. The timer is dependent on a listener. It had to be created (not shown above), but nevertheless still had to be started and stopped (lines 11,15 and 18). Again, that was not necessary in the previous implementations. It is not possible to wrap the code presented above in an object and read the names obtained from the user by calling a method of that object (without resorting to a synchronous call or a channel !).
Moreover, the problem analysed here is quite simple, hi a complex application that requires many GUI interdependent elements, it becomes almost impossible to discover how these elements are related to each other. Design patterns are supposed to help here, but they just add to the nightmare. The reader who wants to convince himself of that fact can simply read the program given at [2], which is assumed to be an example of the use of patterns. The structure of that program is indeed very difficult to understand. 5.5 Connecting Synchronous Calls to the Listeners The following frame shows how a listener can trigger a synchronous call. Actually, this construct rectifies the program control inversion introduced by the listeners. Instead of directly calling the application code, the listener resumes the main program's execution. The accept must just transmit the indication that the button has been pressed to the kernel, but not wait for the rendezvous (which has the same goal as the overwriting of data mentioned in paragraph 5.2).
C. Petitpieme / Synchronous Active Objects
120
class Sbutton extends m u t t o n { / / listener public void pressed 0 { } ; public SButton 0 { addActionListener (new ActionListener 0 { public void actionPerformed (ActionEvent el { accept pressed; / / no wait!
1
1
}I;
1
The programmer must thus pay attention to the fact that this accept must not be reexecuted before method pressed has actually been executed, and to disable the button for that purpose. This construct is thus a bit delicate, but it is usually made by a system engineer and stored in a library reused without any modification in all applications that need it. Moreover, this construct can be used in the same way for all GUI elements, including the mouse. 6 An Example with a Network Connection
The following example is taken fiom [3]. It demonstrates how synchronous objects can simplifL the development of interactive applications and support UML. Figure 5 describes the steps necessary to authenticate a customer in an ATM (automatic teller machine). This diagram does not give any hint as how to implement the corresponding program, and there is a good chance that the programmer will spend much time devising listeners, dialog box, and so on.
Figure 5. Collaboration diagram.
C. Petitpierre /Synchronous Active Objects
121
We rather propose to transform the diagram shown in Figure 5 into the one shown in Figure 6 . This new diagram contains all the devices and almost all the actions that were defined in the former diagram .... We have just grouped the modules and the devices and hooked all actions to the actors that execute them (customer, session). The program is now sequential, and its code is obvious. All calls are synchronous and even do not require selections. We nevertheless added a selection between a cancel button and the waiting of the response of the authenticator to show how simple it is to handle non-determinism. The cancellation of the reception of the message does not introduce any problem here because it is a read statement that does not modify the authenticator.
Figure 6. Diagram based on synchronous objects.
7 Conclusion We have presented a concept of synchronous calls [4,5] that matches the synchronization aspects of CSP very well, and shown that this concept is very useful to circumvent the problems raised by the use of listeners and the program control inversion they induce.
8 Available Tools We have developed a compiler that handles the synchronous calls and the select statements and produces both binary and Java source files. Actually, the statements that handle the synchronization are simple and could very well be coded by hand too. The developer is thus not tightly bound to the use of the compiler, but the program structure is of course clearer when the compiler is used. We have also developed a state analyser that can handle a subset of Java and determine if the code contains deadlocks. This analyser also produces a CCS (Milner’s calculus of communicating systems) description of the application. We have developed several libraries and generators, and our students and ourselves have written thousands of lines of code with much success. Our developments are available on our web server [6].
122
C. Petitpierre / Synchronous Active Objects
Acknowledgements I am grateful for the help that Dr. Fernando Pedone and Professor Sanjiva Prasad provided me in the writing of this paper. References [1] P. Welch and P.O. Austin, The JCSP Home Page, http://www.cs.ukc.ac.uk/projects/ofa/jcsp/, 2002. [2] http://www.javaworld.com/javaworld/jw-10-1999/jw-10-toolbox.html [3] P. Kruchten, The Rational Unified Process - An Introduction, Addison-Wesley, 2000. [4] C. Petitpierre, Synchronous C++, a Language for Interactive Applications, IEEE Computer, September 1998, pp 65-72. [5] C. Petitpierre, A. Elie'ns, Active Objects Provide Robust Event-driven Applications, SERP'02, Las Vegas, June 2002, pp 253-259. [6] Claude Petitpierre, Synchronous Active Objects, http://ltiwww.epfl.ch/sJava [7] Claude Petitpierre, Synchronous GUI Dialog Sources, http://ltiwww.epfl.ch/sJava/version2/Inversion
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
123
Configurable Collective Communication in LAM-MPI John Markus BJ0RNDALEN1, Otto J. ANSHUS1 Brian VINTER2, Tore LARSEN1 1 ' Department of Computer Science, University of Troms0 ^ Department of Mathematics and Computer Science, University of Southern Denmark
Abstract. In an earlier paper, we observed that PastSet (our experimental tuple space system) was 1.83 times faster on global reductions than LAM-MPI. Our hypothesis was that this was due to the better resource usage of the PATHS framework (an extension to PastSet that supports orchestration and configuration) due to a mapping of the communication and operations which matched the computing resources and cluster topology better. This paper reports on an experiment to verify this and represents ongoing work to add some of the same configurability of PastSet and PATHS to MPI. We show that by adding run-time configurable collective communication, we can reduce the latencies without recompiling the application source code. For the same cluster where we experienced the faster PastSet, we show that All reduce with our configuration mechanism is 1.79 times faster than the original LAM-MPI All reduce. We also experiment with the configuration mechanism on 3 different cluster platforms with 2-, 4-, and 8-way nodes. For the cluster of 8-way nodes, we show an improvement by a factor of 1.98 for All reduce.
1
Introduction
For efficient support of synchronization and communication in parallel systems, these systems require fast collective communication support from the underlying communication subsystem as, for example, is defined by the Message Passing Interface (MPI) Standard [1]. Among the set of collective communication operations broadcast is fundamental and is used in several other operations such as barrier synchronization and reduction [2]. Thus, it is advantageous to reduce the latency of broadcast operations on these systems. In our work with the PATHS [3] configuration and orchestration system, we have experimented with microbenchmarks and applications to study the effects of configurable communication. In one of the experiments[4], we used the configuration mechanisms to reduce the execution times of collective communication operations in PastSet. To get a baseline, we compared our reduction operation with the equivalent operation in MPI (Allreduce). By trying a few configurations, we found that we could improve our Tuple Space system to be 1.83 times faster than LAM-MPI[5][6]. Our hypothesis was that this advantage came from a better usage of resources in the cluster rather than a more efficient implementation. If anything, LAM-MPI should be faster than PastSet since PastSet stores the results of each global sum computation in a tuple space inducing more overhead than simply computing and distributing the sum.
124
J. Bj0mdalen et al. / Configurable Collective Communication in LAM-MPI
This paper reports on an experiment where we have added configurable communication to the Broadcast and Reduce operations in LAM-MPI (both of which are used by All reduce) to validate or falsify our hypothesis. The paper is organized as follows: Section 2 summarizes the main features of the PastSet and PATHS system. Section 3 describes the All reduce, Reduce and Broadcast operations in LAM-MPI. Section 4 describes the configuration mechanism that was added to LAM-MPI for the experiments reported on in this paper. Section 5 describes the experiments and results, section 6 presents related work, and section 7 concludes the paper. 2 PATHS: Configurable Orchestration and Mapping Our research platform is PastSet[7][8], a structured distributed shared memory system in the tradition of Linda[9]. PastSet is a set of Elements, where each Element is an ordered collection of tuples. All tuples in an Element follow the same template. The PATHS[3] system is an extension of PastSet that allows for mappings of processes to hosts at load time, selection of physical communication paths to each element, and distribution of communications along the path. PATHS also implements the X-funotions[4], which are PastSet operation modifiers. A path specification for a single thread needing access to a given element is represented by a list of stages. Each stage is implemented using a wrapper object hiding the rest of the path after that stage. The stage specification includes parameters used for initialisation of the wrapper. Nodel Thread
Thread
(GlobalSum) 1 (Remote access) ~*x.
Node 2 Thread
Thread
(Global Sum) I (Remote access)
Server
Figure 1: Four threads on two hosts accessing shared element on a separate server. Each host computes a partial sum that is forwarded to the global-sum wrapper on the server. The final result is stored in the element.
Paths can be shared whenever path descriptions match and point to the same element (see figure 1). This can be used to implement functionality such as, for instance, caches, reductions and broadcasts. The collection of all paths in a system pointing to a given element forms a tree. The leaf nodes in the tree are the application threads, while the root is the element. Figure 1 shows a global reduction tree. By modifying the tree and the parameters to the wrappers in the tree, we can specify and experiment directly with factors such as which processes participate in a given partial sum, how many partial sum wrappers to use, where each sum wrapper is located, protocols and service requirements for remote operations and where the root element is located. Thus, we can control and experiment with tradeoffs between placement of computation, communication, and data location.
J. Bj0mdalen et al. / Configurable Collective Communication in LAM-MPI
125
Applications tend to use multiple trees, either because the application uses multiple elements, or because each thread might use multiple paths to the same element. To get access to an element, the application programmer can either choose to use lowerlevel functions to specify paths before handing it over to a path builder, or use a higher level function which retrieves a path specification to a named element and then builds the specified path. The application programmer then gets a reference to the topmost wrapper in the path. The path specification can either be retrieved from a combined path specification and name server, or be created with a high-level language library loaded at application load-time! Since the application program invokes all operations through its reference to the topmost wrapper, the application can be mapped to different cluster topologies simply by doing one of the following: • Updating a map description used by the high-level library. • Specifying a different high-level library that generates path-specifications. This library may be written by the user. • Update the path mappings in the name server. Profiling is provided via trace wrappers that log the start and completion time of operations that are invoked through it. Any number of trace wrappers can inserted anywhere in the path. Specialized tools to examine the performance aspects of the application can later read trace data stored with the path specifications from a system. We are currently experimenting with different visualizations and analyses of this data to support optimization of a given application. The combination of trace data, a specification of communication paths, and computations along the path has been useful in understanding performance aspects and tuning benchmarks and applications that we have run in cluster and multicluster environments.
3 LAM-MPI implementation of Al 1 reduce LAM-MPI (Local Area Multicomputer) is an open source implementation of MPI available from [5]. It was chosen over MPICH[10] for our work in [4] since it had lower latency with less variance than MPICH for the benchmarks we used in our clusters. The MPI Allreduce operation combines values from all processes and distribute the result back to all processes. LAM-MPI implements Allreduce by first calling Reduce, collecting the result in the root process, then calling Broadcast, distributing the result from the root process. For all our experiments, the root process is the process with rank 0 (hereafter called process 0). The Reduce and Broadcast algorithms use a linear scheme (every process communicates directly with process 0) up to and including 4 processes. From there on they use a scheme that organizes the processes into a logarithmic spanning tree. The shape of this tree is fixed, and doesn't change to reflect the topology of the computing system or cluster. Figure 2 shows the reduction trees used in LAM-MPI for 32 processes in a cluster. We observe that broadcast and reduction trees are different. By default, LAM-MPI evenly distributes processes onto nodes. When we combine this mapping for 32 processes with the reduction tree, we can see in Figure 3 that a lot of messages are sent across nodes in the system. The broadcast operation has a better mapping for this cluster though. Currently, a Python module is loaded for this purpose.
126
J. Bj0mdalen et al. I Configurable Collective Communication in LAM-MPI
Figure 2: Log-reduce tree for 32 processes. The arcs represent communicadon between two nodes. Partial sums are computed at a node in the tree before passing the result further up in the tree.
Figure 3: Log-reduce tree for 32 processes mapped onto 8 nodes.
4 Adding configuration to LAM-MPI To minimize the necessary changes to LAM-MPI for this experiment, we didn't add a full PATHS system at this point. Instead, a mechanism was added that allowed for scripting the way LAM-MPI communicates during the broadcast and reduce operations. There were two main reasons for this. Firstly, our hypothesis was that PastSet with PATHS allowed us to map the communication and computation better to the resources and cluster topology. For global reduction and broadcast, LAM-MPI already computes partial sums at internal nodes in the trees. This means that experimenting with different reduction and broadcast trees should give us much of the effect that we observed with PATHS in [4] and [3]. Secondly, we wanted to limit the influence that our system would have on the performance aspects of LAM-MPI such dial any observable changes in performance would come from modifying the reduce and broadcast trees. Apart from this, the amount of code changed and added was minimal, which reduced the chances of introducing errors into the experiments.
/. Bj0mdalen et al. / Configurable Collective Communication in LAM-MP1
127
When reading the LAM-MPI source code, we noticed that the reduce operation was, for any process in the reduction tree, essentially a sequence of N receives from the N children directly below it in the tree, and one send to the process above it. For broadcast, the reverse was true; one receive followed by N sends. Using a different reduction or broadcast tree would then simply be a matter of examining, for each process, which processes are directly above and below it in the tree and construct a new sequence of send and receive commands. To implement this, we added new reduce and broadcast functions which used the rank of the process and size of the system to look up the sequence of sends and receives to be executed (including which processes to send and receive from). This is implemented by retrieving and executing a script with send and receive commands. As an example, when using a scripted reduce operation with a mapping identical to the original LAM-MPI reduction tree, the process with rank 12 (see figure 2) would look up and execute a script with the following commands: • Receive (and combine result) from rank 13 • Receive (and combine result) from rank 14 • Send result to 8 The new scripted functions are'used instead of the original logarithmic Reduce and Broadcast operations in LAM-MPI. No change was necessary to the Allreduce function since it is implemented using the reduce and broadcast operations. The changes to the LAM-MPI code was thus limited to 3 code lines, replacing the calls to the logarithmic reduction and broadcast functions as well as adding and a call in MPIJnit to load the scripts. Remapping the application to another cluster configuration, or simply trying new mappings for optimization purposes, now consists of specifying new communication trees and generating the scripts. A Python program generates these scripts as Lisp symbolic expressions. 5
Experiments tl = get_usecs(); MPI_Allreduce(&hit, &ghit, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); tl = get_usecs() ,for (i = 0; i < ITERS; i++) { MPI_Allreduce(&i, kghit, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); if (ghit != (i * size)) printf("oops at %d. %d != %d\n", i, ghit, (i * size)); } t2 = get_usecs(); MPI_Allreduce(&hit, &ghit, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
Figure 4: Global reduction benchmark
Figure 4 shows the code run in the experiments. The code measures the average execution time of 1000 Allreduce operations. The average of 5 runs is then plotted. To make sure that the correct sum is computed, the code also checks the result on each iteration. For each experiment, the number of processes was varied from 1 to 32. LAM-MPI used the default placement of processes on nodes, which evenly spread the processes over the nodes.
128
/. Bj0mdalen et al. /Configurable Collective Communication in LAM-MPl
The hardware platforms consists of three clusters, each with 32 processors: • 2W: 16*2-Way Pentium ffl 450 MHz, 256MB RAM • 4W: 8*4-Way Pentium Pro 166 MHz, 128MB RAM • 8W: 4*8-Way Pentium Pro 200 MHz, 2GB RAM All clusters used 100MBit Ethernet for intra-cluster communication. 5.7
4-way cluster
Figure 5 shows experiments run on the 4-way cluster. Configurable schemes vs pristine LAM-MPI 3000 2500 g? 2000
1500 1000
500
I
5
10 15 20 25 Number of processes in system
30
35
Figure 5: All reduce, 4-way cluster
The following experiments were run: • Original LAM-MPI: the experiment was run using LAM-MPI 6.5.6. • Modified LAM-MPI, original scheme: the experiment was run using a modified LAMMPI, but using the same broadcast and reduce trees for communication as the original version used. This graph completely overlaps the graph from the original LAM-MPI code, showing us that the scripting functionality is able to replicate the performance aspects of the original LAM-MPI for this experiment. • Modified LAM-MPI, linear scheme: this experiment was run using a linear scheme where all processes report directly to the process with rank 0, and rank 0 sends the results directly back to each client. This experiment crashed at 18 processes due to TCP connection failures in LAM-MPI. We haven't examined what caused this, but it is likely to be a limitation in the implementation. • Modified LAM-MPI, hierarchal scheme 1: rank 0 is always placed on node 0, so all other processes on node 0 contribute directly to rank 0. On the other nodes, a local reduction is done within each node before the nodes partial sum is sent to process 0.
/. Bj0mdalen et al. /Configurable Collective Communication in LAM-MPI
129
The modified LAM-MPI with the hierarchal scheme is 1.79 times faster than the original LAM-MPI. This is close to the factor of 1.83 reported in [4], which was based on measurements on the same cluster. It is possible that further experiments with configurations would have brought us closer to the original speed improvement. 5.2
8-way cluster
For the 8-way cluster, we expected the best configuration to involve using an internal root process on each node and limit the inter-node communications to one send (of the local sum) and one receive (of the global result) resembling the best configuration for the 4-way cluster. This turned out not to be true. Configurable LAM-MPI vs pristine LAM-MPI 3000
LAM-MPI original code LAM-MPI + path - - - -
2500 2000 1500 1000 500
0
5
10 15 20 25 Number of processes in system
30
35
Figure 6: All reduce, 8-way cluster
Due to the 8 processes on each node, the local root process on each node would get 7 children in the sum tree (with the exception of the root processes in the root node). The extra latency added per child of the local root processes was enough to increase the total latency for 32 processes to 1444 microseconds. Splitting the processes to allow further subgrouping within each node introduced another layer in the sum tree. This extra layer brought the total latency for 32 processes to 1534 microseconds. The fastest solution found so far involves partitioning the processes on each node into two groups of 4 processes. Each group computes a partial sum for that group, and forwards it to directly to the root process. This doubled the communication over the network, but the total execution time was shorter (1322 microseconds). The fastest configuration is 1.98 times faster than the original LAM-MPI All reduce operation. In figure 6, the fastest configuration and the original LAM-MPI All reduce are plotted for 1-32 processes. 5.3 2-way cluster On the 2-way cluster, after trying a few configurations, we ended up with a configuration that was 1.52 times faster than the original LAM-MPI code. This was not as good as for the other clusters. We expected this since a reduction in this cluster would need more external messages.
130
J. Bj0rndalen et al. / Configurable Collective Communication in LAM-MPI
Configurable LAM-MPI vs pristine LAM-MPI 3000 2500
LAM-MPI original code LAM-MPF+ PATHS
2000
1500 1000
500
£ o8.
0
10 15 20 25 Number of processes in system
30
35
Figure 7: Allreduce, 2-way cluster.
Based on the experiments, we observed three main factors that influenced the scaling of the application: • The number of children for a process in the operation tree. More children adds communication load for the node hosting this process. • The depth of a branch in the operation tree. That is, the number of nodes a value has to be added through before reaching the root node. • Whether a message was sent over the network or internally on a node. The depth factor seemed to introduce a higher latency than adding a few more children to a process. On the other hand, adding another layer in the tree would also allow us to use partial sums. We have not arrived at any model which allows us predict which tree organization would in practice lead to the shortest execution time of the Allreduce operation. One difference between the PastSet/PATHS implementation and LAM-MPI is that PastSet/PATHS process incoming partial sums by order of arrival, while LAM-MPI process them in the order they appear in the sequence specified either by the original LAM-MPI code, or the script. This might influence the overhead when receiving messages from a larger number of children. We are investigating this further. 5.4
Scaling up the number of nodes on the 2-way cluster
We also scaled the 2-way cluster along another axis: the number of available nodes for the application. The motivation for this is that clusters do not necessarily have a power of two nodes, such as the clusters we used in the previous sections2. A given algorithm for building an operation tree might result in a good mapping for one cluster configuration, but result in a less efficient operation on another cluster configuration. In this experiment, we scaled the number of nodes from 1 to 16. The number of processes was scaled with the number of nodes, adding two processes for each node added to the experiment. Figure 8 shows an experiment where we compare the original LAM-MPI mappings with a setting where we reconfigure the best algorithm from section 5.3 to fit with a different sized 2
either because of lack of funding, broken nodes, or nodes allocated for other purposes
/. Bj0mdalen et al. / Configurable Collective Communication in LAM-MPI
131
Configurable LAM-MPI vs pristine LAM-MPI 3000 2500
LAM-MPI original code, 1-1^ nodes LAM-MPI + path, expl scheme, 1-16 nodes
10 15 20 25 Number of processes in system
Figure 8: All reduce, 2-way cluster, scaling the number of nodes from 1 to 16. Each node added adds two new processes in the experiment.
cluster. We generally outperform the original LAM-MPI algorithm for all cluster sizes larger than 2 in this experiment. At 16 nodes and 32 processes, we end up with the same cluster configuration and the same latencies as in the previous section. 6 Related work Jacunski et al.[l 1] shows that selection of the best performing algorithm for all-to-all broadcast on clusters of workstations based on commodity switch-based networks is a function of both network characteristics as well as the message length and number of participating nodes in the all-to-all broadcast operation. In contrast our work used clusters with multiprocessors, we did performance measurements on the reduce operation as well as on the broadcast, and we documented the effect of the actual system at run time including the workload, communication performance of the hosts. Bernashci et al.[12] study the performance of the MPI broadcast operation on large shared memory systems using a-trees. Kielmann et al.[13] show how the performance for collective operations like broadcast depend upon cluster topology, latency, and bandwidth. They develop a library of collective communication operations optimized for wide area systems where a majority of the speedup comes from limiting the number messages passing over wide-area links. Husbands et al.[14] optimizes the performance of MPI_Bcast in clusters of SMP nodes by using a two-phase scheme; first messages are sent to each SMP node, then messages are distributed within the SMP nodes. Sistare et al.[15] uses a similar scheme, but focus on improving the performance of collective operations on large-scale SMP's. Tang et al.[16] also use a two-phase scheme for a multithreaded implementation for MPI. They separate the implementation of the point-to-point and collective operations to allow for optimizations of the collective operations, which would otherwise be difficult. In contrast to the above works, this paper is not focused on a particular optimization of the spanning trees. Instead, we focus on making the shape of the spanning trees configurable to allow easy experimentation on various cluster topologies and applications. Vadhiyar et al.[17] shows an automatic approach to selecting buffer sizes and algorithms to optimize the collective operation trees by conducting a series of experiments.
132
7
/. Bj0mdalen et al. I Configurable Collective Communication in LAM-MP7
Conclusions
We have observed that the broadcast and reduction trees in LAM-MPI are different, and do not necessarily take into account the actual topology of the cluster. By introducing configurable broadcast and reduction trees, we have shown a simple way of mapping the reduction and broadcast trees to the actual clusters in use. This gave us a performance improvement up to a factor of 1.98. For the cluster where we observed the performance difference of a factor 1.83 between PastSet/PATHS and LAM-MPI, we arrived at a reduction and broadcast tree that gave us an improvement of 1.79 for All reduce over the original LAM-MPI implementation. This supports our hypothesis mat the majority of the performance-difference between LAM-MPI and PastSet/PATHS was a better mapping to the resources and topology in the cluster. We have also observed that the assumption that doing a reduction internally in each node before sending a message on the network did not lead to the best performance on our cluster of 8-way nodes. Instead, increasing the number of messages on the network to reduce the depth of the reduction and broadcast as well as the number of direct children for each internal node proved to be a better strategy. The reason for this may be found by studying three different factors that add to the cost of the global reduction and broadcast trees: • The number of children directly below an internal node in the spanning tree. • The depth of the spanning tree. • Whether an arc between a child and a parent in the spanning tree is a message on the network or internally in the node. We suspect that these factors are not increasing linearly, and that the cost of, for instance, adding another child to an internal node in the spanning tree depends on factors such as the order of the sequence of receive commands as well as contention on the network layer in the host computer. 8 Acknowledgements Thanks to Ole Martin Bj0rndalen for reading the paper and suggesting improvements. References [1] Mpi: A message-passing interface standard. Message Passing Interface Forum, Mar 1994. [2] D.K. Panda. Issues in designing efficient and practical algorithms for collective communication in wormhole-routed systems. Proc. ICPP Workshop Challenges for Parallel processing, pages 8-15, 1995. [3] John Markus Bj0mdaJen, Otto Anshus, Tore Larsen, and Brian Vinter. Paths - integrating the principles of method-combination and remote procedure calls for run-time configuration and tuning of highperformance distributed application. Norsk Informatikk Konferanse, pages 164-175, November 2001. [4] Brian Vinter, Otto J. Anshus, Tore Larsen, and John Markus Bj0rndalen. Extending the applicability of software dsm by adding user redefinable memory semantics. Parallel Computing (ParCo) 2001, Naples, Italy, September 2001. [5] http://www.lam-mpi.org/. [6] Jeffrey M. Squyres, Andrew Lumsdaine, William L. George, John G. Hagedom, and Judith E. Devaney. The interoperable message passing interface (IMPI) extensions to LAM/MPI. In Proceedings, MPIDC'2000, March 2000.
/. Bj0mdalen et al. / Configurable Collective Communication in LAM-MPI
133
[7] O. J. Anshus and Tore Larsen. Macroscope: The abstractions of a distributed operating system. Norsk Informatikk Konferanse, October 1992. [8] Brian Vinter. PastSet a Structured Distributed Shared Memory System. PhD thesis, Troms0 University, 1999. [9] N. Carriero and D. Gelernter. Linda in context. Commun. ACM, 32(4):444-458, April 1989. [10] William Gropp, Ewing Lusk, Nathan Doss, and Anthony Skjellum. A high-performance, portable implementation of the mpi message passing interface standard. Parallel Computing, Volume 22, Issue 6, September 1996. [11] Matt Jacunski, P. Sadayappan, and O.K. Panda. All-to-all broadcast on switch-based clusters of workstations. 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing, 12 - 16 April 1999. San Juan, Puerto Rico. [12] Massimo Bernaschi and Giorgia Richelli. Mpi collective communication operations on large shared memory systems. Proceedings of the Ninth Euromicro Workshop on Parallel and Distributed Processing (EUROPDP.01),200l. [13] Thilo Kielmann, Rutger F. H. Hofman, Henri E. Bal, Aske Plaat, and Raoul A. F. Bhoedjang. Magpie: Mpi's collective communication operations for clustered wide area systems. Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming, 1999. Atlanta, Georgia, United States. [14] Parry Husbands and James C. Hoe. Mpi-start: delivering network performance to numerical applications. Proceedings of the 1998 ACM/IEEE conference on Supercomputing, 1998. San Jose, CA. [15] Steve Sistare, Rolf vandeVaart, and Eugene Loh. Optimization of mpi collectives on clusters of large-scale smp's. Proceedings of the 1999 conference on Supercomputing, 1999. Portland, Oregon, United States. [16] Hong Tang and Tao Yang. Optimizing threaded mpi execution on smp clusters. Proceedings of the 15th international conference on Supercomputing, 2001. Sorrento, Italy. [17] Sathish S. Vadhiyar, Graham E. Fagg, and Jack Dongarra. Automatically tuned collective communications. Proceedings of the 2000 conference on Supercomputing, 2000. Dallas, Texas, United States.
134
This page intentionally left blank
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
135
Cache-Affinity Scheduling for Fine Grain Multithreading Kurt DEBATTISTA, Kevin VELLA and Joseph CORDINA Department of Computer Science and Artificial Intelligence University of Malta Msida MSD06, Malta, Europe Email: {kurt, kvel, jcord}@cs . urn . edu . mt Abstract. Cache utilisation is often very poor in multithreaded applications, due to the loss of data access locality incurred by frequent context switching. This problem is compounded on shared memory multiprocessors when dynamic load balancing is introduced and thread migration disrupts cache content. In this paper, we present a technique, which we refer to as 'batching', for reducing the negative impact of fine grain multithreading on cache performance. Prototype schedulers running on uniprocessors and shared memory multiprocessors are described, and finally experimental results which illustrate the improvements observed after applying our techniques are presented.
1
Introduction
Multithreaded applications, particularly those of the fine grain variety, deal a severe blow to effective cache use. Frequent context switching disrupts the operation of the locality principle, on which cache hits depend. This problem manifests itself on uniprocessors as well as on shared memory multiprocessors, but in the latter case further issues related to thread migration emerge. With fine grain threads, descheduling points tend to occur very frequently. The expense of repopulating the processor's cache with a newly dispatched thread's footprint becomes significant when viewed in relation to the shortened thread dispatch time. To make matters worse, an individual thread is unlikely to accumulate a significant cache footprint by itself: only when threads are considered in groups can a long term cache footprint be identified. Coarse grain threads and operating system processes do not suffer from either of these shortcomings, as their dispatch times are long enough to push this cost into relative insignificance. On multiprocessors, applications execute well only when all processors are sharing the work load and running threads are located close to the data they access. A system that promotes load balancing aims to ensure all processors service an equal workload through the use of a central or shared run queue. While load balancing is commendable, the shared run queue will frequently migrate descheduled threads onto processors on which they would have not recently executed. This will often mean that the newly dispatched thread is unable to find its data footprint in cache, eventually leading to cache misses in the order of tens to hundreds of nanoseconds. Furthermore, Anderson et al. [1] note how fine grain multithreading is more susceptible to thread migration due to the frequency of inter-thread communication and the large number of threads being executed concurrently. On the other hand, cache-affinity schedulers traditionally utilise per-processor run queues. Squillante and Lazowska [2] discuss the benefits of allowing a process to develop affinity to a
136
K. Debattista et al. / Cache-Affinity Scheduling for Fine Grain Multithreading
processor. Markatos and LeBlanc [3] also demonstrate the benefits of locality management. Subsequently, in [4] they observe that the current imbalance between ever-faster processors and slow memory technology will induce a move towards locality awareness in scheduler design. In particular, they propose techniques to improve performance of fine grain thread scheduling using 'memory conscious scheduling' [5], which groups threads that access the same data areas. In the light of the above discussion, a conflict between load balancing and locality is evident. A scheduling algorithm that complies with the load balancing policy would attempt to migrate threads to balance work load evenly, disregarding the thread's last dispatch locale. Conversely, locality conscious scheduling would opt for threads to spend their entire execution life time on the same processor neglecting that another processor might be left idle. Ideally, scheduling algorithms should aspire to include both policies. Through the use of novel scheduling algorithms which group fine grain threads together into coarser grain entities termed batches [6], we attempt to improve cache exploitation on uniprocessors. Furthermore, on shared memory multiprocessors, such batching algorithms maintain data locality and minimise contention for shared scheduler data structures while still maintaining a balanced workload. In batch schedulers, the scheduled entity becomes the batch. Processors obtain batches from a common batch pool and service the threads on the batch for a number of dispatches maintained by a dispatch counter. When the dispatch counter threshold is considerably larger than the batch size and the entire memory footprint of the batch fits in the cache, each thread is guaranteed to find the data it requires in the cache (assuming that the underlying kernel thread is not descheduled by the operating system, a relatively rare event). This is bound to alleviate the memory access locality problem in fine grain multithreaded applications. On multiprocessors, batching reaps further benefits by decreasing the incidence of false sharing since threads accessing data in a common cache line could be batched together. Moreover, when balancing load across processors, migrating threads in batches alleviates the contention that arises when migrating multitudes of individual threads.
2 Related Work In the context of our discussion, SMP scheduling strategies can be broadly classified into two camps. On one side are the more commonly used shared run queue schedulers, which make load balancing their main priority. At the other end there are the per-processor run queue configurations, which naturally provide cache-conscious scheduling and reduce run queue contention. In this section we survey work related to thread scheduling using both policies. Shared run queue schedulers adhere strongly to the load balancing policy since the approach is ideal for distributing threads equally amongst processors. When a processor is idle it is assigned a job from the queue, when it is preempted (or the job finishes) it selects another from the shared queue. Shared run queue's load balancing policies are attractive since it is easy to ensure that no processor is ever idle when there is work to be done. The shared run queue strategy unfortunately, neglects the principal of locality, since threads are usually migrated across processors, without exploiting cache. However, the absence of locality is not the only issue related to shared run queue schedulers, which traditionally rely on spin locks as a method of access control for shared data structures. An oft-overlooked alternative method of synchronisation is available through the considered use of lock-free structures and algorithms, which dispense with the serialisation of concurrent tasks. Lock-free data structures rely on powerful hardware atomic primitives and careful ordering of instructions to protect them from unsafe concurrent access. Valois [7] discusses lock-free techniques in detail and supplies various definitions of relevance to us. A lock-free data structure is
K. Debattista et al. / Cache-Affinity Scheduling for Fine Grain Multithreading
137
termed non-blocking if an operation on it is guaranteed to complete in finite time. Moreover, if every such operation on the data structure is guaranteed to complete the structure is said to be wait free. Results appearing in [8] indicate that non-blocking data structures outperform their lock-based counterparts. Furthermore, when used for thread scheduling and inter-thread synchronisation, non-blocking data structures have other advantages, including stronger fault tolerance, deadlock freedom and, in priority-based schedulers, elimination of priority inversion. Unfortunately, the non-blocking algorithms usually rely on retries to recover from unexpected alterations performed concurrently by other processors. This can result in unpredictable delays and starvation under high contention. Furthermore, the use of COMPARE-AND-SWAP in most of the algorithms introduces the ABA problem, which necessitates the use of costly memory management techniques to avoid it. On the other hand, wait-free data structures, as discussed by Herlihy [9], do not suffer from the ABA problem and do not ever require retries. As a consequence, starvation is eliminated and the maximum number of instructions executed in the algorithms is fixed at compile-time. No efficient wait-free concurrent queue algorithm is available ruling out a shared run queue version of such an algorithm. We will present a per processor run queue scheduler using wait-free data structures in Section 6. Per processor run queue schedulers befit the principle of locality. On NORMAs, Eager et al, [1.0] favour per processor run queues due to the high costs associated with migration. Bellosa [11] arrives at similar conclusions regarding NUMA multiprocessors. However, the general trend on UMA multiprocessors was to base scheduling policies on shared run queue models, to balance loads evenly [12, 13]. Anderson et al [1] observe that the central queue can be a cause of great contention and find that per processor run queues perform better than shared run queues when scheduling threads at the user level. Anderson's scheduling strategy involves placing new threads on the run queue of the processor on which it was created. Processors first check their own run queue for threads to execute, if none are found they scan through other run queues. Threads that are created on one processor generally spend their entire execution time on that processor's run queue, thus maintaining cache affinity. Rarely do threads migrate to other processors. Based on the above results thread packages such as Filaments [ 14] and Cilk [15] adopt per processor run queues as the scheduling strategy. Cilk's per processor run queue algorithm is based on Arora, Blumofe and Plaxton [16] non-blocking double-ended queue (deque). Their non-blocking version of the work stealing algorithm permits processors to access their local deque only by pushing and popping from the bottom (effectively a local deque functions as a stack), while idle processors acquire work by popping threads off the top of the queue. Results [15] demonstrate the system to be highly efficient due to the combination of per processor run queues and non-blocking data structures.
3 Scheduler Design Overview To investigate the effects of different scheduler configurations on the performance of multithreaded applications, we have implemented a suite of uniprocessor and multiprocessor user-level thread schedulers [17] under the collective name of smash. All uniprocessor and SMP smash schedulers borrow many basic ideas, such as active context switching, from CERN's uniprocessor MESH [18] as well as uniprocessor CCSP [19]. Of the various uniprocessor schedulers we have implemented, we consider two: a basic uniprocessor scheduler (uni-smash) and a batch based uniprocessor scheduler (unibatch-smash). We also consider three SMP schedulers: a traditional lock-based shared run queue SMP thread scheduler (shared-smash), a lock-based batch thread scheduler (smpbatch-smash) and our innovative wait-free SMP thread scheduler (wf-smash).
138
K. Debattista et al. I Cache-Affinity Scheduling for Fine Grain Multithreading
Each scheduler takes the form of a C library, and application programs are written in C and linked to the library. The schedulers provide functionality to create, execute and join (barrier synchronisation on thread termination) threads, as well as facilities for inter-thread communication and synchronisation by means of CSP [20] constructs. Both the uniprocessor and SMP thread schedulers schedule threads entirely at the user level. A kernel thread is permanently bound to each processor, and user-level threads are scheduled atop the kernel threads. In the case of the SMP scheduler's, each processor's identity is stored in a reserved register for fast retrieval (thus avoiding the use of ge tpid ( ) ) . Since the schedulers are expected to operate in a multiprogrammed environment, an idle kernel thread (that is, with no user-level threads to execute) sleeps on a kernel semaphore, rather than busy-waiting. This is the only instance in which we make use of a system call within the scheduler (excluding scheduler start-up and shut-down, of course). uni-smash is our vanilla uniprocessor scheduler composed of a single circular queue acting as the run queue, shared-smash is based on the designs of SMP KRoC [13] and the University of Malta's SMP MESH implementation [12]. Processors acquire threads by accessing the shared run queue. This scheduling strategy ensures that no processor is idle when there are threads on the run queue. This methodology favours load balancing at the expense of minimal cache reuse. Access control to the shared run queue is taken care of by a spin lock. The shared run queue is the main cause of poor performance when scheduling fine grained applications, unibatch-smash will be described in Section 4, smpbatch-smash in Section 5 and wf-smash in Section 6.
4 Uniprocessor Batch Scheduling It is clear that fine grain multithreading has an adverse impact on locality, and thus on the ability of cache hardware to have the necessary data in the cache when it is needed. On shared memory multiprocessors these cache miss problems are compounded by frequent process migration as well as false sharing, when processes executing on separate processors repeatedly write to memory addresses on the same cache line. On uniprocessors, we will demonstrate that performance loss through cache misses alone may be substantial, but can be avoided through specially designed scheduling policies and algorithms. We introduce batching as a scheduling strategy which improves cache exploitation in fine grain multithreading systems in its uniprocessor incarnation unibatch-smash, and analyse its performance compared to uni-smash in Section 7. In Section 5, batching is proposed as a technique to mitigate efficiency problems associated with fine grain parallelism on shared memory multiprocessors. Uniprocessor batching, illustrated in Figure 4, consists of organising runnable threads in queues of queues. The run queue is transformed into a queue of batches of threads which we will refer to as the batch run queue. Each batch is itself organised as a queue of (possibly related) runnable threads grouped together into a coarser grain entity. A batch has two variables associated with it: a size counter BSize and a dispatch counter BCount. The batching strategy also includes a threshold size, MAXBSIZE, which BSize cannot exceed. New batches are created by old batches overflowing, and batches whose constituent threads all terminate are destroyed. Whenever a thread launches a new thread, the newly created thread is placed on the current batch. However, if the current batch has met its size limit, the thread is placed onto a temporary batch which we call the overflow batch. The purpose of the overflow batch is to store excess threads until the overflow batch's BSize meets the size threshold. When an overflow batch is full it is placed onto the batch run queue. When scheduling a new batch, the scheduler picks up the batch at the top of the batch run queue, making it the current batch. The dispatch time of a batch is much longer
K. Debattista et al. / Cache-AffinityScheduling for Fine Grain Multithreading
139
Figure 1: Scheduler data structures and information flow in uniprocessorbatching.
than that of an individual thread, and when BCoun t, reaches a threshold, MAXBCOUNT, the batch is appended to the end of the batch run queue and the next batch is scheduled. The rationale behind this strategy is that the primary benefactors of caches are threads which access the same areas of memory repeatedly over a long period of time, and the ones which are thwarted by multithreading are those which deschedule frequently over their lifetime. This is because intervening threads tend to push the relevant memory areas out of the cache. If the batch size is limited and the dispatch period for a batch is long enough, threads in that batch will get rescheduled several times within the batch’s dispatch period, often finding their memory still intact in the cache. This is because the relatively smaller number of intervening threads will not have managed to wipe out the required cache content. Moreover, if communicating processes gather onto the same batch, they will both get rescheduled on that batch time after time, with high probability, and will therefore still be able to benefit from this scheme.
5 SMP Batch Scheduling The idea of batching processes increases in relevance on shared memory multiprocessors. The coarser grain nature of batches (compared to individual threads) in the SMP case helps not only in improving locality due to cache affinity, but also in reducing false sharing and decreasing contention for shared resources. We present two algorithms that we use to migrate batches thus effectively balancing load amongst processors. The first algorithm implemented in our scheduler srnpbatch-smash is described in this section, and only requires the use of a s p i n l o c k construct. The second, wf-smash, described in Section 6, uses powerful atomic primitives which are not available on all architectures, to construct wait-free data structures reducing the contention for shared data structures to a minimum.
140
K.Debattista et al. / Cacbe-Afihity Scheduling for Fine Grain Multittmading
Figure 2: Scheduler data structures and information flow in shared run queue S M P batching.
5.I
Shared Run Queue Batch Scheduling
smpbatch-smash data structures consist of an overflow batch and current batch, which are local to each processor and a shared batch queue. A three processor configuration is illustrated in Figure 2. As with unibatch-smash each processor maintains a separate BCount counter, incrementing it every time a new thread from the batch is dispatched, and reinserting the batch in the batch run queue when it reaches MAXBCOWT. The batch size is maintained by incrementing and decrementing BS i ze as processes join and leave the batch during its dispatch period. The shared batch run queue obeys the shared run queue model. Access control is maintained by means of a lock BQlck. Idle processors retrieve thteads in the form of batches by first acquiring BQlck then dequeuing a batch from the shared batch queue. Similarly, processors with excess workloads enqueue overflow batches on to the shared batch queue. The shared batch queue exhibits the same load balancing properties akin to shared run queues, and it is easy to ensure that no processor is ever idle when there is a batch on the shared batch queue. Each processor's overflow batch is used in very much the same way as in the uniprocessor batching implementation, to store surplus threads when the current batch size exceeds MAXBSIZE, without causing frequent expensive accesses to the shared batch queue. When the overflow queue itself overflows, it is offloaded onto the shared run queue in the normal manner, acquiring BQlck to protect against concurrent access. Since a batch footprint is limited in this way, the cache-related improvements that were observed in our analysis of uniprocessor batching can be preserved here. This time, the improvements are compounded by the other factors peculiar to SMPs. This activity reflects the workings of a shared run queue model, albeit at a macro level. The thread entity has been replaced by the batch, which has a much longer collective dispatch time, thus reducing the frequency of shared queue access. The expense of retrieving batches from the shared queue remains minimal, so locking duration has not been drastically increased. Accordingly, contention for shared resources should be greatly reduced. In addition, a thread will get scheduled to execute on the same processor on several successive occasions, as the time during which a batch is associated with a processor is substantially longer than the dispatch time of a single thread. There is therefore a greater chance of the
K . Debattista et al. / Cache-AffinityScheduling for Fine Grain Multithreading
131
thread finding a portion of its working set intact in the processor’s cache on being scheduled, especially if the combined working set size of the threads in the batch does not exceed the cache size.
6 Wait-Free Batch Scheduling As with smpbatch-smash, wf-smash makes use of batches as a coarser grain entity to reduce contention and enhance locality. wf-smash is particularly scalable in terms of batch size. In fact results demonstrate the algorithms work well even when migrating single threads for fine grain multithreaded applications [21]. Since wf-smash makes use of wait-free data structures contention is kept at a minimum and batching is used mainly as a vehicle for cachecoherence, even though batches reduce the frequency at which relatively high latency atomic operations (like SWAP and COMPARE AND SWAP) are performed. wf-smash’s scheduling algorithms are discussed in further detail in [21].
Figure 3: Scheduler data structures and information flow in wait-free SMP batching.
wf-smash data structures consist of a batch run queue and overflow batch for each processor which function similarly to unibatch-smash and an additional migration batch. Figure 3 shows these data structures. To understand the purpose of each structure a brief description of the scheduling mechanism is required. As with unibatch-smash the current batch is directly attached to the batch queue. In this case though it also serves to indicate the processor’s state, since the current batch pointer is NULL if and only if a processor lies idle. As with other batch schedulers, whenever a thread launches a new thread, the newly created thread is placed on the current batch and if the current batch’s BS i ze has met MAXBCOUNT the thread is placed onto the overflow batch. In this case when the overflow batch becomes full, an attempt is made to migrate the thread to any idle processor. If sender initiated migration fails, the overflow batch is placed onto the local batch run queue. The migration batch is used when a processor has excess workload and wants to offload a batch, but none of the other processors are currently idle. In this case the excess batch is placed onto that same processor’s migration batch. A processor with a reduced workload can hence obtain more work by scanning the other processors’ migration batches. To improve load balancing, when the
142
K. Debattista et al. I Cache-Affinity Scheduling for Fine Grain Multithreading
// ... launch ARRAY-SIZE/GRANULARITY threads running test-cache () cthread threads [ARRAY.SIZE/GRANULARITY] ; int array [ARRAY-SIZE] ; void cache-test (cthread * ct, int i) { int j , k, g; g = GRANULARITY * i ; for (j = 0; j < PROCESS-LENGTH; j++) { for (k = 0; k < GRANULARITY ;k++) { array [ g + k ] + + ;
} cthread_yield ( ) ;
Figure 4: The benchmark algorithm.
current's batch dispatch counter, BCount expires and the batch run queue consists of more than one batch, the current batch can also be migrated. Batches are migrated in one of two distinct ways. Sender-initiated migration occurs when a processor has an excess batch, in which case it attempts to migrate a batch onto an idle processor. Receiver-initiated migration occurs when a processor runs out of work to do, and this idle processor attempts to migrate a batch from one of the other processors' migration batches. The key to the wait-free algorithms is that the data structures are manipulated locally before being placed onto the globally accessible shared data structure pointers. The shared data structures are accessed only by means of atomic COMPARE AND SWAP and SWAP instructions to maintain their integrity. Further details of the wait-free migration algorithms and other aspects of wf-smash are available in [21].
7 Results In this section we analyse the performance of our uniprocessor batch scheduler (unibatchsmash) and the SMP batching implementations, smpbatch-smash and wf-smash, compared with the standard uniprocessor (uni-smash) and shared run queue implementation shared-smash. Tests were performed on a quad-processor machine with 256MB RAM. The processors were PHI Xeons running at 700MHz, each equipped with 512KB of second-level cache (as well as 16KB instruction and 16KB data first-level caches). The underlying operating system was Linux (kernel version 2.2.12-20). For each of the experiments involving unibatchsmash, MAXBSIZE was set to 32 and MAXBCOUNT was set to 128. In order to test the performance of our batching schedulers, we present a benchmark (Figure 4) which performs a series of operations on the same array data. Three variables are used, an array size (ARRAY-SIZE) which represents the size of the data that will be handled, a variable representing the thread's longevity (PROCESS_LENGTH) and a granularity variable (GRANULARITY). Granularity determines the number of computations between each communication and segments the area of the data to be computed by each thread accordingly. Thus, we can vary the granularity without changing the problem size. Effectively the number of threads launched would be set to ARRAY-SIZE/GRANULARITY.
I43
K. Debattista et al. / Cache-Affinity Scheduling for Fine Grain Multithreading
Process Length 1 - - - - l - - i
I
i
1
--
Process Length 1 (Detail) 0.1
I
unt-smash unibatch-smash --x--shared-smash Q smpbatch-smash a wf-smash -I
uni-smash -+unibatch-smash - -x -
0.08 h
8
0.06
I
; 0.04
v
i= 0.02 Y
0
*, 20
.*, 40
...,
L
60 80 Granularity
100
I
0
120
20
Process Length 4
40
60 80 Granularity
100
120
Process Length 4 (Detail)
2
-8
15
D v)
$
1
g
05
I
0 0
20
40
100
60 80 Granularity
120
I
I
20
40
Process Length 16
I
I
60 80 Granularity
I
I
100
120
Process Length 16 (Detail)
8 7 h
6
wf-smash
ffl
F 5
shared-smash smpbatch-smash wf-smash
-m-
*
u
-
I-
$ 4
v
g 3 2 1
0 0
20
40
60 80 Granularity
100
120
20
Process Length 128
40
60 80 Granularity
120
100
Process Length 128 (Detail)
uni-smash --funibatch-smash - x smpbatch-smash wf-smash
40
wf-smash
-m -
m
8
$ 30
v
W
;20 10 0 0
20
40
60 80 Granularity
100
120
20
40
60 80 Granularity
100
120
Figure 5: Benchmark results for an array size ARRAY-SI ZE fixed at 262,144. SMP schedulers (shared-smash, smpbatch-smashand wf-smash utilise four processors, while the uniprocessor schedulers (uni-smash and unibatch-smash)make use of only one processor.
144
K. Debattista et al. I Cache-Affinity Scheduling for Fine Grain Multithreading
In our experiment, GRANULARITY was varied from 2° (1) through to 28 (128), to span the entire range between very fine granularity, where each increment operation is followed by a context switch, and coarser granularity. PROCESS-LENGTH was varied between 2° (1) and 28 (128), to experiment with both short processes that do not reuse cached data, and very long processes that repeatedly access the same data. ARRAY-SIZE was set to 218 (262,144), representing large data segments that cannot entirely fit in cache. Results are presented in Figure 5. The different plots represent varying process lengths of 1, 4, 16 and 128, and trace granularity values of 1 to 128 against execution time. The curves indicate execution time for normal uni-smash (upper curve), unibatch-smash, shared-smash, smpbatch-smash and wf-smash (lower curve). The results for the uniprocessor schedulers (uni-smash and unibatch-smash) were taken utilising only one processor. The SMP scheduler (sharedsmash, smpbatch-smash and wf-smash) results utilised all four processors available on the system. For unibatch-smash, improvements of up to 100% have been observed for very fine granularities when compared to uni-smash. As expected, the difference in execution time tails off as GRANULARITY increases. For PROCESS-LENGTH from 16 upwards improvements can be observed even at GRANULARITY of 64. It can be seen that as process length increases, the improvements very quickly reach larger and more sustained levels, since caching is better exploited. In fact, an improvement is seen immediately when PROCESS-LENGTH is raised from 1 to 4. Further experiments, not shown in the graphs, demonstrate that improvements can even be seen at a PROCESS-LENGTH of 2. These results indicate that on uniprocessor machines, implicit context switching overheads due to cache misuse are significantly more pronounced for data sizes that do not fit in the cache. Comparing the results of the three SMP schedulers for all PROCESS-LENGTHS it can be clearly noted that the performance of shared-smash suffers when compared to smpbatchsmash and Wf-smash. For a PROCESS-LENGTH of 1, since none of the threads do not benefit from cache re-use, the improved performance of the batch schedulers compared to shared-smp is due mainly to the batching scheduler's reduced contention for shared data structures. For higher values of PROCESS-LENGTH the improvement is more pronounced since when scheduled on any of the SMP batch schedulers threads exploit the cache. When comparing results of the SMP schedulers to the uniprocessor schedulers, we note that for a PROCESS-LENGTH of 1, no speedup is achieved by any of the SMP schedulers up to a granularity of 128. Speedup is eventually achieved at higher granularities (not shown in graph). In the case of shared-smash, the main reason for poor performance is due to contention on the run queue. In the case of the SMP batch schedulers, the initial lack of speedup is due to the use of batches. Since the threads are short lived, not all processors are serviced with batches constantly. When the maximum batch size (MAXBSIZE) is reduced, improvements would be seen at finer granularities, particularly for wf-smash. The performance of wf-smash compared to smpbatch-smash is discussed at length in [17]. For larger PROCESS-LENGTH values a marked improvement in speedup is clearly visible for both smpbatch-srnash and wf-smash due partially to cache exploitation and partially to the reduction in contention for migration. Results for PROCESS-LENGTHS of 16 and 128 demonstrate that both batch schedulers achieve super scalar speedup due to the added advantage of exploiting the caches on each of the four processors. On the other hand, sharedsmash's performance degrades even further when compared to the other SMP schedulers due to the high contention for the shared run queue, further accentuated by the lack of cache exploitation.
K. Debattista et al. / Cache-Affinity Scheduling for Fine Grain Multithreading
8
145
Conclusion and Future Work
We have conducted an investigation into the effectiveness of cache-conscious scheduling using batches. The experimental results obtained indicate that a significant reduction in execution times can be gained through the use of such techniques. However, further experiments with real applications are required to gather information about performance under more realistic conditions. It should be noted that batch-based thread scheduling as presented here may be subject to problems when the pre-set batch size limit is greater than the total number of threads being executed in the application, since all threads would be serviced by a single processor. While this can be advantageous in identifying situations where parallel processing is not worth the effort, pathological cases may well occur in specific applications. Automatic modification of the batch size limit could be envisaged, whereby the batch limit is dynamically set to match the current application's needs. At the moment, threads are grouped into batches by locality and indirectly through communication, so that threads created on a common processor are placed onto the same batch. An additional grouping criterion could be based on the frequency of inter-thread communication or rely on object-affinity [22, 23]. Furthermore, the application programmer could be given the opportunity to manually specify viable thread groupings to override the automatic batching arrangements adopted by the scheduler. The investigations presented here fit into the wider context of a general purpose serverside parallel processing system composed of a cluster of shared memory multiprocessors with high speed user-level CSP communication over Gigabit Ethernet between nodes in the cluster, as well as gigabit speed user-level TCP/IP connectivity to the outside world [24]. Many of the constituent components have already been developed or are at an advanced stage of development. References [1] Anderson, T., Bershad, B., Lazowska, E. and Levy, H. The Performance Implications of Thread Managment Alternatives for Shared-Memory Multiprocessors. In IEEE Transactions on Computers, 38(12): 1631-1644, December 1989. [2] Squillante M. and Lazowska E. Using Processor-Cache Affinity Information in Shared-Memory Multiprocessor Scheduling. IEEE Transactions on Parallel and Distributed Systems, 4(2):131-143, February 1993. [3] Markatos, E. and LeBlanc, T. Load Balancing vs. Locality Management in Shared-Memory Multiprocessors. Technical Report 399, Computer Science Department, University of Rochester, October 1991. [4] Markatos E. and LeBlanc T. Shared Memory Multiprocessor Trends and the Implications for Parallel Program Performance. Technical Report 420, Computer Science Department, University of Rochester, May, 1992. [5] Markatos E. and LeBlanc T. Memory Conscious Scheduling in Shared-Memory Multiprocessors. Technical Report, Computer Science Department, University of Rochester, December, 1991. [6] Vella, K. Seamless Parallel Computing on Heterogeneous Networks of Multiprocessor Workstations. Ph.D. Thesis, University of Kent at Canterbury, December 1998. [7] Valois, J. Lock-Free Data Structures. Ph.D. Thesis, Renesselaer Polytechnic Institute, New York, 1995. [8] Michael, M. and Scott, M. Non-Blocking Algorithms and Preemption-Safe Locking on Multiprogrammed Shared Memory Multiprocessors. Journal of Parallel and Distributed Computing, 1998. [9] Herlihy M. Wait-Free Synchronization. ACM Transactions on Programming Languages and Systems, 11(1):124-149, January 1991.
146
K. Debattista et al. / Cache-Affinity Scheduling for Fine Grain Multithreading
[10] Eager, D., Lazowska, E. and Zahorjan, J. The Limited Performance Benefits of Migrating Active Processes for Load Sharing. In Proceedings of ACM SIGMETRICS, May 1988. [11] Bellosa, F. Memory conscious scheduling and processor allocation on NUMA architectures. Technical Report TR-14-06-95, Computer Science Department, University of Erlangen-Nurnberg, May 1995. [12] Cordina, J. Fast Multi-Threading on Shared Memory Multiprocessors. B.Sc. IT. Final Year Project Report, Department of Computer Science and Artificial Intelligence, University of Malta, May 2000. [13] Vella, K. and Welch, P. CSP/occam on Shared Memory Multiprocessor Workstations. In Proceedings of WoTVG 22: Architectures, Languages and Techniques, Volume 57 of Concurrent Systems Engineering, IOS Press, April 1998. [14] Engler D., Andrews R. and Lowenthal D. Filaments: Efficient Support for Fine-Grain Parallelism. Technical Report TR 93-13, Department of Computer Science, The University of Arizona, February, 1994. [15] Randall, K. Cilk: Efficient Multithreaded Computing. Ph.D. Thesis, Massachusettes Institute of Technology, June 1998. [16] Arora, N., Blumofe, R. and Plaxton, G. Thread Scheduling for Multiprogrammed Multiprocessors. ACM Symposium on Parallel Algorithms and Architectures, pages 119-129,1998. [17] Debattista, K. High Performance Thread Scheduling on Shared Memory Multiprocessors. M.Sc. Dissertation, University of Malta. February 2001. [18] Boosten, M., Dobinson, R.W. and van der Stok, P.D.V. Fine-Grain Parallel Processing on Commodity Platforms. Volume 57 of Concurrent Systems Engineering, pages 263-275. IOS Press, April 1999. [19] Moores, J. CCSP - A portable CSP-based run-time system supporting C and occam. Volume 57 of Concurrent Systems Engineering Series, pages 147-168, IOS Press, April 1999. [20] Hoare, C.A.R. Communicating Sequential Processes. Prentice-Hall, 1985. [21] Debattista, K. and Vella, K. High Performance Wait-Free Thread Scheduling on Shared Memory Multiprocessors. In The 2002 International Conference on Parallel and Distributed Processing Techniques and Applications, Las Vegas, NV. June 2002. [22] Chandra, R., Gupta, A. and Hennessy, J. Data locality and load balancing in COOL In Proceedings of the Fourth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, May, 1993. [23] Fowler R. and Kontotnanassis L. Improving Processor and Cache Locality in Fine-Grained Parallel Computations using Object-Affinity Scheduling and Continuation Passing. Technical Report TR411, Department of Computer Science, University of Rochester, 1992. [24] Cordina, J. High Performance TCP/IP for Multi-Threaded Servers. M.Sc. Dissertation, University of Malta. March 2002.
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
147
Ana CAVALCANTI and Jim WOODCOCK Computing Laboratory, University of Kent at Canterbury, Canterbury, Kent, CT2 7NF, England Abstract. Circus is a combination of Z and CSP; its chief distinguishing feature is the inclusion of the ideas of the refinement calculus. Our main objective is the definition of refinement methods for concurrent programs. The original semantic model for Circus is Hoare and He's unifying theories of programming. In this paper, we present an equivalent semantics based on predicate transformers. With this new model, we provide a more adequate basis for the formalisation of refinement and verificationcondition generation rules. Furthermore, this new framework makes it possible to include logical variables and angelic nondeterminism in Circus. The consistency of the relational and predicate transformer models gives us confidence in their accuracy.
1
Introduction
Modern computing systems typically run on distributed, heterogeneous networks, and are subject to complex constraints on functionality, performance, fault tolerance, security, and timing. If we want such systems to be dependable, then we must address all these issues. There has been much progress in providing sound mathematical foundations for these different aspects of complex systems, but it is usually done in relative isolation. This is because researchers apply the principle of separation of concerns, allowing them to find a thorough solution to a particular problem without being distracted by others. Subsequent researchers can then build on this fundamental research by composing its theories. Two examples of this separation of concerns lie in the theory of program specification and development. First, research on state-based, model-oriented specification languages originally focused on the specification and refinement of sequential software, avoiding the complications of concurrency and distribution. Second, when research on process algebras started in the 1970s, the major schools focused on studying the semantics and theory of concurrency and communication, abstracting from the details of data types and their operations. In both cases, researchers knew that they would have to address the wider concerns in order for the techniques to scale up to describing industrial-sized systems. Recently, there has been considerable interest in bringing these two strands of research together, and in particular combining Z [1, 2] and CSP [3,4] in various ways; Fischer has surveyed some of this work [5]. It is clear, however, that little has been accomplished in understanding the formal development of programs starting from specifications in these combined formalisms. Circus [6, 7, 8] is a language for writing specifications, designs, and programs for concurrent, communicating systems. It combines the languages of Z, CSP, the refinement calculus [9], and guarded commands [10]. As such, it is very convenient for capturing and reasoning about static, dynamic, and reactive aspects of concurrent and distributed systems, such as may be written in Occam and Java. The complete integration of the four languages involved in Circus has been achieved by giving a single denotational semantics that describes all their constructs, in the manner of
148
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
Hoare and He's unifying theories of programming [11]. In Hoare and He's work, a number of different paradigms for programming are studied as a series of linked theories, each formalised in an alphabetised variant of Tarski's relational calculus. A long-standing trend in the formal development and verification of programs is the use of predicate transformers, either directly or as a foundation [10,12, 13, 14]. In the unifying theory, the relational model for sequential programs is linked to a weakest-precondition model. Later, the relational model is augmented to cope with concurrency and communication, and the link to the weakest-precondition setting turns out not to be valid for the augmented model. We present a new predicate transformer: the weakest reactive precondition. It characterises the weakest precondition that guarantees that a given condition holds in all observable states of a reactive program. We define the weakest reactive precondition of a unifying theory relation that defines a reactive system. From this, we calculate a weakest reactive precondition semantics for Circus. This new semantic model is a convenient step towards the complete justification of our extension to an existing refinement calculus for Z [15] that includes all Circus constructs [16, 17]. Moreover, the weakest precondition semantics is a natural starting point for the proposal of rules for verification-condition generation for Circus. Finally, as an added benefit to the effort of calculating a new semantics for Circus, we now have the possibility of modelling logical constants and angelic nondeterminism, which are important for refinement techniques. The two models also provide confirmation for the precision of each other. In [18], laws are presented that completely characterise Occam's semantics, and which are cast in terms of the denotational semantics of [19], although no proof of equivalence was carried out. The laws presented in that work, however, are equalities; they aim at characterising the semantics of the language, instead of supporting the development of programs. In the next section, we give an overview of Circus. Section 3 presents the basic concepts of the unifying theory. The weakest precondition semantics of Circus is the subject of Section 4. Finally, in Section 5 we present our conclusions.
2
Circus
A Circus program is a sequence of paragraphs; each of these may either be a Z paragraph, a channel definition, a channel set definition, or a process definition. In the BNF description of the syntax of Circus in Figure 1, we omit part of the syntax of processes (Proc) and the syntax of communications (Comm), which we exemplify later on in this section. CircusPar* denotes a possibly empty list of elements from the category CircusPar; similarly for PPar*. N+ denotes a comma-separated list of valid Z identifiers (elements of N), and similarly for Exp+. The syntactic categories Par, Schema-Exp, Exp, Pred, and Decl include the Z paragraphs, schema expressions, expressions, predicates, and declarations; their definitions can be found in [1]. Finally, the syntactic category CSExp of channel set expressions contains the empty set of channels {])}, channel enumerations enclosed in {] and |}, and set expressions formed by the usual set operators. A process encapsulates state and behaviour. The basic form of process definition describes the state and operations, mainly as in a standard Z specification. In the context of Circus, the operations are called actions and can be specified using schemas, CSP operators, and guarded commands. The predicate transformer semantics calculated here characterises the behaviour of actions. By way of illustration, we consider a little example due to Hoare [3]. The process WSum below inputs natural numbers from a channel in, and outputs through a channel out the
A. Cavdcanti and J. Woodcock / A Predicate TransformerSemantics for Circus
Program
149
Circuspar*
..-
..- Par I ChanDef I ChanSetDef I ProcDef .._ ..- channel CDecl
CircusPar
1.-
ChanDef CDecl SimpleCDecl
.._ ..- SimpleCDecl 1 SimpleCDecl; CDecl
..-
N+ I N+ : Exp I Schema-Exp
ChanSetDef
.._
chanset N == CSExp
ProcDef
....-
process N
Proc
.._ ..-
begin PPar* 0 Action end I Proc 0 Proc
PPar
.._ ..- Par I N 2 Action .._ ..- Schema-Exp I CSPActionExp I Command
Proc
I Action CSPActionExp
.._ I I I
Skip I Stop 1 Chaos Comm --+ Action I Pred & Action I Action; Action Action 0 Action I Action I[ CSExp]]Action I p N 0 Action Decl Action I Action(Exp+)
.._ ..- N+ : [ Pred, Pred] I N+ := Exp+ I if GuardedActions fi I var Decl 0 Action
Command
..GuardedActions .._
Pred --+ Action I Pred
--+
Action 0 GuardedActions
Figure 1 : Circus syntax
weighted sum of its current and previous input. channel in, out : N; The weights are defined as constants, using the Z notation. /a,b:N The state S records its previous input last and the value to be output Val. process WSum begin S [last,Val : N] We present the action Compute, which takes an input x'? and updates the state accordingly. Compute
[AS; x? : N I Val'
=a
* last i- b * x ? A last' = x? 1
There is a nameless action at the end of a process description, which defines its behaviour; we refer to this action as the main action of the process. In our example, it is as follows. 0
(p X
0
in?x
--+
Compute: out!val+ X )
end We use the prefixing, sequence, and recursion constructs of CSP. First the input is taken through the channel in and into the input variable x. Afterwards, the schema action previously defined is used to update the state. In sequence, the Val component of the state is output through out and the process recurses. We explain the other action operators below through a more substantial example, and in Section 4, as we present their semantics. The CSP operators can also be used to combine processes: their states are conjoined and their main actions are combined using the CSP operator applied. The weakest precondition of a process is that of its main action. The following example, inspired by the Sieve of Eratosthenes presented in [20], illustrates the use of processes.
1 50
A. Cavalcanti and J. Woodcock I A Predicate Transformer Semantics for Circus
Example: The objective of the Sieve of Eratosthenes is to produce the list of prime numbers. In our example, the list is output through the channel out. channel out : N We use the set primes that contains all the prime numbers. Its Z specification is as follows. primes —= { n : N | (V/w : N • m divides n =>• m = 1 V m — n)} In this definition, we use the function -divides-, with the obvious meaning: m divides n if and only if n is a multiple of m. The definition of this function in Z is simple and omitted. So, a number is prime if its only divisors are 1 and itself. The specification is a parametrised process, Primes. Its parameter / limits the size of the list of prime numbers: we are interested only in those primes less than or equal to /. process Primes = I : N • begin PrimesState = [s : FN] Primeslnit = [PrimesState1 \ s1 = (2 . . /) n primes] Output = [ APrimesState; m\ : N | s ^ 0 A m\ = min 5 A ^ = 5 \ { m ! } ] • Primeslnit; I s 7^ 0 —» var m : N • Output; out\m —> X fi end
The process's only state component, specified in PrimesState, is a finite set s of natural numbers. The main action describes the behaviour of the process: first it initialises the state with Primeslnit; then it recursively executes a conditional action. The notation that we use is that of Dijkstra's guarded commands [10]. The initialisation sets s to the set of numbers that are both between 2 and / and also prime. The recursive action then outputs these numbers in ascending order. If s is empty, then the task is finished, and the recursion terminates. Otherwise, a local variable m is declared to hold the next output, which is selected by the operation Output. The schema Output requires that s must not be empty, and the after- value of m—here denoted by m! to emphasise that it is a result of the operation—is set to the minimum value in s; m\ is then removed from s to form the after-value of the state, s1 . Primes is refined to the process Eratosthenes, defined below, which specifies a concurrent implementation of the Sieve of Eratosthenes algorithm. It is defined as the parallel composition of other processes, in the manner of a systolic array. The first process, which we call Start, outputs 2, which is the first prime, through out. Afterwards, it outputs the list of odd numbers, or rather, the numbers that are not multiples of 2, through a channel sievein. channel sievein : N process Start = begin StartState= [n : N] • outl2 —> n := 3; (nX • sieveinln —+ n := n + 2; X)
end
A. Cavalcanti and J. Wbodcoci / A Predicate Transformer Semantics for Circus
151
The second process is a Filter. It takes a prime p through the channel sievein, outputs it, and then outputs the list of numbers that are not multiples of/?, through a channel sieveout. channel sieveout: N process Filter = begin • sievein'lp —» out\p —> LtX • sievein ^m —> if p divides m —> X | -i (p divides m] —> sieveoutlm —> X fi end The idea is that Star? sends through sievein the number 3, which is the next prime after 2, and all the odd numbers. Filter outputs 3 and filters out from sievein the multiples of 3. The multiples of 2 have already been removed by Start, so we are left only with numbers that are neither a multiple of 2 nor 3. The list of such numbers are output through sieveout. We use a piping operator to take the list output through sieveout as input to Filter again. This operator takes the form [n <-» c^} n • P, and we define it as follows, for n > 0, and for a channel m not used in P.
[GI «-» c-2\ n • P = var v :N • v := n]
p,X» i f v - 1 -»P | v > 1 -> v := v - 1; (P[m/Cl] \[{\ m \} }\ X[m/c2] )\{\m[} 11 The operator forms n copies of P, and links them together by connecting the c\ channel of the i-th process to the C2 channel of the (i + l)-th process. The definition achieves this by recursion, limited by the local variable v. On every recursive step, except the last, a copy of P is connected to its right neighbour. The copy has its Ci channel renamed to m; the neighbour (the recursive call of X) has its c% channel also renamed to m. They are then composed in parallel, communicating only through m, which is hidden to prevent interference. The result is a process with a single c^ channel on the left and a single c\ channel on the right. Its internal structure is a linear sequence of processes communicating pairwise on private channels. Of course, P may have other channels, and they are not linked in any way by this operator. For example, COPY (taken from [3]) is a one-place buffer that repeatedly copies its inputs from the left channel and outputs them on the right channel: channel left, right: N process COPY = begin • ( f j , X • leftlx —> rightlx —>• X ) end We construct a three-place buffer as [right <-» left] 3 • COPY. Once the recursion has been completely unfolded, v is irrelevant, so the process is equivalent to the expression: (COPY[m/right]
IW » HI
( ( COPY{m/right] |[ fl m |} ]| COPY[m/left]
)\{\m\}) [m/left]
) \ fl » D This three-place buffer has two external channels, left and right, and two internal ones. Data arrives on the left channel, makes its way via the two internal channels, and finally appears on the right channel. The result is a pipeline of replicated components, like a systolic array. In a similar way, we construct our sieve as a pipeline from a series of / filters. process Sieve = / : N • ([sieveout <-> sievein] I • Filter)
152
A. Cavalcanti and J. Woodcock /A Predicate Transformer Semantics for Circus
At the far end of the pipeline we have the process Finish, which simply takes a value through sieveout and outputs it. process Finish = begin • sieveoutln —> out\n —>• Skip end Now we can connect together our components to form the implementation. process Eratosthenes = / : N • (Start \[ Qsieveinfy ]\ Sieve(l) \[ Qsieveoutfy ]\ Finish) \ Qsievein, sieveout[}• The out channel is not in the synchronisation set of the parallel composition, and so communications from the pipeline's components through this channel occur independently: the results are interleaved. In fact, in spite of this interleaving, the ascending order required by Primes is preserved. Each filter stage of the pipeline outputs on out only after it has received its first communication from its predecessor, but before its first communication to its successor. Together with the initial and final behaviours of Start and Finish, this forces the strict sequencing of the output. Interference on state components is not a problem in the above parallel compositions, since the state of a process is encapsulated. Proving that Primes is refined by Eratosthenes is not within the scope of this paper. Nonetheless, the technique put forward in [16, 17] can be used to calculate Eratosthenes' behaviour from Primes in the refinement calculus style. The semantics presented in this paper is a foundation for that technique. 3 Unifying Theories of Programming The semantics of Circus [8, 21] is based on the unifying theory, but with the Z notation used as the concrete syntax for the relational calculus; thus, a Circus program denotes a Z specification. The use of Z is not essential to our approach, but it is convenient as Z is well-suited to the definition of relations, has a precise semantics, and has supporting tools. In the unifying theory, Tarski's relational calculus is used to give a denotational semantics to constructs taken from different programming paradigms. This common framework allows connections to be established between different paradigms and between a theory and its implementable subtheories. Specifications, designs, and programs are all interpreted as relations between an initial and a subsequent observation of a computing device. Distinguished variables are used to describe relevant observations. The relations are defined as predicates over observational variables and their dashed counterparts; they represent the corresponding values before and after the observation. In accordance with the philosophy of the unifying theory, Circus brings together Z, CSP, and the refinement calculus in a language with a single, coherent semantics. The observational variables describe stability from divergence (okay), termination (wait), a history of interaction with the environment (tr), and a set of refused events (ref). Together with program variables, these observational variables and their associated healthiness conditions define the subtheory of imperative, communicating, sequential processes and designs. The result is a state-based expression of the failures-divergences model with embedded imperative features. Because their semantics are so close, Circus can be used as a development method for Occam programs. Hoare and Roscoe use a style similar to the unifying theory in their semantics for Occam [22]: the meaning of a program is given as a predicate over the values of computations, traces, refusals, and a state variable that records termination and divergence in a similar way to our okay and wait variables. The unifying theory includes a relational definition of the weakest precondition of a sequential mechanism; that definition, however, is not valid for reactive systems. This is actually not surprising, since in the context of sequential programs, weakest preconditions are
A. Cavalcanti and J. Woodcoclc / A Predicate Transformer Semantics for Circus
153
concerned with final and initial states only, but in a reactive system, intermediate states are also relevant. It is in these intermediate states that a reactive process is waiting for interaction with its environment. This motivates our generalisation; the same sort of issue is also discussed by Lamport [23]. In the next section, we propose the weakest reactive precondition of a relation in the subtheory used to model Circus. This characterises the weakest precondition that guarantees that a given condition holds in every observable state: either final or not. Refinement in terms of weakest reactive preconditions is as usual for predicate transformers, and is equivalent to that in the unifying theory. 4
Weakest Reactive Precondition Semantics
The set of channels in scope is relevant to the semantics of the actions. We record them in a channel environment defined as follows. ChanEnv =— ChanName -** Expression As already mentioned, as a meta-language, we use Z with a few extensions that we explain as they arise. A channel environment associates a channel name, an element of the given set ChanName, to a Z expression that gives its type. The relational semantics of actions is given by the function I-]]"4.
I-]]"4 : Action -H- ChanEnv -++ N -+» Schema-Exp It takes as extra parameters a channel environment and the name of the schema that defines the user state. It gives as result a schema over the observational variables: the components of the process and the user states. The process state is defined as follows. ProcessState = [tr : seqEvent; ref : PEvent; okay, wait: Bool] Event is a free type determined by the channels in scope: for each such channel c, we have a constructor C that takes a value of the type of c to an Event. We also use a free type Bool to model true and false; boolean variables as used as predicates, for simplicity. Changes over the process state can only increase the trace of events. ProcStateObs = [ ^.ProcessState \ tr prefix tr'} Other restrictions over the state are more conveniently enforced by the semantic definitions and are discussed later in this section. The semantics of actions are operations over a state that also includes the components of the user state in the process description. State = UserState A ProcessState Here, we assume that the user state is UserState. A change in State is a process observation. ProcObs = ^.UserState A ProcStateObs Both the restriction above on changes to the process state and any existing restrictions on changes to the user state are enforced. Actually, we consider families of schemas ProcObs(USt) and State(USt): one for each user state USt. We need this generalisation because the user state can be extended by input and local variables, and by parameter declarations.
154
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
The definition of [[-J^is as follows. Definition 1 RAl^y USt = [A]]Av7 USt V Diverge(USt) V Wait(USt) The behaviour of A in the situation where the previous operation has not diverged and has not terminated, or rather, in a state where okay and -• wait hold, is characterised by [-fl"^. This function takes the same arguments as [[-J"4, and also gives a schema as result. It is further discussed later in this section. The family of schemas Diverge(USt) gives the semantics when okay is false. Diverge(USt) = [ProcObs(USt) \ -> okay] In this case, there are no guarantees as the previous operation diverged. Finally, if the previous operation has not finished, A behaves as follows. Wait(USt) = [EState(USt) \ okay A wait] The state is not changed by A, because its execution has not started yet. In this work, we present a new semantic function d_]]m' for actions. Definition 2 |_]]vv7': Action -+» ChanEnv -+> N -+* Schema-Exp —> Schema-Exp BAl™" iUStp = [State(USt); Inp \ wrpUSl.$AlA.V>] where wrpus,.p.^ = VState(USt)' • p => ^ and Inp is the declaration of any input variables in scope for A. This function gives the weakest reactive precondition of A, with conditions expressed using schemas. It takes a channel environment 7 and a user state name USt as arguments. It also takes a schema (p to yield another schema expressing the situations in which (f holds in all subsequent states of A: both intermediate and final. The first schema is a relation on ProcObs, and the second defines a restriction on State(USt) and any input variables Inp in scope. The predicate in the schema [[A]]VVT' 7 USt ^ is an application of the function wrpus, to the relational model [A]"4 of A and 0. For historical reasons, we use the dot notation for application of the wrp function, rather than the relational notation used in the unifying theories work or the Z notation. In the sequel, for brevity, we omit the parameter USt when it is clear from the context. The function wrp takes two predicates as arguments; in Definition 2, we are using the schemas [A]]"4 and ? as predicates, a usual practice in Z. For predicates p and t/>, the weakest reactive precondition for/? to establish 0, or rather wrp.p.ip, is that, in all subsequent states, which are characterised by the dashed state components, ifp holds, so does t/>. We can deduce the following from Definitions 2 and 1. Theorem 1 wrp.[lA]]'47 USt.iJj = wAp.|A]]Av'7 USt.t^ A wrp.Diverge.*]) A wrp.Wait.iJf This means that we provide a weakest reactive precondition model for an action A by considering the weakest reactive precondition of Diverge, of Wait, and of the relational model
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
1 55
separately. For Diverge, we have the following. Theorem 2 wrp. Diverge.^ = ( -> okay => V State' • tr prefix Ir1 => 0 ) This means that, in the presence of divergence, -> o&ay, whatever property ^ we want to ensure has to be valid under only the assumption that the trace is extended: tr prefix tr'. For Wait, the result is as follows. The function a gives the set of components of a given schema. Below, we make use of a slight abuse of notation and write i/)[aState / aState'] to mean substitution in ty of every state component for the corresponding dashed one. Theorem 3 wrp. Wait. i/j = (okay A wait => -ij)[aState / aState'] ) If the previous operation has not finished, okay A wait, A cannot establish any property that does not already hold, as it cannot start. The theorem below provides a way of calculating wrp.p.i/), for an arbitrary tp, in such a way that wrp.p is actually applied to a predicate that does not involve undashed variables. Theorem 4 For a list cl of fresh constants, —
(wrp.p.i/j[cl/aState])[aState/cl]
First, the undashed variables of i\) are replaced with fresh constants cl; next wrp.p is calculated for this new predicate i/j[cl/aState}; and afterwards, the undashed variables are restored. Based on this theorem, we calculate w/p.[[A]] ^7 USt.i/j only for predicates ^ that do not involve undashed variables; we call these predicates conditions. The proof of this theorem is a simple application of predicate calculus and substitution properties. In the definition of |A]j jV 7 USt, we use the family of schemas below, which characterises the situation in which the previous action has not diverged and has finished, okay A -> wait, and so A can proceed. Normal(USt) = [ProcObs(USt)
okay A -. wait]
For the calculation of wrp. [[AjA*^ USt.^, it is useful to define wrpn, the weakest reactive precondition that guarantees that p establishes a condition when it is actually activated. Definition 3 .1^ — wrptjs,.Normal(USt)
f\p.ip
We are using Norrnal(USt) above as a predicate. A simple calculation lets us obtain the following result. Theorem 5 wrpn.p.ip = okay A -> wait =4> wrp.tr prefix tr1 A p.i/j
In words, if p can proceed, then i/j has to hold whenever the trace is extended in the way prescribed by p.
156
4. 1
A. Cavalcanti and J. Woodcock I A Predicate Transformer Semantics for Circus
Schema Expressions
The relational semantics of a schema expression is defined as follows. Definition 4 7 USt = SExp A OpNormal V OpDiverge OpNormal = [Normal(USt) \ tr1 = tr A okay A -> wait*} OpDiverge = [Normal(USt); SExp V -. SExp \ -. pre SExp] When a schema action SExp is activated, if its precondition does not hold, then it diverges: no conditions are guaranteed; this is characterised by OpDiverge above. In that schema, the inclusion of SExp V -i SExp has the sole purpose of bringing any input variables into scope. If, on the other hand, the precondition holds, then the action changes the user state as prescribed in SExp and terminates successfully without changing the trace. This is captured by the schema OpNormal. The theorem below gives the weakest reactive precondition of SExp. If the precondition does not hold, SExp diverges and the required condition ip has to hold with the only assumption that the trace is increased. If the precondition of SExp holds, then 0 has to hold when the trace is not changed, okay1 is true and wait' is false, as SExp does not communicate any values and terminates. The quantification over ref means that no restrictions on refusals are guaranteed: SExp refuses all communications. Theorem 6 wrp \SExpl** i USt.il) = okay A -i wait => ( -> pre SExp => (V State' • tr prefix tr1 => ijj) ) A ( Vref : ¥ Event; USt1 • SExp => ip[tr, true, false /tr1, okay/, wait1} ) Proof Hvp.JSExp]]Av7 USt. if} — wrp.(SExp A OpNormal V OpDiverge).^ = wrp.SExp A OpNormal.il) A wrp. OpDiverge. ifr 1
[Definition 4] [property of wrp]
1
= wrpn.SExp A tr = tr A okay A -< watf.if) A w/p/i.-i pre SExp.il) [definitions of wrpn, OpNormal, and OpDiverge] — (okay A -> wait => Vref : P Event; USr1 • SExp => if)[tr, true,false/tr/, okay1, wait1}) A (okay A -i wait =» -. pre SExp =* V State' • tr prefix tr1 =» il)) [definitions of wrpn and wrp, and predicate calculus] = okay A -> wait => ( -i pre SExp => (V State* • tr prefix tr1 => V) ) A (Vref : P Event; USt1 • SExp => t/>[fr, truejalse/tr1, okay1, wait1] ) [predicate calculus] D
In the sequel, we omit simple proofs for the sake of conciseness.
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
1 57
Example: We consider again the process WSum presented in Section 2. It includes the schema action Compute reproduced below. Compute = [ AS; xl : N | val' = a * last + b* x? A last1 = xl } This action updates val, so that it records the weighted sum of the input xl and last with weights a and b, and updates last to record the value of the input. We calculate the weakest reactive precondition for Compute to finish and record as output the value 2. 0 = [ AS | okay' A -i waif A val' = 2 ] By definition of [_] IComputel
WP
, we have the following. 7 S.i!) = [S; *? : N | wrp. [Compute] ^ j
According to Theorem 1 , we can calculate wrp. [Compwte] .t/j in terms of the weakest reactive precondition for [Compwte] N , for Diverge, and for Wait. We start with Diverge. wrp. Diverge.il> = -> okay => V State' • tr prefix tr1 =>• ofary' A -> wa/V A va/' = 2
[Theorem 2]
= -i okay => /afae
[predicate calculus]
= ofccry
[predicate calculus]
This means that Diverge, which is characterised by -i okay, can only establish if) if okay; therefore, Diverge cannot establish the condition itself. Divergence has to be avoided if ^. For Wait, we proceed as follows. wrp. Wait. 41 = okay A wait =>• okay A -> wait A va/ = 2
[Theorem 3]
= -i okay V -> wazf V (o&ay A -i waiY A va/ = 2)
[predicate calculus]
= -i (okay A vrazY)
[predicate calculus]
Even though it is possible for Wait to establish val' = 2, if val is already 2, HiwY cannot terminate, and ip requires that it does; so, we also have to avoid okay A wait to obtain ip. Finally, we consider [Compute] A/'. We observe that pre Compute = true.
wrp. = okay A -i wa/f => [Theorem 6] 1 ( -i rrwe => \/ State' • tr prefix tr => '0 ) A ( V «/' : W Event] S' • val' = a* last + b* x'? A last' = xl =» va/' = 2 ) = ofeay A -i wai'f =^> a * /a*? + b * x? = 2
[predicate calculus]
In summary, based on Theorem 1, we can conclude the following. wrp . [ Compute^ . ip = okay A -• (okay A wait] A (ofary A -> wait =$• a * last + b * x? = 2)
[from above]
= okay A ~ wait A (okay A -i wazY =r- a * last + b * x? = 2)
[predicate calculus]
= okay A -i waiY A a * /a,yf + ^ * x? = 2
[predicate calculus]
To establish ip, Compute must be able to start (okay and -> wait), and /as/ and x? have to be adequate (a * last + b*x? = 2).
158
4.2
A. Cavalcanti and J. Woodcock I A Predicate Transformer Semantics for Circus
CSP Expressions
The weakest reactive precondition of the CSP processes Skip, Stop, and Chaos are as follows. Theorem 7 For a condition ip, okay A -• wait => V ref : P Event • il)[tr, true, false, USt/tr1. okay? . wait', US/] okay A -« wait =>• V ref : P Event • 0[fr, true, true, USt/tr1 .okay1'. wait". USf] wrp. \ChaosY1'*" 1 USt.ip = ( okay A -• wait =*> V State' • tr prefix tr1 =*• t/> ) is the process that terminates successfully, immediately. The required condition $ has to hold when the trace and the user state are not changed, and okay1 is true and wail1 is false. All communications are refused, and so ref is universally quantified. For example, for any event e, the weakest reactive precondition of Skip with respect to tr1 = tr ^ (e) is false. We also obtain precondition false for the conditions -• okay1 and wait1, because Skip cannot change the trace, diverge, or deadlock. On the other hand, the weakest reactive precondition with respect to val' = 2 is okay A -> wait =>• val = 2, since Skip does not change the state. For Stop, which deadlocks immediately, the difference is that the condition has to hold when wait' is true. Finally, the definition for Chaos, which diverges immediately, is similar to that for the case when the precondition of a schema expression action does not hold. For sequences, we have the following result. Theorem 8 For a condition if), H7p.[[A]]Av7 USt.(il) A (okay1 A - wait1 => (wrp.^B^
USt.il))'))
Usually, in weakest precondition semantics, the semantics of sequence is functional composition, so that the weakest precondition for a sequence to establish a postcondition is the weakest precondition for the first component to establish the weakest precondition for the second component to establish the postcondition. For weakest reactive preconditions, however, not only the final state is relevant: all intermediate states, including those of A, are relevant. Theorem 8 requires that t/> holds during the execution of A. Moreover, if A finishes (okay A -> waif), then the weakest reactive precondition of B with respect to t/> also has to hold. Since conditions are given in terms of dashed variables and preconditions in terms of undashed variables, the free variables of w/p.[[fl]]^A 7 USt.if) need to be dashed. For a predicate p, we define p' to be the predicate obtained by dashing the free occurrences of undashed observational variables in p. A relational definition for the prefixing operator is presented below: sequential composition of a communication with the prefixed action. The communication can be observed before it actually takes place, and afterwards. These observations are described in the sequel by CWaiting(USt) and CDone(USt). The communication itself, Comm(USt), is described by the disjunction of these schemas. A communication takes as input the set accEl of acceptable events. Before the communication takes place, the state and the trace do not change, the acceptable events cannot be refused, and the action does not diverge or terminate. After the communication, the trace is augmented by one of the acceptable events and the action terminates successfully. The state is still not changed.
A. Cavalcanti and J. Woodcock/A Predicate Transformer Semantics for Circus
159
In the definition of the semantics of prefixing, we use a substitution notation for schemas not directly available in Standard Z, although it is in Z/Eves [24]. For a schema S, with a component v of type T, and an expression e of the same type, we denote by S[v := e] the schema 3 v : T • [v = e] A S. For an output prefixing, the set of acceptable events of the communication contains only the event obtained by applying the channel, which is an event constructor, to the output value. For an input prefixing, this is the set of events resulting from applying the channel to all the values of its type, obtained from the channel environment 7. A communication also outputs the value el communicated. For an output prefixing, this information is ignored by hiding. For an input prefixing, this information is used to initialise the input variable. Input variables are modelled as extensions to the state; they are local to the prefixing, and so they and their dashed counterparts are quantified. Definition 5 . CWaiting(USt) Normal(USt) accE? : P Event EUSt tr' = tr A accEl n ref okay' A wait'
.CDone(USt) Normal(USt) accEl : P Event e\ : Event EUSt el e accEl tr' = tr~ {el} okay' A -i wait'
Comm(USt) = CWaiting(USt) V CDone(USt) \c\e -» /4J AV 7 USt = (Comm(USt)[accE! := {c(e)}} \ el) ? ^Aj USt lc?x -» A]]Av7 USt = 3x,xf:jc» Comm(USt)[accE?, el :={>•: 7 c • c(y)}, G(JC')] 5 W"^ USt The weakest reactive precondition of the output prefixing operator is given below. Theorem 9 For a condition ijj, wrp.^de -> ^^"7 USt.1(1 = wrp.CWaiting[accE7 := {c(e)}].^ A wrp.CDone[accE? := {c(e}}} \ el(ip A (wrp.^A^
USt.^)')
The condition has to hold before the communication takes place, after it takes place, and during A, as well. For input prefixing, we have a similar result, but since the input is a local variable, it is universally quantified as usual in weakest precondition semantics.
1 60
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
Example: We consider the action outlval —»• Skip, in the context of the process WSum, and calculate its weakest reactive precondition with respect to tr1 = tr. wrp\out\val -> Skip^^ USt.tr1 = tr = (wrp.CWaiting[accE? :- {out(va/)}].r/ = tr0 A [Theorems 4 and 9] wrp.CDone[accE? := {out(vo/)}] \ el.^r1 = tr0 A (wrp\Skip\A~i USt.tr1 = fr 0 )') We use a fresh constant fr0 to replace the occurrences of tr in t/>. We calculate the weakest reactive precondition of Skip with respect to tr1 = trQ separately. USt.tr1 = tr0 ip§'v~f VSt.tr1 = tr0 A wrp.Diverge.tr1 = tr0 A wrp.Wait.tr1 = tr0 [Theorem 1] — (okay A -> wait =$> tr = tr0) A okay A (okay A waif =3> tr = fro)
[Theorems 7,2, and 3] = okay A tr = tr0
[predicate calculus]
This means that the previous operation must not diverge and the trace has to be already trQ for Skip to guarantee tr1 = fro. For the weakest reactive precondition of CWaiting, we have the following result. wrp.CWaiting[accE1 := {out(va/)}]./r/ = tr0 = V State' • [definitions of wrp and CWaiting] 1 Normal A USr" = USt A tr = tr A (out(va/)} n ref = 0 A okay1 A waif => = okay A -i wa/r =» Vref : P Event • (OUt(va/)} n ref - 0 => tr = tr0 [predicate calculus] = okay A -i wait => ((3 re/' : P Event • (out(va/)} n ref = 0) =» fr = fr0) [predicate calculus] = okay A -> waiY => tr = tr0
[predicate calculus]
This means that the trace has to be already tr0 for CWaiting to establish the condition; CWaiting cannot change the trace. Finally, the weakest reactive precondition of CDone can be calculated as follows. wrp.CDone[accE? := {out(va/}}] \ el^tr1 = tr0 A okay1) = V State' • [definitions of wrp and CDone] Normal A USt1 = USt A tr1 = tr ~ (out(va/)) A okay/ A -> waif => tr1 = fr0 A okay — okay A -« wait => tr ~ (out(va/)} = fr0 1
[predicate calculus]
In words, for CDone to guarantee tr = tr0, we need that tr0 differs from tr just in that it has
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for CiKUS
161
an extra element: the output. The calculation of the weakest reactive precondition of outlval —>• Skip can proceed as shown below. wrp\out\val —> Skip^AMj USt.tr' = tr = ((okay A -> wait =$> tr = fr 0 ) A (okay A -> wa^ =» tr ~ (out(va/)) = tr0})[tr/tr0] = -i (okay A -> wait]
[by the calculations above]
[predicate calculus]
In summary, if the output prefixing starts, we cannot guarantee that the trace is not changed, as should be expected. The weakest reactive precondition semantics of a guarded action is rather intuitive. Theorem 10
wrp (p => ivrpn.[A]] ^7 USt.if) } A ( -> p =>• wrp.pto/?]] ^7 USt.il)
If the guard/? holds, then;? & A behaves as A. Otherwise, it behaves like Stop. The behaviour of a parametrised action D • A is that of A taken in an extended state DUSt that includes the components declared in D. Instantiation fixes the value of these components. Theorem 11 wrpUSl.lD • AI^
The function Of extracts from a declaration the set of variables it introduces. In A(e), we assume that the parameters of A are given by D. The correspondence between the parameters and the arguments e is positional: in the substitution we use the set aD as the list of the parameters in the order they are declared. 4.3
Commands
The weakest reactive precondition of a specification statement x : [pre,post] does not insist on termination. The list aUSt\x includes the variables of the user state other than x. Similarly, aUSt' \ y lists their dashed counterparts. Theorem 12
okay A -i wait =3(pre A (Vref : P Event; x1 : X • post =>• V>[fr, true, fake, (USt \ xj/tr*, okay', wait, (USS \ x1)}}) V (-ipre f\ V State' • fr prefix tr' =>• ^) where X is the type ofx. Roughly, if the precondition holds, then for all final values of A; that satisfy the postcondition,
162
A. Cavalcanti and J. Woodcock /A Predicate Transformer Semantics for Circus
ii> is required to hold. This is the usual weakest precondition semantics of specification statements. There are, however, a few extra issues. First, no restriction on refusals are guaranteed, since the specification statement terminates. Second, ip cannot require changes to the trace or to state components other than jc; nor can it require divergence or nontermination. Finally, if the precondition does not hold, ^ has to hold under only the assumption that tr prefix tr1. The weakest reactive precondition for the miraculous x : [true.false] to establish any condition i/> is true, as expected. For abort, x : [false, true], we have okay A -< wait =*> ( V State1 • tr prefix tr1 => iA )
This is the same we obtained for Chaos. In other words, the semantics of Chaos and abort is the same, as should be expected. The semantics of an assignment x :— e is given by substitution as usual. Theorem 13 wrp\x := eJAv> USt.il; = okay A -i wait => Vref : IP Event • il>[tr, true, false, e', (aUSt \ AT)/*/', okay/, wait'.x', (aUSt' \ x1)} As with specification statements, an assignment refuses all communications, does not change the trace, terminates immediately, and changes no variables other than x. The semantics of conditionals is as follows. Theorem 14
wrp. |[if D i • gi -» Ai fij-4! USt = (-. (V i • gi) =>• wrp.lChaosl**'^ USt.il>) A (A i • gt =>• wrpn\A^A^ USt.il>) If none of the guards is valid, then the conditional aborts: it behaves like Chaos. Otherwise, any of the valid guards can be chosen, and so the condition has to be guaranteed for each of the associated actions. For blocks, we have a standard definition. Theorem 15
The weakest reactive precondition of the body of the block is taken in an extended state xUSt that includes the declared variable. 5
Conclusions
The refinement calculus [9, 15] has been influential in understanding the development of sequential programs from their formal specifications. One of the aims of the Circus project [6, 7, 8] is to extend this calculational approach to reactive programs. Since the sequential refinement calculus is based on weakest precondition semantics, in this paper, we have presented a new notion of weakest reactive precondition, wrp, to reason about concurrent programs in a similar manner. Based on this new notion and on a relational semantics for Circus, we have calculated a weakest reactive precondition semantics for a subset of this language. The notion of refinement in the predicate transformer model is equivalent to that in the unifying theory: the basis of our relational model.
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
163
A weakest precondition is an extreme solution to a Hoare triple with one unknown: the precondition. The idea of extending Hoare logic, weakest precondition semantics, and even the refinement calculus to the development of parallel programs has a long pedigree. The work of Owicki and Gries [25, 26, 27, 28, 29] marks the first significant attempt to extend Hoare logic. Their theory extends Hoare's deductive system for proving the partial correctness of sequential programs [30] by adding parallelism in the form of co-blocks, synchronisation, mutual exclusion, and wait statements. In their method, parallel processes are considered in isolation and a proof of sequential correctness is obtained. These proofs must then be shown to be free from interference: no wait statement or assignment outside a wait statement in one process interferes with the proof of any other. The specification of the parallel program is then the conjunction of the preconditions and the postconditions of the individual components. An important point is that this method is not compositional, since the verification of each process P requires an interference-freedom proof involving information not in P's specification. Lamport also generalised Hoare logic for concurrent programs [31]. The Hoare triple P { S } Q means that if P is true before the execution of S, and S terminates, then Q holds. Lamport changed this meaning in the presence of concurrency: if execution is begun anywhere within 5 with P true, then P remains true until S terminates, whereupon Q is true. In this way, Lamport suggests that a program is better thought of as an invariance maintainer, rather than as a predicate transformer. Although this technique is compositional, it requires a global invariant, and parallel processes must each have this as their specification. Some independence is gained from structuring this invariant with auxiliary control-flow predicates, but the logic is awkward to use as a method of specifying module interfaces. In [23], Lamport generalises the weakest liberal precondition and the strongest postcondition to the weakest and the strongest invariant. The method allows the verification of a concurrent program to proceed without knowing the granularity of atomicity of operations, and behavioural arguments to be replaced by assertional reasoning. The difference between those predicate transformers and wrp lies in the set of observable states. In our semantics, we consider the relationship between the initial state of a process and any subsequent state, whereas Lamport considers the relationship between any state in which execution is resumed and the subsequent state in which it is suspended. Back [32] applies the refinement calculus to the stepwise refinement of parallel and reactive programs. Action systems are used as the basic program model: they are sequential programs that can be implemented in a parallel fashion. Reactive programs are data refined using techniques originally developed for the sequential refinement calculus. The main difference between our approach and Back's arises from the fact that Circus is based partly on a process algebra, not on an action system. In an action system, control flow is encoded in state information, with variables playing the role of program counters; in a process algebra, control flow is described using the process combinators of the algebra. Therefore, our predicate transformers and refinement calculus deals with these combinators. In our work, we have derived our weakest precondition semantics from an existing relational semantics. As a result, our weakest reactive precondition calculator is very close in spirit to predicate transformers for sequential programs. Moreover, since our model includes refusal sets (the ref observational variable), we can reason about liveness as well as safety properties. We are currently seeking to extend our calculations to obtain compositional definitions for some of the operators. We plan to use this semantic model to justify fully the refinement calculus presented in [16,17]. We are also starting an effort to propose rules for verification-condition generation based on this work. We plan to develop a series of examples and case studies in the use of wrp. Our confidence in the semantics of Circus has been greatly increased through this work.
164
A. Cavalcanti and J. Woodcock /A Predicate Transformer Semantics for Circus
In the unifying theory, the model of reactive programs is further constrained by a number of healthiness conditions to obtain theories for particular mechanisms of synchronisation and communication. We expect that the wrp models corresponding to these restricted theories also satisfy a number of healthiness conditions of their own. It is also our intent to further explore these relationships. Acknowledgments We would like to thank Augusto Sampaio and the anonymous referees for their inspirational suggestions. This work is partially supported by EPSRC grant GR/R43211/01 on "Refinement calculi for sequential and concurrent programs" and the QinetiQ grant CU009-019346 on "Advances in formal modelling and concurrency as represented by the Circus language". Ana Cavalcanti is partly supported by CNPq grant 5207613/98-0 and by Universidade Federal de Pernambuco. References [1] J. M. Spivey. The Z Notation: A Reference Manual. Prentice-Hall, 2nd edition, 1992. [2] J. C. P. Woodcock and J. Davies. Using Z - Specification, Refinement, and Proof. Prentice-Hall, 1996. [3] C. A. R. Hoare. Communicating Sequential Processes. Prentice-Hall International, 1985. [4] A. W. Roscoe. The Theory and Practice of Concurrency. Prentice-Hall Series in Computer Science. Prentice-Hall, 1998. [5] C. Fischer. How to Combine Z with a Process Algebra. In J. Bowen, A. Fett, and M. Hinchey, editors, ZUM'98: The Z Formal Specification Notation. Springer-Verlag, 1998. [6] J. C. P. Woodcock and A. L. C. Cavalcanti. A Concurrent Language for Refinement. In A. Butterfield and C. Pahl, editors, IWFM'Ol: 5th Irish Workshop in Formal Methods, Dublin, Ireland, July 2001. [7] J. C. P. Woodcock and A. L. C. Cavalcanti. The steam boiler in a unified theory of Z and CSP. In 8th Asia-Pacific Software Engineering Conference (APSEC 2001), 2001. [8] J. .C. P. Woodcock and A. L. C. Cavalcanti. The Semantics of Circus. In D. Bert, J. P. Bowen, M. C. Henson, and K. Robinson, editors, ZB 2002: Formal Specification and Development in Z and B, volume 2272 of Lecture Notes in Computer Science, pages 184 - 203. Springer-Verlag, 2002. [9] C. C. Morgan. Programming from Specifications. Prentice-Hall, 2nd edition, 1994. [10] E. W. Dijkstra. A Discipline of Programming. Prentice-Hall, 1976. [11] C. A. R. Hoare and He Jifeng. Unifying Theories of Programming. Prentice-Hall, 1998. [12] R. J. R. Back. On The Correctness of Refinement Steps in Program Development. PhD thesis. Department of Computer Science, University of Helsinki, 1978. Report A-1978-4. [13] C. C. Morgan, K. Robinson, and P. H. B. Gardiner. On the Refinement Calculus. Technical Monograph TM-PRG-70, Oxford University Computing Laboratory, Oxford - UK, 1988. [14] A. Kaldewaij. Programming: The Derivation of Algorithms. Prentice-Hall, 1990. [15] A. L. C. Cavalcanti and J. C. P. Woodcock. ZRC - A Refinement Calculus for Z. Formal Aspects of Computing, 10{3):267 - 289, 1999. [16] A. C. A. Sampaio, J. C. P. Woodcock, and A. L. C. Cavalcanti. Refinement in Circus. In Proceedings of Formal Methods Europe FME'2002, Lecture Notes in Computer Science. Springer-Verlag, 2002. To appear.
A. Cavalcanti and J. Woodcock / A Predicate Transformer Semantics for Circus
165
[17] A. L. C. Cavalcanti, A. C. A. Sampaio, and J. C. P. Woodcock. Refinement of Actions in Circus. In Proceedings ofREFINE'2002, Eletronic Notes in Theoretical Computer Science. Eletronic Notes in Theoretical Computer Science, 2002. Invited paper. To appear. [18] A. W. Roscoe and C. A.R. Hoare. The Laws of Occam Programming. Theoretical Computer Science, 60(2): 177-229, 1988. [19] A. W. Roscoe. Denotational Semantics for Occam. In Seminar on Concurrency, volume 197 of Lecture Notes in Computer Science, pages 306 - 329, 1984. [20] C. A. R. Hoare. Proof of a structured program: The Sieve of Eratosthenes. BCS Computer Journal, 15(4):321-325, 1972. [21] J. C. P. Woodcock and A. L. C. Cavalcanti. Circus: a concurrent refinement language. Technical report, Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD UK, 2001. [22] C. A. R. Hoare and A. W. Roscoe. Programs as executable predicates. In Proceedings of the International Conference on Fifth Generation Computer Systems 1984 (FGCS'84), pages 220-228, Tokyo, Japan, November 1984. Institute for New Generation Computer Technology. [23] L. Lamport, win and sin: predicate transformers for concurrency. Report 17, Digital Sytems Research Centre, 1989. [24] I. Meisels. Software Manual for Windows Z/EVES Version 2.1. ORA Canada, 2000. TR-97-5505-04g. [25] S. Owicki and D. Gries. Axiomatic Proof Techniques for Parallel Programs. PhD thesis, Cornell University, 1975. [26] S. Owicki and D. Gries. An axiomatic proof technique for parallel programs I. Acta Informatica, 6:319 •340, 1976. [27] S. Owicki and D. Gries. Verifying properties of parallel programs: an axiomatic approach . Communications of the ACM, 19(5):279 - 285, 1976. [28] E. W. Dijkstra. A personal summary of the Gries-Owicki theory. In Selected writings on computing: a personal perspective. Springer-Verlag, 1982. EWD554, March 1976. [29] D. Gries. An exercise in proving parallel programs correct. Communications of the ACM, 20(12):921 930, 1977. [30] C. A. R. Hoare. An Axiomatic Basis for Computer Programming. Communications of the ACM, 12:576 - 580, 1969. [31] L. Lamport. The "Hoare Logic" of concurrent programs. Acta Informatica, 14:21 - 37, 1980. [32] R. J. R. Back. Refinement of parallel and reactive programs. In Proceedings of the Summer School on Program Design Calculi, Lecture Notes in Computer Science. Springer-Verlag, 1992.
991
This page intentionally left blank
Communicating Process Architectures 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
167
Keconnetics: A System for the Dynamic Implementation of Mobile Hardware Processes in FPGAs Ralph MOSELEY Computing Laboratory, University of Kent, Canterbury, KENT CT2 7NF [email protected] Abstract. The capacity to utilise FPGA designs in such a way that they are compositional, mobile, and 'interactive', offers many new possibilities. This holds true for both research and commercial applications. With the system described here, hardware becomes as easy to distribute as software and the mediating links between the two domains allow for manipulation of physical resources in real time. Designs are no longer monolithic 'images' downloaded at one time, but mobile entities that can be communicated over a distance and dynamically installed at run-time many times and at many places. Such twin domain designs can be as complex as a processor, or as small in scale as a few logic gates. The run-time system, Reconnetics, provides an environment of high-level control over such elements, which requires little knowledge of the underlying hardware technology.
1 Introduction Field Programmable Gate Array (FPGA) technology is improving rapidly in terms of gate size, on-chip memory and support infrastructure [1]. Conventional cycles of design for such large devices can prove to be time-consuming and inflexible. It is now possible to interact with reconfigurable devices in such a way that dynamic configuration is fast and unobtrusive, even to an active unit. The run-time system and associated infrastructure presented here allows the user to control FPGAs locally or remotely. This functionality is further enhanced by the utilisation of special design entities. These are developed using any conventional design tool and further processed by software which generates a program. In this new form the design has both a software and physical hardware component. The software part provides a mediating link to communicating systems and the means to implement the circuit, which is detailed within its code. This paper details how the system is constructed, its basic model and its functionality. It also introduces the ideas behind its transferable components, here defined as mobile hardware processes (MHP). Firstly, the background of the work is detailed and its relation to the current research in the area. The system itself is then presented in overview, together with a more detailed explanation of the main sub-systems. An exploration of the uses of the system is then given, together with examples. Finally, some further developments and direction are noted.
168
R. Moseley / Reconnetics
2 Background From the early developments of programmable electronics, the goal has always been to capture some of the flexibility of logic and memory present in conventional software based systems. FPGA technology took this a step further by being able to implement complex circuits, logic and memory, in ways that were easily modifiable. A FPGA can be visualised as a 2D-matrix, which contains a great many controllable resources. Using these resources, complex logic circuits can be built up. Normally the user of a system will develop at a high-level, utilising a design environment that has either Hardware Description Language (HDL) or graphic design capability. The languages used can describe the circuits structurally or behaviourally. It has become possible, using languages based on C and occam, to describe the functionality of a circuit algorithmically, that is, there becomes little difference between a program written for a conventional computer and a FPGA. Where the conventional computer program may be compiled to native code, the FPGA-targeted program is synthesised to gates and finally placed on to the surface of the matrix. For a large design this can take many minutes. Normally, a design is a discrete block, not divisible or alterable, fed into the FPGA at some point, such as power-up. Relatively recent developments allow configuration of these devices to take place while it is active, modifying only specific areas. For example, a circuit may take up half of a device's resources in a roughly rectangular area, on one side of the matrix. This may be active, in the sense that it is powered up and functioning, while the free half of the circuit is configured with a new design. There is still a large gap between the design capabilities of FPGA hardware and the software which supports it. Designs are, in the main, usually downloaded as one large image. Even though the general view of the engineer is based on components or modules, this is relegated to a 'static' design-time. Previous work in this area has realised both the difficulties and possible applications of such devices to be configured on-the-fly [2]. The work has primarily focused on developing a means of mapping designs at run-time[3], introducing ideas such as logic caching [4] [5] and dynamic synthesis [6]. Compton and Hauck offer an overview of reconfigurable systems to date [7]. The work detailed in this paper builds upon efforts by Xilinx and others, utilising the dynamic reconfiguration capabilities in the Virtex series ICs [8]. The scope of this work is centrally defined by the concept of introducing flexibility to the FPGA design process in the form of mobility. For example, it should be possible to place a design as a reusable compositional unit anywhere on the surface of a FPGA, or transfer it over networks. Going a step further than this, the work attempts to bridge the gap between the physical circuit and any software which engages with it. This may be for information exchange, mobility of the entity itself, or for manipulating the actual form of the hardware object. To allow access to low-level resources and fast manipulation of the bitstream, Xilinx JBits Java classes were used, as well as other techniques, developed as the system evolved. It is beyond the scope of this paper to explain the usage of JBits or discuss the hardware mechanisms of low-level dynamic reconfiguration [9]. Such information is available elsewhere - particularly through the Xilinx website [10] and its accompanying library of applications papers. While JBits provided a means of low-level access to resources and to dynamic reconfiguration options, methods had to be developed for manipulating processes as a whole and the individual components contained within them. With the JBits classes providing access to low-level resources and routing, there is still a question of handling various collections of resources which form a process; a very complex, physical object.
R. Moseley / Reconnetics
169
Methods were required for copying CLBS’ and associated wiring within and outside of their ori inal bitstreams. Mechanisms were also added for manipulating connection trees, BRAMs!?[ 111, physical pads and the capability to read in hex data files. Many high-level features of JBits were not required, but can be integrated if so desired; for example, ‘cores’ can be turned into MHPs for manipulation in real time at a high level. 3 System Overview
Figure 1. Reconnetics overview
The Reconnetics system allows dynamic interaction with a reconfigurable device. The user supplies designs, as a synthesised ‘bitstream’, written with any suitable HDL or graphic tool. The design itself can be as large and as complex as required, although it would make more sense to build from smaller blocks, compositionally. These are captured and placed in an archive for later utilisation by the design engine. The engine is directed by a high-level user program, to load, place and interact with these processes, as well as manage on-board resources. design, synthesise target
produced bitstream to Capture system
Design Control Program
Design Engine/ Supervisor
Figure 2. Development cycle
The language used to control the engine is deliberately simple and focused at a high level; all low-level wiring and resource handling is taken care of by the system itself. A user needs only to decide which processes need to be connected together and what kind of interaction is required. It may be that only placement is desired, or diagnostic work. This is possible but a wider scope of control and communication is also offered. For example, processes can be interacted with by Reconnetics in such a way that a user can supply input
’ Configurable Logic Block, a programmable unit containing logic, flip-flops and multiplexors. Block Random Access Memory, a discrete memory unit, separate to the main programmable matrix area.
170
R. Moseley / Reconnetics
directly to the hardware without external I/O connections or other methods. This is true also for collecting output, making the system particularly suitable for development work. The capture system analyses a bitstream, which can be the product of any design method or language (HDL, schematic etc.). Its task is to produce a Java class which can rebuild the circuit at any given point on the surface of an FPGA. This includes status of clocked logic, routing and any other necessary resource usage. The class is primarily produced for use by the run-time system but is sufficiently self-contained to be incorporated into any user written Java program too. After capture, the class is placed in an archive known as the pool, which is simply a directory. Together, when instantiated in their appropriate environments, they are known as a MHP. The generated Java program's principal function is to act as a vehicle for the circuit and to reproduce it at any given point on an FPGA matrix. Another function of the Java program is to act as a mediator between the Design Engine/Supervisor (DES) and the physical circuit. A user can communicate with the process in this way, collecting information or directly manipulating the circuit's form or connections. DES manages the device itself, along with organisation of the processes. It can run userwritten scripts which direct activities, these are known as design control programs (DCP). All low-level resource management is left to DES, making the actual command language very simple, an example of which is given later. The language also includes various diagnostics, if required. The server system provides the ability to communicate over TCP/IP to device nodes on the current machine which is running Reconnetics, or a device at a remote location identified by an IP address. It is the job of the server system to synchronise between Reconnetic^ internal representation of a device and the device itself.
4 Capture This section considers the capture system in detail; if you do not require a low-level perspective, then re-join at section 53. As stated above, the main task of the capture sub-system is to produce Java classes which can reproduce a circuit at some other location. It must capture all essential design data, such as the status of multiplexors, look-up tables (LUTs) and routing. A large number of resource types are available in devices - some of which are not mapped into JBits at this time. During analysis of a circuit this discrepancy led to incomplete route tracings. This was resolved by building in a certain amount of design knowledge. A first scan by the system makes a list of incomplete nets. Solutions to an incomplete net can be found by recognising specific attributes of the end resource and by being aware of disrupted routes elsewhere, which conform to certain criteria. In this way, the system can apply a patch to fix an incomplete tracing. This method has not yet been known to fail in its attempt to re-establish a problem design. It is possible that this technique could be applied to correct corrupted bitstreams, or instantiated designs, up to a certain point. A further optional stage is available, called pre-capture, which adds connection points to a process, allowing several modes of communication with other processes, or external I/O. Further details are provided below.
3
A suitable starter guide is available from Xilinx [12].
R. Moseley / Reconnetics
4.1 Scanning Here resource usage is determined and mapped. Three different areas are checked: the CLBs, BRAMs and external I/O. The supplied bitstream is analysed and each resource found to be in use within the design is marked for further examination. When the entire matrix bitstream has been scanned, an object is built which contains the details of the used resources and their relationships to each other, such as positioning. This object is a sub-matrix of the larger containing matrix within the bitstream. The sub-matrix can be manipulated in terms of its geometry; it can use its default shape, or have new positioning imposed on it, while still maintaining correct connections. 4.2 Code Production The scanning procedure gives a basic map of the CLB and BRAM usage. This map is then used to analyse, in greater depth, specific resources and their settings. Finally, a program is generated which describes the implementation. There are many types of resource contained within each of the basic areas, whose state has to be captured and translated into program statements, such as multiplexors, switches, LUT arrays and routing, some of which may be to external I/O points. The Java program produced is made up of several key methods. These allow a control program to build, destroy, communicate and route a process's physical circuit. From the point of view of the average user of the system, this method information is not required. DES and the command language take care of such detail. The basic 'shape' of the design is kept intact; that is, the map which shows resource usage also holds the basic relational details. At the time of code production such detail is captured by producing offset row and column data between each CLB and a point of origin. The entire design can then be relocated anywhere on the surface of an FPGA, while maintaining its relational and connection constraints. This ensures that issues such as timing and gate propagation are met, as they were when initially compiled. It is possible to override positioning data for CLBs if necessary. Used BRAMs are analysed and separate methods built for their construction. As before, in the case of the other resources, appropriate statements are generated to implement correct settings. Data content in the BRAM is translated into an array format, held in the method, which is downloaded at build-time. Finally, statements are written for output wiring. This connects the BRAM to its owning process. It is important to note here that it is possible to position the BRAM at any BRAM location and it will still be wired to correctly. This makes it easy for a run-time system to place and route such resources in whatever way is most suitable at the time. In all the above produced code, no routing data is generated that leads to external IOB positions, however, details are stored and checked, for uniqueness and integrity. Only end point IOB information is used as default connection data, stored in the generated program. 4.3 Design Considerations and Constraints A process is extracted from a configuration bitstream by analysing resource usage and I/O requirements. Nets which lead to I/O points are viewed as the process-to-be's main interface points to the outside world. If such points are named in an accompanying User Constraints File (UCF), in the usual format, then the name persists through from the design/programming stage to run-time.
172
R. Moseley / Reconnetics
There are few alterations necessary when supplying a design to the capture system; BRAMs will be automatically captured and available for run-time loading. Bi-directional wires are simply split into two uni-directional wires. While processes can easily be used on devices they were originally compiled for, there is also a degree of device independence. Re-use capability is determined by three main parameters: the actual device type, clock speed and clock routing. Using this system the device type (currently within the Xilinx Virtex series) can be changed quite easily. A design originally intended for a small device (such as a XCV50), can easily be transferred to a much larger device and vice versa, resources allowing. The clock speed can be maintained at its old frequency, or changed within the DCP. The clock routing for a specific bitstream can also be altered, with the supplied tools. A process owns its output routing but not any wires connected to external inputs. When the process is unloaded, its internal logic and routing is released, as well as any output links that it owns. Routing to a process's input port persists after its destruction and must be unrouted (if required) by the owning unit. 4.4 Input / Output Connection Points It is possible to pre-process a bitstream, so that any I/O is passed through a special connection point. This pre-processing stage is particularly useful for designs which already exist (for example, as a legacy design) and there is a need to turn it into a fully selfcontained process. It is also possible (though not as simple), to manually add a connection point, using the system itself at run-time. In both cases a special system object known as procio is used. This unit added in is actually very small, being only a quarter of a CLB for each wire. clock in
PROCIO
direct out
input clocked out
Figure 3. PROCIO unit showing main connections There are three functions for the I/O points: to decouple, to simplify routing to other processes, and finally to act as a controllable gateway to the world. The idea here is to allow a degree of control and buffering at I/O points. When this is done it is possible to control the connection point with DCP commands, such as inputON and inputOFF. The functionality of the procio object allows I/O signals to be disconnected, clocked, or fed straight through. 4.5 Recapture Recapture is the name given to the ability to capture a process at run-time. Many of the techniques used in the pre-run-time capture stage are reused within recapture, although obviously initial design details are not available (such as the data contained in the UCF,
R Moseiey i Recometics
173
giving port labels). Information for capture at run-time is gathered from two sources; the instantiated circuit and the encapsulating Java program, via communication with its p o r t ( ) method. These two units that make up an MHP provide current resource details, structure, and port I/O information. Together, the information gathered allows a new entity to be produced. This contains any changes made to it since its initial instantiation, captured in real time, while maintaining persistence of initial design semantics and high-level user details, such as port names. This generated program is then dynamically compiled and its separate files organised into the appropriate directories: . j ava to the design directory and .class to the pool directory. This all occurs at run-time, automatically, without the user intervening in any way. The new MHP can then be used in the same user program, if so desired. 5 Design Engine / Supervisor When DES runs a user-supplied program, it initialises a device model, which is a representation of the external physical device. The processes, initially in the form of the Java classes, are loaded into memory via a Java class loader from the archive pool. Processes can then be directed to instantiate onto the FPGA, route to I/O and interact.
Figure 4. Run-time visualisation
This is visualised in figure 4. The readback and implementation loop, which in reality takes place over a local or network TCP/IP connection, occurs only when necessary, such as an event instigated on either side. Any exchange of data is limited to packets rather than whole bitstreams, to enable fast update or information retrieval.
174
R. Moseley / Reconnetics
One of the main tasks of DBS is to run the DCP script written by the user. There are several areas that this control language deals with: •
Initialising device and internal representation model. This includes commands which configure the system to a connected device (which may be local or remote), so the model, for example, conforms to the correct type. • Location or port definitions. Locations can be assigned names which can be used for referencing within the program. Some locations, such as a process's ports, have already defined names associated with them. • Device control. This includes such commands as resetting the device, clock stepping, and forcing a 'readback'. • Process control. A number of commands are available for the placing and routing of processes. They may be automatically placed, or organised within a userspecified pattern. Various ports (or locations) can be defined on a process, then observed or linked to the I/O of other processes or external device pads. Processes can be unloaded and the resources associated with them can be freed for further use. A process's I/O can be controlled and switched on and off for isolation. A process may also be 're-captured', for storage and transport to another location. • Interconnect. This set of commands allows defined locations to be routed or unrouted. Locations can be device pads, lOBs or process I/O. • Interaction. Using defined locations it is possible to collect data from active processes, which can then be acted on. • Program sequence and control. A range of repetition and control structures exist. • Diagnostics. Although most low-level activity is shielded from the user, it is still possible to use diagnostic functions, if required, to check circuits. A good example of this is the tree command, which enables a user to view a hierarchical tracing of connections from a defined point.
6 Communication Interaction is possible between the processes themselves and between the processes and the system. Inter-process communication is, of course, wire-based. Such wire connections can be controlled within the user-written DCP. The system itself can observe and interact with the instantiated design, by the user defining locations, usually registers, in the circuit. These usually are clocked registers, which may be, for example, buffered I/O points. As well as collecting data from such points, it is also possible to set values at inputs. Processes can be placed (using loadO), interacted with and destroyed (using unload ( ) ) . If a design as a whole is to function properly during its manipulation (in particular, its destruction), then certain protocols need to be adopted. A problem could occur, for example, when some communication may be taking place between processes and the system wants to extract one of them. This situation necessitates the use of communication signals to determine the state of a process at a given time. A protocol in this way could be used to interact with the run-time system to make its safe extraction possible. This is also true during the re-capture of a process when it must first enter a 'safe' state that must be signalled, via defined points. The least interfering and non-invasive way to achieve safe extraction is to isolate the process. This involves switching off any inputs to the process and possibly generating a
R. Moseley / Reconnetics
i75
stand-by signal to the connected units. Input isolation is achieved through the inputOFF ( ) command. It would be a simple matter to provide the correct hand-shaking protocol and routing for a stand-by signal in processes, if this is required.
7 Using the System There are several ways that the system can be used. At the most simplistic level it can be used to build designs, controlling placing and routing to a specific pattern. This may include no dynamic interaction. The bitstream produced can then be used, as is normally the case, to configure devices. A further simple use of the system would be as a distributor of updates to hardware. The main use of the system is for dynamic design, which envisages a design cycle extending into run-time. Instantiated processes can be interacted with, verified and data downloaded from. The device can be amended, as required, on specific events taking place. A device may also request certain processes to be instantiated; for example, on a fault condition arising. Program
Description
Tests
Pulse Counter
Two processes (pulse generator and pulse counter) plus link
Basic functioning of system
Microcontroller 1
Larger more complex process (third party design). Program runs on hardware and interacts with DES
Handling and placement of larger processes. Interaction
Microcontroller 2
As above but utilising multiple processes placed dynamically at staggered points in time
Installation of multiple complex processes, dynamically
Flash
Process to link to various external pads
External IOB linking
Dataltnk
Setup various processes experiment with communication protocols
Inter-process communication
Generator
One process communicates with DES to request loading of new processes
FPGA initiated control
Reconfigurable Computer
On-going computer design, reconfigurable components
based
on
Large application. Tests data sharing, swapping various units, sharing of data with host system
Table 1. Test programs
Table 1 shows a suite of test programs devised to experiment with various possibilities. More information on these is available at the Reconnetics website [13].
176
R. Moseley / Reconnetics
8 Performance Table 2 shows averaged performance figures in milliseconds of various activities with the system. They are intended to give a rough idea of system capabilities rather than precise information. The system, as currently implemented, is pure Java with JBits API extensions. There are identifiable areas for optimisation and various techniques could be used to achieve speed-up. If speed is the critical factor, rather than adherence to the platform independence of Java, then certain functionality can be sped up using the Java Native Interface (JNI) to a pre-compiled language. All data was gathered on IGHz computers and a Celoxica RC1000-PP board incorporating an XCV1000 Xilinx FPGA. RC1000-PP Board (XCV1000 device) Local
Over LAN lOOMbps
Process 1
Process 2
Process 1
Process 2
CLB size
2
35
2
35
Auto place
2000
8300
2600
8930
User place
1890
8000
2200
8812
Destroy
140
420
250
600
Recapture
4000
12000
5580
13200
Recapture and deploy
7500
15000
8570
16500
Single wire link
1000
1200
1580
1780
Variable Read
800
1200
1500
1320
Table 2. Performance
9 Example 1 Example 1, below, shows a DCP which loads a microcontroller (labelled kprocessor in the archive), with a defined output port named REGOUT and a width of 8. This microcontroller is available from Xilinx in Electronic Design Interchange Format (EDIF) [14]. The hardware design could have equally been developed by the user with a structural, behavioural or algorithmic language, inside an electronic design environment. We will now step through the program. Initialisation concerns selecting the correct internal representation model, initial bitstream (or canvas) and choice of connection to the physical device over a network or locally. This is done with the device () statement. In
R. Moseley / Reconnetics
177
this case the device is a XCV300, with an appropriate blank bitstream and an IP address of 192.168.0.168. Once connected, the device is reset and the clock frequency selected. Two processes are then defined using the process () statement. This takes the form of an identifying name and the name of the archived process class in the pool. The first process is then loaded up using an automatic placing method. The Java program builds the circuit and stays resident to act as mediator between the run-time system and its physical side. A variable is then defined for use in the program; this is associated with an output register on the process and its bit size defined. Names set at design-time are still available and persist through to run-time programming. In this case the var statement consists of the variable name regO, on processor!, with a port named REGOUT and the width being 8 bits. The next step involves loading a program into the BRAM, which is connected to the processor using the readHex ( ) command. In the example, a simple counting program was fed to the processor. This increments the register, which is then printed for confirmation. Note here how it is the process's Java program which is collecting the value from the physical circuit and passing it back to the run-time system. When the loop is terminated, the second process, defined earlier, is then loaded. A wait () is used to keep the hardware processes running until the regO variable reaches the value 200. Finally, both processes are unloaded. Unloading first releases used resources on the FPGA matrix and then used resources within the host, such as the running process object. // initialise device device(xcv300, nullSOOGCLKl, 192.168.0.168); reset(); frequency(10); // set a clock speed of 10MHz //get the processes process(processorl, kprocessor); process(processor2, kprocessor); //load a process load(processorl); //define test location var(regO, processorl, REGOUT, 8); // load program into BRAM readHex(0,0,16,"test.hex"); do {
message("regO:"); print(regO); } while (regO < 10) ; // load the next process Ioad(processor2); wait (regO,200); unload(processorl) ; unload(processor2); Example 1. Simple DCP
R. Moseley / Reconnetics
178
10. Example! This example shows the two different kinds of communication, between processes and between process and the run-time system. DESIGN ENGINE / SUPERVISOR (Run-Time system)
Counter (Java object)
currentCount
Pulse Generator (Java object)
done
Counter (hardware process)
freqSetup
finishAt pulsein stop
pulseOut
Pulse Generator (hardware process)
startStop
Figure 4. Communicating processes
Two processes are used; a 16-bit counter and a pulse generator. Figure 4 clearly shows how the Java program acts as a mediator between the run-time system and its hardware circuit, once instantiated. The counter counts pulses on its pulsein port. The current value appears on its currentCount port. When the value (programmed via finishAt) has been reached, a stop signal is generated. The pulse generator process can be programmed with a frequency rate and stopped through its startStop port. Using a DCP, the system is initialised and processes loaded with manual place this time: device(xcvlOOO, null100OGCLK3, localhost); reset (); frequency(3.5); // set a clock speed of 3.5 MHz process(counter, counterl6),process(generator, pulsegen); loadAt(counter, 10,10); loadAt(generator, 20,10); //define test variables var(currentCount, counter, COUNTOUT, 16); var(finishAt, counter, COUNTIN, 16); var(stop, counter, STOPOUT, 1); var(pulseln, counter, PULSEIN, 1); var(done, counter, FINISHED, 1) ,var(freqSetup, generator, FSETUP 8); var (pulseOut, generator, PULSEOUT, 1) ,var(startStop, generator, ONOFF, 1);
R. Moselev / Reconnetics
Links are established: link(stop, startstop); link(pulseOut, pulseln)• The frequency and counter are programmed using the inputval ( ) command: inputVal(freqSetup, 10) ; inputVal(finishAt, 1000); The process counts the pulses, taking done high when it has received the amount of specified pulses. During the counting time, the run-time system simply waits: wait(done, 1); To show the transport of a process from one location to another we can capture it: recapture(counter, new_counter!6); then release resources: unlink (stop) ,unlink(pulseOut); unload(counter); Finally, re-deploying somewhere else: process(counter2, new_counter!6); load(counter2); // using auto-place Variables can be then defined again and links re-established. The counter can be told the required amount of pulses and another wait () statement executed. When finished, the program can end with resources being released, through unlink ( ) and unload ( ) .
11. Further Development Besides on-going improvements to the system, there are two main enhancements that research has brought to light. They are worth mentioning, although it is beyond the scope of this paper to detail them fully here. The first concerns the ability of such run-time systems to be embedded into devices themselves, allowing self-reconfiguration. The second idea follows from this, in allowing the processes themselves some degree of autonomy, providing access to configuration features. These have been named intelligent Mobile Hardware Processes (iMHP). Such work as described here implies a merging of the software and hardware domains, creating entities that are composites of both worlds and realising the benefits of each.
12. Acknowledgements Thanks to Xilinx for developing the Virtex hardware and most all to the JBits team who have been very helpful in my endeavours. Thanks also to my supervisor, David Wood, and the University of Kent Computing Laboratory for supporting this project.
180
R. Moseley / Reconnetics
References [I] Xilinx. Virtex 2.5V Field Programmable Gate Arrays Data Sheet, Xilinx Inc. 2001. [2] N. McKay and S. Singh, Dynamic Specialisation ofXC6200 FPGAs by Partial Evaluation, Field Programmable Logic and Applications. Tallinn, Estonia. Springer-Verlag, 1998. [3] M. Wirthlin and B. Mulchings. Sequencing Run-Time Reconfigured Hardware with Software, Dept. of Electrical and Computer Engineering, Brigham Young University, Utah, 1996. [4] P. Lysaght and J. Dunlop. Dynamic Reconfiguration of FPGAs, Dept. of Electronic and Electrical Engineering, University of Strathclyde, 1995. [5] K. Compton and S. Hauck. Configuration Caching Techniques for FPGA, University of Washington, Seattle, 2000. [6] N. McKay and S. Singh. Debugging Techniques for Dynamically Reconfigurable Hardware, Proceedings: IEEE Symposium on FPGAs for Custom Computing Machines: April 21-23,1999, Napa Valley, California, edited by K.L. Pocek and J.M. Arnold (IEEE Computing Society, 1999). [7] K. Compton and S. Hauck. Reconfigurable Computing: A Survey of Systems and Software, Dept. Of Electrical Engineering, University of Washington, Seattle, 2000. [8] Xilinx. Technical Overview, Xilinx Application Notes XAPP097, Xilinx Inc. 2000. [9] Xilinx. Dynamic Reconfiguration, Xilinx Application Notes, XAPP093, Xilinx Inc. 1998. [10] Xilinx. Xilinx Home Page. Available at: http://www.xilinx.com [II] Xilinx. Using the Virtex Block SelectRAM+ Features, Xilinx Application Notes XAPP130, Xilinx Inc. 2000. [12] Xilinx. Xilinx FPGAs: A Technical Overview for the First-Time User, Xilinx Application Notes, XAPP097, Xilinx Inc. 1998. [13] R. Moseley. Reconnetics Home Page, 2000. Available at: http://www.ralph-moseley.co.uk/reconnetics [14] Xilinx. 8-Bit Microcontroller for Virtex Devices, Xilinx Application Notes XAPP213, Xilinx Inc. 2000.
Communicating Process Architectures - 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
181
Mark GREEN and Ali E. ABDALLAH Centre for Applied Formal Methods, South Bank Technopark, 103 Borough Road, London, SE1 OAA, UK. Abstract. Improving performance is the main driving force behind the use of parallel systems. Models for performance evaluation and techniques for performance optimisation are crucial for effectively exploiting the computational power of parallel systems. This paper focuses on methods for evaluating the performance of parallel applications built from components using the software architecture methodology. Minor differences in the low-level behaviour of functionally equivalent processing elements can have a dramatic effect upon the performance of the overall system; this confounds attempts to predict performance at any step prior to implementation. VisualNets is a tool supporting the construction and graphical manipulation of interacting systems built from components. It makes use of specifications in the formal method CSP, which enables the relevant component behaviours and the linkage between components to be concisely described. The tool allows the behaviour of the whole system over time, and the patterns of interaction between the components, to be visualised through graphical animation. The graphical display produced facilitates the analysis and evaluation of performance, and highlights areas where performance could be improved via better utilisation of parallel resources. VisualNets also allows the timing properties of the components and of the architecture that underlies them to be changed, to represent different component implementations or platform configurations. A case study, based on the dual pipeline architecture, is presented to show how the graphical animation capability of VisualNets can be used, firstly to evaluate performance, and secondly to guide the development of more efficient versions of the parallel system.
1
Introduction
The primary objective of parallelising many applications is improved performance; the primary tradeoff is the cost of additional computing resources required to support parallel execution. Thus an important goal when designing a parallel system is to ensure that the resources of the parallel platform on which the system executes are utilised efficiently, so that the performance increase obtained makes the tradeoff worthwhile. Managing the patterns of interaction between parallel components such that improved performance is achieved is one of the most fundamental problems in the field. It involves issues from high-level software architecture principles right down to low-level implementation details and hardware characteristics. The emerging software architecture paradigm [1] emphasises the construction of parallel applications from reusable components. A number of frequently used parallel patterns, such as the pipeline and process farm, can be considered architectures [2] which define specific structures for combining interacting components. Parallelism can be introduced to a system by selecting a suitable architecture, dividing the overall task of the system into subtasks, and assigning these subtasks to processing components within the architecture. However, no matter how much care is taken in the selection of the architecture and the partitioning of the task, no guarantee of high performance (the best possible utilisation of the available computing resources) can be obtained. There are a large number of lower level
182
M. Green and A. E. Abdullah / Performance Analysis and Timing of Communicating Systems
issues which will impinge significantly on performance; even minor aspects in the implementation of the individual processing components may have a substantial effect on the overall performance of the system. Thus, having performed the initial design of the parallel system, it is still necessary to tune the behaviour of the components involved in order to ensure that high performance is achieved. In this paper the Communicating Sequential Processes (CSP) notation [3] is used to concisely capture the behaviours and interfaces of components within a parallel architecture. The VisualNets tool [4, 5] then enables animation and performance profiling to be carried out directly on the abstract CSP description, without the need for lower level implementation. This paper demonstrates the effectiveness of this method in tuning the components of a parallel architecture to maximise performance. Although a large number of tools allow visual performance profiling of parallel systems [6, 7, 8, 9, 10], they mainly operate on implementations as opposed to design descriptions. This inhibits their use until a late stage in the development cycle. Although these tools facilitate the detection of poor performance, they generally do not provide facilities for tracing the behaviour that causes this poor performance. This is inevitably a difficult task at the implementation level. By basing its profiling on a design description rather than an implementation, VisualNets can precisely trace the behaviour of all components in the system and thus make it easier for the developer to trace the source of inefficiency. Furthermore amending the description to rectify the inefficiency will be a simpler process for the developer than amending an implemented system in the same way. 2 Background 2.1
Communicating Sequential Processes
The formal method of Communicating Sequential Processes (CSP), first specified by Hoare [3], describes concurrent processes in terms of sequences of synchronisation events and channel communications. CSP is an extremely effective notation for representing the architecture of parallel systems, since the architectural style imposed by CSP is extremely flexible. The only stipulation is that the connecting elements must either be message-passing channels or multiprocess barrier synchronisations. CSP has no notation for formally specifying the properties of data elements, meaning that it is more suited for processing-based analysis; behaviour tuning falls into this category. Processing elements are defined by combinations of primitive operators that represent sequence, choice, communication, parallelism, and similar. This use of primitive concurrency operators^simplifies the process of making the minor behaviour changes (such as varying the order in which communications are carried out) which are required in performance tuning. In implemented code, making these minor changes may be more difficult as several statements may be involved in preparing and performing the communication, some of which may have side effects which will cause problems if their sequence is changed. The fact that standard CSP does not consider the implementation of operators can be valuable in performing design-level behaviour tuning, as some designed-in properties of behaviour may reduce performance regardless of platform. However, if the hardware on which a system will run is specified in advance, it is necessary to allow for properties related to it to completely optimise the system. An extension to CSP defined by Schneider [11] allows timing properties of a system to be represented. These can be used to model the performance properties of the hardware platform or parallelism library on which the system will run. Thus once untimed CSP has been used to model the design of a system, timed CSP can be use to
M. Green and A. E. Abdallah I Performance Analysis and Tuning of Communicating Systems
183
predict its behaviour on a particular platform without incurring the costs of implementation. Once the specification has been completed, producing an implementation based on it is a relatively simple process. 2.2
VisualNets
The VisualNets tool, developed at Reading and South Bank [4,5], allows systems specified in both untimed and timed CSP to be visually manipulated and animated. For performance evaluation, the most important function it provides is the production of a timing diagram which visualises the progress of the network over time and can also provide indications of where performance loss is taking place. This paper focuses on the use of the timing diagram to diagnose performance problems. VisualNets can also produce a structure visualisation which corresponds to a "processing view" of the system as defined in [1], and provides a visual display of component status which is updated as animation proceeds; where a system has a large number of components, modular abstraction can be employed to simplify the diagram by combining several components into a subsystem which appears as a single element on the diagram. The tool will detect any deadlocks that arise during animation runs. Additionally, a feature under development in VisualNets allows untimed CSP specifications to be transformed into timed CSP specifications based on a seperate model of a hardware platform. This enables a parallel algorithm written in untimed CSP to be profiled via simulation on a number of different hardware platforms without any need to develop multiple implementations or to modify the specification to reflect the timing properties of the platforms. At present, this feature is implemented although the method used for modelling the hardware is quite primitive. VisualNets's implementation consists of two parts: a front end module written in Java [12, 13], which produces the graphical visualisations and tables and provides the user interface; and a back end module in the functional language Haskell [14], which actually performs the animation of the user's specification and supplies the results to the front end through an interface developed as part of the research [15]. The tool is platform independant, and can operate on both Windows and Unix environments. The prototype tool is not distributed at this time as it depends on a commercial graph layout system which cannot be realistically licensed for academic distribution; it is hoped that this dependancy will shortly be removed. 3 Case Study To illustrate the need for behaviour tuning, and the use of CSP and the VisualNets tool in performing it, we consider a system implemented with a standard parallel architecture: a dual pipeline. The system in question is intended to perform long multiplication in any base; each process in the system deals with a single digit in the value to be multiplied. The multiplier is "hard wired" into the system; each process is assigned one digit of the multiplier. Less significant digits appear earlier in the pipeline flow. The multiplicand is input to the pipeline one digit at a time, with the least significant digit first, and with a stream of zeroes following it (effectively leading zeroes). Since the structure is a dual pipeline, the result is also returned from the process at the start of the pipeline, in the same order and also followed by a stream of zeroes which continues as long as the input does. The data elements involved are single digits of the multiplier, multiplicand, result, and intermediate values; the connection elements are message-passing channels. The processing elements of the pipeline are nodes. With the exception of the end nodes, each node X is connected to the node N holding the multiplier digit one place more significant than the one held by X, and to the node P which holds the digit one place less significant than
184
M. Green and A. E. Abdullah / Performance Analysis and Tuning of Communicating Systems
the one held by N. X will have an input from P and an output to N; this forms the input pipeline, the direction of which will be termed "down". X will also have an input from N and an output to P; this forms the result pipeline, the direction of which will be termed "up". The first node of the pipeline substitutes for P the input loading and output collector processes; the final node has no N and therefore has only the two channels which refer to P. The functional specification of each node is quite simple: it must input (from the input pipeline) a digit of the multiplier, and then (at some point before the next digit is input) return a digit of the result on the result pipeline. For all but the first digit produced, each node will need to allow for the carry values from its own prior multiplications, and the result of the multiplication of the remaining digits. The carry can be tracked within the node. To obtain the result for the remaining digits, each node must pass the digits it recieves down the input pipeline to the next node, which will perform the multiplication; at some point before sending another digit to the input pipeline, the node must obtain the result from the next node via the result pipeline for use in its own calculations. The exception to this rule is the last node in the pipeline, which holds the most significant digit. Since there are no more digits to multiply, it needs only to keep track of the internal carry. 3.1 Refinement to CSP The first step is to refine the functional specification into CSP notation. The notation actually used is CSP/?, the machine readable form of CSP accepted by the VisualNets tool. In the CSPf description below, die variable k holds the number of the node in the network, a is the digit of die multiplier assigned to the cell, and z is used to track the internal carry and should be initialised to 0 when the process is instantiated, msize is the number of digits in the multiplier and thus the number of nodes in the pipeline, x! a indicates that message a is output on channel x, and x?a indicates that a message is read from channel x and placed in variable a. -> indicates prefixing (sequence), and A < I x I > B is a guarded choice structure which behaves as A if x is true, and B otherwise. NODE a k z = ( inl?(x :: Int) -> outl!(((a*x)+z) 'mod' base) -> NODE a k (((a * x)+z) 'div' base)) <| k == (msize - 1) |> ( inl?(x :: Int) -> out2!x -> in2?(y :: Int) -> outl!((((a*x)+z)'mod' base)) -> NODE a k (y+(((a*x) + z) 'div' base)))
This refines the functional description given above. Each node that is not the end of the pipeline (ie, where k, die node's index, is not equal to msize — 1, the highest index) first receives a digit of die multiplicand (on channel ml), and passes it down the pipeline (on channel out2). It collects the result from the multiplication of the remaining digits (input from channel in2 and stored in y), then performs its own multiplication and passes its own result up, updating its own stored carry value based on die result obtained from the rest of die pipeline. The last node in the pipeline only retrieves results and performs its own multiplication; there is no further pipeline to pass values down to. Note that the structure of the specification is very strongly based on die communicating behaviour of die process. All of die computation performed by the process is encapsulated by the expressions for output values and guards. (For processes that perform more complex computation, CSP includes an assignment operator that can be used to represent purely computational processing steps.)
M . Green and A. E. Abddlah /Performance Analysis and Tuning of Communicating Systems
I85
Direction of input pipeline ("down")
-
Input digits
Multiplier Digit Internal Carry Result digits
*
b
Input digits propagated
out2
Results of rest of multiplication in2
Direction of result pipeline ("up")
Figure 1 : A single node of the multiplication pipeline.
Figure 2: Structure visualisation of the Dual Pipeline Multiply.
VisuaZNets can be used to construct a pipeline of these processes, and to add additional processes representing the input and output ports. The structure visualisation of the network produced is shown in figure 2.
186
M.Green and A. E. Abdallah / PerformaoceAnalysis and l h h g of CommunicatingSystems
Each box on the visualisation represents a process - in this case a cell of the pipeline. The lines represent channels connecting the processes; the indicators at the end of each line are colour coded to represent the status of each process with regard to that channel. A red indicator means that the process is not offering communication; a green indicator means that it is doing so. Where a communication is offered by one of the processes (even if no cooperation is being received), an arrowhead is added to indicate the direction of the communication. The visualisation clearly shows the dual pipeline structure, with each node having one channel to the prior node and one to the subsequent node. The multiplier in this case is 3528 1, visible distributed through the parameters of the nodes. When the network is animated, the behaviour of the network is as shown in the timing diagram in figure 3. The multiplicand, 49152, is input to the network and the solution to the multiplication, 1734131712,is shown as the output on channel d.0.
Figure 3: T i g s for the first version of the Dual Pipeline Multiply.
The left hand side of the timing diagram shows the channels in the network; the top scale indicates the passage of time. A value in bold type indicates a successfully communicated value. Since no timings were specified for this system, KsualNets has applied the default timing model, in which computation takes zero (or negligable) time and any communication takes one time step.
M. Green and A. E. Abdallah I Performance Analysis and Tuning of Communicating Systems
187
The timing diagram also shows blocked communications. A blocked communication occurs whenever a process is attempting to communicate but is unable to do so as the other process involved is not cooperating. A value in faded type indicates that an attempt to output that value was blocked as the other process was not inputting; an italicised letter I indicates that an attempt to recieve input on the appropriate channel was blocked as the other process was not outputting anything. This blockage information is useful because it shows areas where performance loss is potentially being caused by difficulties in the concurrent design, as opposed to delays mandated by the timing model (ie, time taken by hardware) or caused by computation. This information can be removed from the timing diagram if the user wishes. The total time passing between successful communications on a channel can be read as the number of spaces between boldfaced values; the delay imposed on a given communication by blockage within the system can be read as the number of faded values or italicised i's preceding the bold value representing the communication in question. The timing diagram shows that the performance of this system is extremely bad; the last digit of the result is not sent until time step 100. The timing diagram further shows that the parallel properties of the architecture are being completely suppressed; only one process is ever engaged in communication at once. This has occured in spite of the fact that all of the processes are behaving in accordance with the functional specification, and in spite of the absence of any significant delays in the default timing model. In fact, it is likely that any system whose implemention behaved in the manner described by the CSP refinement would exhibit this performance problem regardless of the hardware on which it executed. However, the failures information in the timing diagram informs the user that this performance problem may be rectifiable. The long string of failed inputs before each process indicates that time is being wasted. If these did not exist, then the time would be being spent on other purposes (such as computation) and it may genuinely not be possible to speed the network's operation up. 3.2
Performance tuning
By examining the timing diagram it can be seen that the delays tend to occur in between a node sending a value to be multiplied up the input pipeline and retrieving the result. Other tasks could be accomplished during this time. Since the multiplication result returned from the remainder of the pipeline for the current digit affects only the future digits of the result (the current result digit is not affected), it would be entirely possible for the node to send its result digit up the pipeline before recieving the result from the rest of the pipeline, provided it still did receieve the result from the rest of the pipeline before returning the result for any further digits. This can be represented in the specification as follows: NODE a k z = (inl?(x :: Int) -> outl!(((a *x)+z) 'mod' base) -> NODE a k (((a * x)+z) 'div' base)) <| k == (msize - 1) |> ( inl?(x :: Int) -> out2!x -> outl!(((a*x)+z) 'mod' base) -> in2?(y :: Int) -> NODE a k (y+(((a * x) + z) 'div' base)))
The specification is the same as the previous one, except that the last two steps of behaviour have been reversed. Since no change has been made to the architecture, the structure visualisation of the system remains the same, but animating the system to produce a timing diagram shows radically altered performance. The timing diagram produced when animating this version of NODE is shown in Figure 4.
188
M. Green and A. E. A W a h /PerformanceAnalysis and fining of CommunicatingSystems
Figure 4: Timings for the second version of the multiplication pipeline.
This version of the system shows dramatically improved performance;the last digit of the result is output on time step 39. Although a large number of failures are still shown on c.0 and d.0, this is only due to the fact that these channels are connected to the pipeline loader and the result collector, both of which continuously attempt output and input respectively. Since each node sends its result for each digit back before recieving the correspondingresult from the rest of the pipeline, the result digits can be propagating up the earlier nodes of the pipeline at the same time as the input digits propagate down the later nodes. The only failures remaining are occasional failures on c.4 and d.4. These channels are connected to the pipeline end process, which has a different timing cycle to the others due to the fact that it does not have to pass its values down the pipeline. The shorter cycle means that it completes and returns its result slightly faster than any other node; this means it tries to return its result immediately after the input is sent, while the previous node is still sending its own result up the pipeline (this causes the output failures on d.4). Once its output is sent it does not have to wait for a result from a further node, meaning that it becomes ready for input slightly earlier than the previous node (this causes the input failures on c.4). 3.3 Further performance tuning The previous specificationworked quite well, but there is still a possible refinement that could be explored. In the previous specification, each node sent its own result up before receiving the result from the rest of the pipeline. Could performance be improved if each node sent its own result up before even propagating the input to the rest of the pipeline? Testing this requires only a simple change to the specification: alpha NODE = [inl, outl, out2, in21 NODE a k z = (inl?(x :: I n t ) -> outl! (((a*x)+z) 'mod'base) -> NODE a k (((a * x)+z) 'div' base)) < I k == (msize - 1) I> (inl?(x :: Int) -> outl!((((a*x)+z)'mod'base)) -> out2!x -> in2?(y :: Int) -> NODE a k (y+(((a * x) + z ) 'div' base)))
In this case, the second and third events in nonend nodes have been reversed, so the result is now sent immediately after the input is recieved, before the input is propagated down the pipeline. As before, this does not change the architecture of the system, but the performance is affected as shown by the timing diagram:
M , Green and A. E. Abdallab /Performance Analysis and Tuning of CommunicatingSystems
189
Figure 5: Timings for the third version of the Dual Pipeline Multiply.
This version of the specification shows a very small performance improvement - the last digit of result is sent one time step earlier, on step 38 rather than 39. The first digit of result is also sent a time step earlier. Both of these are entirely due to the one time step saved by returning the result before propagating the input. The failures caused by the loading and collecting processes remain (and cannot be avoided), and the failures caused by the different cycle of the end process also remain, except that they now appear as two input failures rather than one input and one output failure. This is because each node is now ready to recieve the result from the rest of the pipeline immediately after sending the input to it (which eliminates the output failure), but it has two communications to do instead of one between reading the result from the rest of the pipeline and propagating the next input digit (which causes the second input failure). The CSP above describes the most efficient version of the dual pipeline multiply [3]. Most other tunings based on changing the order of the communicationsare either nonsensical (involving sending a calculated value before receiving a value on which the calculation is based) or cause an immediate deadlock. Although the system discussed here is very simple, the fact that such major performance variations can exist within such a simple system demonstrates the value of performance tuning. In order to tune a more complex system it may be necessary to use modular abstraction to represent the system as a number of communicating subsystems. The CSP notation and the VisuaZNets tool facilitate this. The network of subsystems can be visualised, and the communications between them analyzed to identify if any particular subsystem is causing a performance problem; a problematic subsystem can then be scrutinised in detail within the overall context to establish if the problem is caused by poor performance within the subsystem or poor integration of the subsystem with the rest of the system. 4
Conclusion
The benefits of the use of simulation in performance tuning are clear, but the level of specification granularity necessary to produce an accurate simulation of a parallel system can inhibit its use. The CSP notation provides an adequate balance, as it specifies enough information about behaviour to enable reasonable simulation of the network, while remaining close to an architectural specification of the functions of processing nodes. WsualNets’s static visualisation provides a strong tie back to the architectural design of the system, and its behaviour visualisations provide a clear indication of present performance and of scope for performance improvement.
190
M. Green and A. E. Abdallah / Performance Analysis and Tuning of Communicating Systems
When further developed, the platform modelling features of VisualNets will enable performance optimisation to be performed with regard to specific platforms, or for multiple platforms without any need for reimplementation. It is also our intent to add support for additional visualisations (which will provide multiple views of the architecture of the system), and support for other input notations (such as parallel path expressions [16] as used in [1]). References [ 1 ] D. E. Perry and A. L. Wolf. Foundations for the study of software architecture. ACM Software Engineering Notes, 17(4):40-53,1992. [2] J. L. Ortega Arjona and G. Roberts. Architectural patterns for parallel programming. In Proceedings of the 3rd European Conference on Pattern Languages of Programming and Computing, 1998. [3] C. A. R. Hoare. Communicating Sequential Processes. Prentice-Hall International, 1985. [4] M. Green. Visualisation and Animation of Concurrent Systems specified in CSP. PhD thesis. University of Reading, 2002. [5] A. Abdallah and M. Green. A CSP-based integrated tool for the visualisation, animation, and performance evaluation of message passing algorithms. In Proceedings of the IEEE Conference on Formal Engineering Methods 2000,2000. [6] Topol, Brad, Stasko, Vaidy, John, and Sunderam. Pvanim: A tool for visualization in network computing environments. Concurrency: Practice and Experience, 10(14):! 197-1222,1998. [7] A. Benguelin, J. J. Dongarra, G. A. Geist, R. Manchek, and V. S. Sunderam. Graphical development tools for network-based concurrent supercomputing. In Proceedings of Supercomputing '91, pages 435-444, 1991. Paper freely downloadable at http: //citeseer.nj .nee. com/beguelin91graphical.html. The HeNCE tool is downloadable from http: //elib. zlib.de/netlib/hence. [8] T. Bemmerl and P. Braun. Visualisation of message passing parallel programs with the topsys parallel programming environment. In Journal of Parallel and Distributed Computing 18, 2, pages 118-128, 1993. ISSN 0743-7315. [9] P-grade. http://www.lpds.sztaki.hu/projects/p-grade/. [10] S. Utter-Honig and C. M. Pancake. Graphical animation of parallel fortran programs. In Proceedings of Supercomputing '91, pages 491-499,1991. [11] S. Schneider. Concurrent and Real-time systems: The CSP approach. John Wiley and Sons, Ltd, 2000. ISBN 0-471-62373-3. [12] T. Lindholm and F. Yellin. The Java Virtual Machine Specification. Sun Microsystems, 1999. [13] J. Gosling, B. Joy, and G. Steele. The Java Language Specification. Sun Microsystems, 1999. [14] S. Peyton Jones and J. Hughes. Report on the programming language Haskell, a Non-Strict, Purely Functional Language, http: //www. haskell. org/def init ion/. [15] M. Green and A. Abdallah. Interfacing Haskell with Java. In G. Michaelson and P. Trinder, editors, Trends in Functional Programming. Intellect, November 2000. ISBN 1-84-150024-0. [16] R. H. Campbell and A. N. Habermann. The specification of process synchronisation by path expressions. In Lecture Notes in Computer Science Number 16, pages 89-102,1974.
Communicating Process Architectures 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
191
Keith PUGH Computer Science Department, Keele University, Keele, ST5 5BG, UK. Abstract. A home network comprising many heterogeneous devices requires a scaleable interconnect capable of satisfying the often vastly different network resource demands of the devices. In addition, it is likely that over time the configuration of the network will change as new devices are added to the network and older ones removed or replaced. While it is acceptable for enterprise networks to be managed by a trained network administrator it is unreasonable to expect a home owner to have such expertise. Consequently, a free topology network coupled with a capacity for plug and play that allows nodes to be added anywhere, and at any time without interruption to the operation of the networked system are essential requirements. IEEE 1355 standard technology has the potential to satisfy these criteria. The demand for a free topology and a capacity for plug and play require that the configuration of the network is re-discovered and mapped automatically, and at regular intervals. This paper describes such a configuration mapping process.
1
Introduction
The modern home has separate networks for heating, telephone, security entertainment, lighting, etc. Advances in digital technologies for consumer electronic devices and high performance networks have led to the possibility of uniting these dissimilar networks into a single ubiquitous home network that allows the interconnection of numerous heterogeneous devices. This diversity of devices imposes many different demands on the system's network design. For example, devices that perform status gathering or simple on/off control operations consume relatively little of the communication network's bandwidth - they can share bandwidth with other processes. On the other hand, real-time audio or video data transmissions need to be delivered at a high and sustained bit rate, which may require exclusive use of the network's bandwidth. A communication network performs best when the latency of the interconnect is low and the throughput is high. Achieving low latency and high throughput requires fast hardware, and efficient communication protocols that minimise communication software overhead. Scalability and cost are important considerations for the design of an effective network. The network must continue to perform well as it grows in size, and the cost of the interconnect should not be allowed to dominate the cost of the system [1]. Many of the existing methods of interconnecting the components in a digital network within a limited area such as a room or a building are based on the restrictive model of a bus and do not meet these criteria. In a bussed system the bandwidth must be shared among the nodes of the network, consequently, performance declines as more nodes are added. More recent high performance network designs utilise routing switches to solve this problem by permitting simultaneous data transfers between disjoint pairs of nodes. A message
192
K. Pugh / Configuration Discovery and Mapping of a Home Network
generated at a source node travels through one or more routing switches to reach its destination node. To achieve the routing function knowledge of the network topology is needed, which, depending on the routing strategy, may be contained in routing tables in the source nodes or in the routing switches. In these networks the overall bandwidth grows as more nodes are added, thus greatly increasing the system's potential throughput. One method of interconnection that utilises routing switches is specified under the IEEE 13551995 standard [2]. This standard allows very low cost implementations, and is used in a wide range of applications. The space industry, for example, uses IEEE 1355 technology for its reliability and heterogeneity, and large parallel supercomputers exploit its performance characteristics. Over time the configuration of the network will almost certainly change as new devices are added to the network and older ones relocated or removed. While it is acceptable for enterprise networks to be managed by a trained network administrator it is unreasonable to expect a home-owner to have such expertise. Consequently, in a network that allows nodes to be added anywhere, and at any time without interruption to the operation of the networked system the ability to plug and play is an essential requirement. Cook argues that current home network architectures, which rely on a set of communications protocols defined to accommodate all present and future devices, are inefficient and restrict extensibility [3]. Any protocol that encompasses all devices is, by definition, one that must accommodate the most complex device envisioned. The definition of such a protocol will inevitably be overtaken by events. Furthermore, it is completely inappropriate to incorporate the support for an all-encompassing protocol into a simple device such as a lamp, where the processing capabilities are likely to be extremely limited. This approach suffers on the one hand from being short sighted and on the other hand being prohibitively expensive for the simpler case. Cook proposes instead an architecture in which each device supplies its own proxy driver to support the device's functionality, and also provides the, possibly unique, communication protocol to access it. The proxy device drivers are uploaded and run separately within a central controller. Using this scheme it is anticipated the central controller and the devices need only support a basic common communication protocol in order to provide a home network that is future proof. It is this concept that our work explores. But to begin, the combination of a free topology network and a capacity for plug and play requires that the configuration of the network is rediscovered and mapped automatically, and at regular intervals. This paper describes such a configuration discovery and mapping process.
2 IEEE 1355 Technology The IEEE 1355 standard enables the construction of scaleable low-latency serial interconnect systems based on high-speed point-to-point links and routing switches. The standard specifies the physical media and a set of lightweight protocols (up to the packet layer), for a family of flow controlled, point-to-point, communication technologies with speeds and media ranging from 10 Mbps to 1 Gbps in both copper and optic fibres. IEEE 1355 links can be used as a reliable transport mechanism by higher level protocols, e.g. ATM. The work presented in this paper focuses on the IEEE 1355 DS-link technology, operating at speeds from 10 Mbps to 200 Mbps. This technology was designed initially as a multiprocessor interconnect, but it is equally suitable for local area networks, and has demonstrated scaleable performance from one to over a thousand nodes [4], [5].
K. Pugh / Configuration Discovery and Mapping of a Home Network
header
payload
eop
Figure 1: IEEE 1355 packet format The packet format is shown in Figure 1. Information is transferred in packets comprised of a header, containing routing information which is used to route a packet through the switching fabric, followed by a payload of zero or more bytes of data e.g. 53 byte ATM packet, and finally an end-of-packet termination character. There is no prescribed packet length, however these should be limited in size to prevent them monopolising the network resources and blocking other packets. 2.1 Routing A network's routing strategy determines how a message traverses its path through the network. In IEEE 1355 networks messages may be broken into a sequence of smaller more manageable packets, and individually routed from the source to their destination through routing switches. The packet header contains routing information that is interpreted by the routing switches when directing the packet to its destination. 2.1.1 Switch Architecture A routing switch consists of a number of input and output ports, buffers to queue packets, an internal interconnect which connects the inputs to the outputs and an arbiter to resolve contention between the ports. The number of output ports is equal to the number of input ports. The internal interconnect in an IEEE 1355 routing switch is generally a non-blocking crossbar [6]. In a non-blocking routing switch any permutation of input and output ports can be connected. However, not more than one input port can be connected to the same output port simultaneously. Consequently, contention occurs if more than one packet arrives on different inputs at the same time all destined for the same output port. The arbiter in the routing switch allows one of the packets to proceed while the other packets are blocked and must be queued until the output port is free. Ultimately, on average, the input port selection algorithm provides each input port with an equal opportunity to send packets. However, if the packets are not all the same length, this selection scheme does not guarantee fair sharing of the output link bandwidth. IEEE 1355 architecture routing switches improve the efficiency of the packet switching strategy by enabling the use of wormhole routing [7], and header deletion techniques, which provides very low switching latency and also requires minimal buffering in the routing switches. 2.1.2 Wormhole Routing Usually, packet switching networks route packets using a store and forward technique. This requires each intermediate node to store an incoming packet and to decode the packet's header in order to forward the packet to the next node. This is undesirable for high speed, low cost interconnect systems because: • Storage is necessary at each node to store packets. This limits the packet size and increases costs.
194
K. Pugh / Configuration Discovery and Mapping of a Home Network
• Delay (latency) is increased because the complete packet must be received before it can be forwarded to the next node. The wormhole switching technique ameliorates these problems. In wormhole routing networks only the first part of the packet is required to make a routing decision. The routing decision is taken as soon as the packet header has been read by the routing switch. The header is then sent to the chosen output link and the rest of the packet is copied directly from the input of the routing switch to its output. This means that a packet can be passing through several routing switches at the same time, and the head of the packet may be received by the destination before the whole packet has been transmitted by the source. This method can be thought of as a form of dynamic circuit switching, in which the header of the packet, in passing through the network, creates a temporary circuit (the 'wormhole') through which the data flows. As the tail of the packet is pulled through, the circuit vanishes. The ability to start outpurting a packet while it is still being input can reduce delay significantly, especially in lightly loaded networks. The delay can be further minimised by keeping the headers short and by using fast, simple hardware switching to determine the link to be used for output. 2.1.3 Header Deletion An approach that simplifies the routing of packets in a network is to provide two levels of header on each packet. The first header specifies the output link from the routing switch, and is removed as the packet leaves the routing system. This exposes the second header which tells the destination device which process this packet is for. Any output link can be set to do header deletion, i.e. to remove the routing header from the front of each packet after it has been used to make the routing decision. The first part of the remaining data is then treated as a header by the next node to receive the packet. hi summary, IEEE 1355 technology has many advantages over other interconnect methods [6], [8]: • IEEE 1355 provides a set of lightweight protocols for bi-directional flow controlled, point to point communication. • Networks of any size and topology can be built using routing switches to provide complete connection between all devices. • The communications bandwidth does not saturate as more communicating devices are added to the system. Rather, the larger the number of devices, the greater the total communications bandwidth of the system. • A credit based flow control mechanism built into the fabric of each communication link means packets are never lost (and have to be retransmitted), and there is no possibility of data overrun at the receivers. This simplifies the higher levels of the protocol, since it removes the need for retransmission unless an error occurs. • A small protocol overhead makes the links very efficient, even for short packets. • A flexible packet format allows IEEE 1355 links to be used as a carrier for other higher level protocols. These features provide a solution to the bottleneck problem normally associated with a high-speed communication system.
K. Pugh / Configurution Discovery und Mapping qf a Home Network
3 TheNetwork The network depicted in the undirected graph shown in Figure 2 illustrates the components of a home network. The nodes represent one controller, four routing switches, and three devices. The edges between nodes are their interconnecting communication links. Device are leaf nodes of the network. In a typical network there will be an assortment of devices of varying complexity. For example, simple devices such as lamps with minimal storage and processing ability may coexist with sophisticated devices such as video recorders and satellite receivers. The Controller is a device which in addition to its normal function has overall responsibility for the organisation of the configuration discovery process, and for maintaining a contemporary map of the network configuration. The controller may be any device with the necessary storage and processing capabilities, for example a digital set top box or a personal computer. Routing switches are multi-port switches that channel messages from their source node to their destination node. Links are point to point connections between each pair of nodes. All nodes implement a full IEEE 1355 interface on each of their external ports. In addition, each node must provide its own hard coded unique identifier, and an internal management port (port #0) to enable access to the unique identifier and other relevant configuration information.
Figure 2: a typical network 4 Configuration Discovery Algorithm The need for a free topology and a capacity for plug and play require that the configuration of the network is re-discovered and mapped automatically, and at regular intervals. In a typical switched network [9] a processor in each switch monitors the state of the network and triggers a distributed configuration discovery algorithm whenever a change in the network topology is detected. However, this approach is considered inappropriate for a home network because of the limited processing capability of many of the nodes in the network and the desirability of low cost. Instead, the discovery process is centralised at the controller from where the other nodes in the network are polled for their configuration details. The process is summarised by the following algorithm: for each node on the network for each port at this node identify the node connected to this port
196
K. Pugh / Configuration Discovery and Mapping of a Home Network
The mapping of a network is achieved by visiting all of the nodes in the network to acquire details of their configuration. Nodes are discovered in order, the closest nodes first. As the discovery progresses the configuration knowledge is extended by one link for each request. The discovery process begins by interrogating the node linked directly to the controller. If the information obtained relates to a routing switch, configuration details are requested from each of the nodes connected to the ports of that routing switch. This procedure is repeated until the discovery is complete. The information required from each node is its unique identifier, the number of ports at the node, and the transmission rate that each of its output port(s) can sustain, hi addition, the interrogated node provides the number of the input port on which the identify request is received. Table 1: adjacency list format node identifier route from controller to node route from node to controller number of ports node (identifier) & port number connected to port #1 port #1 link speed node (identifier) & port number connected to port #2 port #2 link speed
node (identifier) & port number connected to port #n port #n link speed
From this information an adjacency list [10] is constructed to describe the network topology. The format of the information in each row of the adjacency list is shown in Table 1. At the start of the discovery cycle the only node of which the controller is aware is itself. Consequently, the only details registered in the adjacency list are those of the controller. As each new node is discovered the adjacency list is extended to register the details of the new node. 4.1 Concurrent Discovery At any moment, during the discovery cycle, there may be one or more configuration inquiries in progress, but the order in which their responses arrive is unpredictable. A means of associating a response with the request that produced it is needed in order to locate the relevant entry in the adjacency list. To facilitate this tracability the node identifier and port number of the investigating node are included in the identify request. These data are echoed in the corresponding response, and used subsequently as an index to the relevant fields in the adjacency list.
K. Pugh / Configuration Discovery and Mapping of a Home Network
197
4.2 Closed Loops The freedom to link the nodes of the network arbitrarily in any configuration creates the possibility for interconnected nodes to form closed loops. In Figure 2 there are many closed loops - for example, the links between any two adjacent nodes, or the sequence of nodes rtrl to rtr2 to rtr3 and back to rtrl. While alternative links between nodes increases the bandwidth between nodes, and is beneficial for the robustness of the network it could lead rapidly to a situation in which the discovery process is continually rediscovering the same set of nodes. To avoid this situation a node's unique identifier is returned as part of the node's configuration details. The identifier enables the discovery process to detect whether a node has been visited previously during the discovery cycle. If the node's identifier is not recognised then the node must be a new discovery and a new entry can be created for it in the adjacency list. Conversely, if the identifier is recognised, it must have been returned in response to an earlier request, and the discovery process need only update the relevant port connection information in the existing adjacency list entry. 4.3 Disconnected L inks Routing switches return a link speed of '0' (zero), if a port is not connected. Identify node requests are not sent to ports that are not connected. However, a link could become disconnected during the interval between sending a request and the arrival of the corresponding response, in this situation the discovery process could wait indefinitely for the response. To resolve this problem the response to an identify node request must be received within a bounded time. If the discovery process does not receive a response within this time it is assumed that there is no connection on the selected port, and it is assigned a link speed of '0'. The use of a time out period, however, can lead to other difficulties if a response arrives too late. 4.4 Orphaned Responses The transfer path lengths in a free topology network cannot be predicted. Consequently, the time needed for a packet to propagate through the network is indeterminate. It is possible, therefore, for a response to be received after the timeout period has expired. If there has been a change of configuration, since the previous discovery cycle, the information contained in the response may be incorrect and lead to numerous inappropriate discovery investigations. To avoid this problem, a process cycle identifier is included in each identify node request. This cycle identifier is returned by the target node in its response, enabling the discovery process to detect, and subsequently discard, late responses. 4.5 Deadlock An essential property of a communication network is that it does not deadlock, i.e. arrive in a state where further progress of a packet between nodes is impossible. Whilst every attempt is made to ensure the configuration discovery process is free from deadlock (for example, applying the design rules for communicating processes advocated by Welch et al. [11]), the late arrival of a response to an identify node request, during the interval between the controller completing one discovery cycle and beginning the next, has the potential to cause a deadlock. To illustrate this deadlock problem consider the model of a routing switch shown in Figure 3.
198
K. Pugh / ConJguration Discovery and Mapping of a Home Network
Figure 3: deadlock scenario The controller’s output process begins sending an identify node request. Meanwhile, a (late) response from the network arrives at port in[y], and is transferred to port out[x] en route to the controller. This response can make no W e r progress towards the controller since the controller is busy sending the identify node request and consequently has no input process ready to receive the response. The identify node request from the controller arrives at port in[x] of the routing switch, and is passed to the identify node process via the internal management port out[O]. Immediately, the identify node process begins its response by echoing the return path fiom node field, etc. The identify node response is sent to the internal management port in[O] from where it must be transferred to port out[x], and on to the controller. But port out[x] already contains data from the late response and cannot accept the new response data. The controller’s output process is now blocked, and the system is deadlocked. This deadlock problem can be avoided by either: ensuring that the packet containing the entire identi@ node request is delivered to the routing switch, or guaranteeing that the routing switch is able to forward the late response to the controller. In either case we must ensure the controller is able to complete its output process and proceed to its input process. Both remedies can be accomplished by providing a buffer to store and forward a packet. The decision of which solution to choose can be decided by the size of the buffer needed to store the packet. For a free topology network the length of the path to (and from) a node is variable, therefore the length of the routing information in the identify node request packet is unknown. Consequently, the size of the buffer needed to store the packet is indeterminate. Whereas, the size of buffer needed to store a response is known because the routing information will have been stripped off before the response is sent on the final stage of its journey to the controller. In addition, the use of a buffer to store incoming responses can be refined W e r by monitoring the cycle identifier of responses and discarding those that do not contain the current cycle identifier.
K, Pugh / Configuration Discovery and Mapping of a Home Network
5
199
Configuration Discovery Protocol
Because of the constraints of many of the nodes in the network the discovery protocol is implemented using low level message passing techniques that require a minimum of processing resources in each node. 5.1 Identify Node Request path to node
identify request ;
identify reply
reply path from node
;
| cycle id ! sender node id & port
eop
An identify node message is sent to the nodes of a network requesting configuration details of any node connected to their output ports. The information required from each node are its unique identifier, the number of ports at the node, and the transmission rate that each of its output port(s) can sustain. In addition, the interrogated node provides the number of the input port on which the identify request is received. In detail: • The path to node contains the packet header to route the message through the network's routing switches to its destination node. The header is comprised of the series of output port numbers that lead from the controller to the node that is the object of the identify request. The final port number in the header is addressed to the internal management port #0. • The identify request routes the message to the configuration discovery handling process once it has arrived at its destination node. • The reply path from node contains the header needed by the object node to return the response through the network's routing switches to the controller. The header is comprised of the series of output port numbers that lead from the object node to the controller. The final port number in the header is addressed to the internal management port #0. • The identify reply routes the response to the discovery handling process in the controller. • The cycle id identifies a complete cycle of the discovery process to enable late responses to be detected. • The sender node id & port identify the node, and its port, currently under investigation. There may be several identify node requests in progress simultaneously. The sender node id & port are returned in the response message to provide an index that enables the discovery process to locate the relevant entries into the adjacency list. To simplify the processing needed at the destination node, the reply path from node, identify reply, cycle id and sender node id & port fields are embedded in the outgoing identify node request message, and are echoed by the destination node.
200
K. Pugh / Configuration Discovery and Mapping of a Home Network
5.2 Identify Node Reply The destination node responds to an identify node message with an identify node reply message. The reply path, identify reply, cycle id and sender node id & port fields are echoed from the identify node request message. reply path
identify reply
; cycle id ; sender node id & port ;
: responder node id ! incoming port ! link speed(s) I eop
The fields are: • The reply path contains the header needed by the object node to return the response through the network's routing switches to the controller. The header is comprised of the series of output port numbers that lead from the object node to the controller. The final port number in the header is addressed to the internal management port #0. • The responder node id is the identifier of the connected node. • The incoming port is the port on which the responder node received the identify node request. This information is used to route subsequent responses to the controller. • The link speeds declare the transmission capability of the nodes outgoing link(s). There is one link speed entry for each port on the responder node. If there is no connection at the port a link speed of zero is returned. Table 2 shows the mapping table that is produced when the configuration discovery process is applied to the network in Figure 1.
6
Conclusions
This paper describes a simple discovery protocol suitable for mapping the configuration of an irregular home network comprised of routing switches and heterogeneous devices. In software simulations the discovery process has mapped a variety of irregular network configurations correctly, and has proven to be robust in its operation over a period of several weeks. In hardware demonstrations involving simple networks [12], the discovery process has proven effective also. The configuration data obtained during the discovery process is essential for the tasks planned for the future, including, the evaluation of quality of service requirements, and the provision of both unicast and multicast message routing strategies.
Acknowledgements The author is grateful to the EPSRC and to 4Links Limited for their sponsorship of this work.
K. Pugh / Configuration Discovery and Mapping of a Home Network
201
Table 2: adjacency list for the Figure 2 network Node identity
Links*
Controller to node route
Node to controller route
devA
[1,2,1]
[1,2,1]
1
rtr2/l,(10)
rtr4
11, 3J
[3,1]
4
devC/l,(10)
rtr3/3, (70)
rtrl/3,(80)
rtr2/4, (250)
rtr3
IU41
[4,1]
4
devB/l,(10)
rtr2/3,(50)
rtr4/2, (60)
rtrl/4, (150)
rtr2
[1,2]
[2,1]
4
devA/l,(10)
rtrl /2, (30)
rtr3/2, (40)
rtr4/4, (200)
rtrl
(1)
[1]
4
ctrl/1, (10)
rtr2/2, (20)
rtr4/3, (90)
rtr3/4, (100)
Ctrl
I.J
[]
1
rtrl/l,(10)
devC
[1,3,1]
[1,3,1]
1
rtr4/l,(10)
devB
[1,4,1]
[1,4,1]
1
rtr3/l,(10)
Number of ports port 1
port 2
port 3
port 4
* The node's link(s) port details contain the node/port this port is connected to, and link transmission speed.
References [1]
IEEE 1355 (ISO/IEC 14575) Rationale, http://grouper.ieee.org/groups/1355. 1998.
[2j
IEEE 1355-1995 Standard for Heterogeneous Interconnect (HIC), "Low-Cost, Low-Latency Scalable Serial Interconnect for Parallel System Construction".
[3]
B. M. Cook and N. H. White, "Java Joins IEEE 1355 in the Home Network", Architectures, Languages and Techniques for Concurrent Systems, Edited by B.M.Cook, IOS Press. ISBN 90 5199 480 X, 1999.
[4]
S. Haas, D.A. Thornley, M. Zhu, R.W. Dobinson, B. Martin, "The Macrame 1024 Node Switching Network". Microprocessors and Microsystems, No. 21, pp. 511-518, Elsevier, 1998.
[5]
S. Haas, D.A. Thornley, M. Zhu, R.W. Dobinson, N. A. Madsen and B. Martin. "Results from the Macrame 1024 Node Switching Network". Microprocessors and Microsystems, No. 21, pp. 206-210, Elsevier, 1998.
[6]
Networks, Routers and Transputers: Function, Performance and Applications. Edited by M.D.May, P. W.Thompson and P.H.Welch, IOS Press, ISBN 90 5199 129 0, 1993.
[7]
W. J. Dally and C. L. Seitz. The Torus Routing Chip, Journal oj Distributed Computing, Vol. 1, No. 3, pp. 187-196, 1986.
[8]
P. Walker, "ATM performance with RS232 cost for links between microprocessors - using the IEEE 1355 standard".
[9]
Digital Equipment Corporation. "Autonet: A High-speed Self-Configuring Local Area Network Using Point-to-Point Links". SRC Research Report 59, 1990.
[10]
A.V. Aho and J.D. Ullman, Foundations of Computer Science, C Edition. Computer Science Press 1995. ISBN 1 716782847.
202
K. Pugh / Configuration Discovery and Mapping of a Home Network
[11]
P.H.Welch, G.R.RJusto, and CJ.Willcock. "Higher-Level Paradigms for Deadlock-Free HighPerformance Systems" In R.Grebe et al. editors, Transputer Applications and Systems '93, Proceedings of the 1993 World Transputer Congress, Vol. 2, pp. 981-1004, Aachen, Germany. IOS Press, Netherlands. ISBN 90-5199-140-1. September 1993.
[12]
4Links Limited, http://www.4links.co.uk/. 2002.
Communicating Process Architectures 2002 James Pascoe, Peter Welch, Roger Loader and Vaidy Sunderam (Eds.) IOS Press, 2002
203
Brian Vinter Department of Mathematics and Computer Science University of Southern Denmark, Odense, Denmark [email protected]
Peter H. Welch Computing Laboratory, University of Kent at Canterbury, CT2 7NF, England P. [email protected]
Abstract. Hoare's algebra of Communicating Sequential Processes (CSP) enables a view of systems as layered networks of concurrent components, generating and responding to events communicated to each other through channels, barriers and other (formally defined) synchronisation primitives. The resulting image and discipline is close to hardware design and correspondingly easy to visualise, reason about, compose and scale. JCSP is a library of Java packages providing an (occam) extended version of this model that may be used alongside, or as a replacement for, the very different threads-and-monitors concurrency mechanisms built into Java. The current release (JCSP 1.0) supports concurrency within a single Java Virtual Machine (which may be multi-processor). This paper reports early experiments with JCSP.net, an extension of JCSP for the dynamic construction of CSP networks across distributed environments. The aims of JCSP.net are to simplify the construction and programming of dynamically distributed and parallel systems. It provides high-level support for CSP architectures, unifying concurrency logic within and between processors. The experiments are on some classical HPC problems, an area of work for which JCSP.net was not primarily designed. However, low overheads in the supporting infrastructure were a primary consideration - along with an intuitive and high-level distributed programming model (based on CSP). Results reported show JCSP holding up well against — and often exceeding — the performance obtained from existing tools such as mpiJava and IBM's TSpaces. The experimental platform was a cluster of 16 dual-processor PHI Linux machines. It is expected that future optimisations in the pipeline for the JCSP.net infrastructure will improve the results presented here. JCSP and JCSP.net were developed at the University of Kent.
1
Introduction
The aims of JCSP.net[\] are to provide simple ways to build efficient, richly functional, scalable, distributed and dynamically evolving systems. Its primitives are based on Hoare's algebra of Communicating Sequential Processes (CSP[2, 3, 4]), allowing applications to be formally and directly modelled. It enables the expression of concurrency at all levels of system - for example between parallel running devices (whether home, cluster or Internet distributed) or within a single machine (which may be multi-processor). The same concepts and theory are used regardless of the physical realisation of the concurrency - shared memory and distributed memory systems are modelled and programmed in the same way. This is especially useful for exploiting the mixed parallel architecture (CLUMPS) of our target platform, with no different programming techniques needed to deal with each style.
204
B. Vinter and P.M. Welch / Cluster Computing and JCSP Networking
JCSP[5, 6, 7, 8, 9] views the world as layered networks of communicating processes, each layer itself being a process. Processes do not interact directly with other processes only with CSP sychronisation objects (such as communication channels, event barriers, shared-memory CREW locks) to which groups of processes subscribe. In this way, race hazards are largely designed out and deadlock/livelock analysis simplified (and, if necessary, formalised in CSP for mechanically assisted verification). The strong decoupling of the logic of each process from all others means we only have to consider one thing at a time - regardless of whether that thing is a process with a serial implementation or a network layer of concurrent sub-processes. A consequence is that CSP concurrency logic scales well with complexity (as well as distributing naturally across physically parallel systems). Such properties are not held by the built-in Java concurrency model (nor by parallel machine models such as MPI or Linda). JCSP follows the CSP model pioneered by the occam[10, 11] concurrency language in the mid-1980s, which was the first commercial realisation of the theory that was both efficient and secure. It extends the occam model by way of some of the proposed extras (such as shared channels) for the never implemented occam3[12] language - and by taking advantage, carefully, of the dynamic features of Java. The channel mechanism of CSP lends itself naturally to distributed memory computing indeed, this was one of the most exciting capabilities of occam systems mapped to transputer networks. Currently, JCSP provides support for shared-memory multiprocessors and, of course, for uniprocessor concurrency. This paper presents work with an early (alpha) release of JCSP.net, an extension of JCSP that extends the CSP concurrency model to networked systems. It enables the dynamic distribution and connection of JCSP processes with no central or pre-planned control (although that is easy to impose if necessary). Details of the underlying network fabric and control (such as multiplexing, flow-control, network addresses) are hidden from the JCSP programmer, who works entirely at the application level using CSP communication and synchronisation primitives. A simple brokerage service - based on channel names (the Channel Name Server, CNS) - is provided to let distributed JCSP components find and connect to each other. Distributed JCSP networks may securely evolve, as components join, leave and migrate at run-time with no centralised or pre-planned control. Processes themselves need not be aware that they are networked, which enables simple re-configuration and load balancing. The seamless unification of external and internal concurrency models, the dynamic construction of networked applications and the formal compositional basis of the underlying CSP model (which may either be ignored by users or employed directly for formal analysis and/or model checking) will be of interest, we hope, to workers in the field of HPC. This paper assumes some familiarity with core JCSP mechanisms - channel interfaces, concrete channels (one-one, any-one etc.), synchronised communication, buffered plugins for channels, processes, parallel process constructors and alternation (i.e. the passive waiting for one from a set of events to happen). The paper is structured as follows. Section 2 presents the basic model, semantics and API provided by JCSP.net, along with some generic templates for standard parallel decomposition paradigms (farming and space division). Section 3 describes the design aims for the DANISH benchmark suite. Section 4 outlines the alternative parallel mechanisms used for comparison (mpUava and IBM's TSpaces). Section 5 presents the applications benchmarked in this work and Section 6 gives the results. Finally, some concluding remarks are made in Section 7.
B. Vinter and P.H. Welch/ Cluster Computing and JCSP Networking
205
2 Motivation We want to use the same concurrency model regardless of the physical distribution of the processes of the system.
Figure l(a). Single processor system
Figure 1 (b). Two processor system
Figure l(a) shows a six process system running on a single processor (M). Figure l(b) shows the same system mapped to two machines (P and Q), with some of its channels stretched between them. The semantics of the two configurations should be the same - only performance characteristics may change. 2.1 Basic Infiastructure JCSP.net provides mechanisms similar to the Virtual Channel Processor (VCP) of the T9000 transputer [13]. Users just name networked channels they wish to use. Details of how the connections are established, the type of network used, machine addresses, port numbers, the routing of application channels (e.g. by multiplexing on to a limited supply of socket connections) and the generation and processing of acknowledgement packets (to preserve synchronisation semantics for the application) are hidden. The same concurrency model is used for networked systems as for internal concurrency. Processes may be designed and implemented without caring whether the synchronisation primitives (channels, barriers etc.) on which they will operate are networked or local. This is as it should be. The user does not need to know how, for example, Figure l(b) is implemented. In fact, a bi-directional link is established between the two machines and the application-level channels are multiplexed over this - see Figure 2.
206
B. Vinter and P.H. Welch / Cluster Computing and JCSP Networking
Figure 2. Channel multiplexing over a link
JCSP.net establishes one - and only one - such link between each pair of machines that contain processes wishing to communicate. Details of the underlying mechanisms (including a very simple software crossbar connecting application-level processes with link driver processes) may be found in [ 11. 2.2 Semantic Integrity
JCSP.net designers and programmers need no knowledge of the infi-astructure outlined in the previous subsection. JCSP processes only see channel interfaces - for the most part, they need not even be aware whether the actual channels plugged into them are local or networked. It is essential, however, that link multiplexing preserves the independence of the channels carried. So, the semantics of the networks in Figures 2 and l(b) remain identical. For example, if process E is slow at taking (or simply refuses to take) input from its networked channel, this must not block communications from A to D nor from F to C. Like normal JCSP channels, users may specify any amount (and several varieties) of buffering for networked channels - with zero buffering being the default. The link infrastructure introduces no additional buffering. Hence, the synchronisation semantics of Figures l(b) and l(a) are identical. However, there are two caveats. Firstly, current JCSP.net network drivers use Java serialisation to transmit and receive objects - so objects for networked communication must implement the Java serial i zable interface. Secondly, networked channel communication has copy-semantics - whereas across internal channels, communication is by reference. Fortunately, for secure JCSP designs (i.e. those free from shared-reference race-hazards), this will make no difference.
2.3 Establishing Network Channels JCSP.net does not require applications to be pre-configured statically. For example, the number of participating machines or application processes need not be known in advance nor the mapping of the latter to the former. The model is very dynamic - processes find each other by knowing (or finding out) the names (or locations) of the network channels they need to use. This is much more flexible than the static configuration offered, for example, by MPI. At the lowest level, this mechanism is brokered by the JCSP.net Chnnef Name Server (CNS), which maintains a table of network addresses (e.g. IP-addresdport-number, depending on the type of network), de-multiplexing keys (for navigating the software crossbar implementing the JCSP.net infrastructure on each node) and channel types (e.g. streamed, overwriting) against user-defined channel names (which can have a rich structure akin to URLs).
B. Vinter and P. H. Welch / Cluster Computing and JCSP Networking
207
Within a processing node, which may itself be SMP, networked application channels are either to-the-net or from-the-net. These channels may have one or any number of application processes attached - which gives us the set {One2NetChannel, Any2NetChannel, Net2OneChannel, Net2AnyChannel}. The '2Net' channels implement only write methods and the 'Net2' channels only reads. In line with core JCSP[5, 6], only the '2One' channels may be used as guards in an Alternative. To construct networked channels using the CNS, we only need to know their names. For example, at the reading end, the statement: Net20neChannel foo = new Net2OneChannel ("ukc.foo");
constructs the channel and necessary local multiplexing infrastructure, registering the name "ukc .foo" with the CNS (along with the network address of the constructing node and navigation information to enable correct de-multiplexing). If the CNS cannot be found or the name is already registered, an exception is thrown. Details of that supporting local infrastructure are reported in [1]. Note that this foo channel only has a read method. At the writing end, the statement: One2NetChannel foo = new One2NetChannel ("ukc.foo");
interrogates the CNS for registration details on "ukc. foo", blocking until that registration has happened (time-outs may be set that throw exceptions), and constructs the channel together with its supporting infrastructure. This foo channel, which is in a different process and on a different machine to the previous one, only has a write method. Each node maintains a table of machines to which it has open links. If a link does not exist to the machine registered as the reading-end of "ukc.foo", that machine is contacted and the link is made (and the tables at each end updated). If network channels are being constructed sequentially by an application, all input channels should be set up first — otherwise, there will be deadlock! JCSP.net channels are unidirectional and support any-one communication across the network. This means that processes on any number of remote machines may open a named output network channel and use it safely — their messages being interleaved at the input end. 2.4 Networked Connections (Client-Server) Establishing two-way communication, especially for a prolonged conversation between one of many remote clients and a common server, can be done with channels ... but is not particularly simple or elegant. Individual network reply channels to each remote machine process are needed, together with a network sync channel to queue the conversations and a network question channel. So, JCSP.net provides connections, which give a high-level abstraction for two-way channels for use in networked client-server applications. For example, at the server end: Net2OneConnection barO = new Net20neConnection ("ukc.barO");
sets up the infrastructure needed to service a bidirectional network connection, registering the name "ukc. barO" (and all relevant type and location information) with the CNS.
208
B. Vinter and P.M. Welch / Cluster Computing and JCSP Networking
At each client end of the connection, the statement: One2Net Connect ion barO = new One 2Net Connect ion ("ukc.barO") ;
sets up the network client infrastructure, getting the server details from the CNS. Application processes only see client and server interfaces (rather than writer and reader ones) to these connections. Each provides two-way communications with support for an extended and uninterrupted conversation: interface Connect ionClient { public void writeRequest (Object o) ; public Object readReply ( ) ; public boolean stillOpen ();
// if unsure
interface Connect ionServer { public Object readRequest () ; public void writeReply (Object o) ; public void writeReplyAndClose (Object o) ;
// keep open // close
JCSP.net connections are bi-directional and support any-one client-server transactions across the network. This means that any number of remote machines may open a named client network connection and use it safely - transactions being dealt with atomically at the server end [1]. A transaction consists of a sequence of request-reply pairs, ending with the server making a write-request-and-close reply. If it doesn't know, the client can find out when the transaction is finished by invoking stillOpen ( ) on its end of the connection. The server infrastructure locks the server application process to the remote client for the duration of the transaction. Other clients have to wait. Any application process attempt to violate the alternating request-reply sequence will raise a (run-time) exception. A client must commit to read reply messages from its server (and, if the transaction was kept open, make a follow-on request). A server must commit to read follow-on request messages from its client, if it kept the transaction open with its last reply. This is sufficient to keep them synchronised with no additional system acknowledgements generated across the network - connection transactions are self-synchronising. Note that connections may not be buffered. Note, also, that a connection is not open until the first reply has been received. Clients may open several connections at a time - but only if all clients open