This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
, which points into a contiguous subset of a Java array. In lines –, the array is A) writes P:* { /* Ordinary quicksort partition */ int p = quicksortPartition(A); /* Split array int two disjoint pieces */ final DPJPartition segs = new DPJPartition (A, p); cobegin { /* writes segs:[0]:* */ quicksort(segs.get(0)); /* writes segs:[1]:* */ quicksort(segs.get(1)); } }
D
D
Deterministic Parallel Java
class Body
Deterministic Parallel Java. Fig. Code using an index-parameterized array 10
Deterministic Parallel Java. Fig. Parallel quicksort
90
...
...
...
Body
Body
double force Root : [10]
double force Root : [90]
Deterministic Parallel Java. Fig. Runtime representation of the index-parameterized array segs
DPJPartition
DPJArray<segs:[0]:*>
static
DPJArray<segs:[1]:*>
p
Deterministic Parallel Java. Fig. Runtime representation of the dynamically partitioned array
split at index p, creating a DPJPartition object holding references to two disjoint subarrays. The cobegin in lines – calls quicksort recursively on these two subarrays. Figure shows what this partitioning looks like at runtime, for the root of the tree and its children. A
DPJPartition object referred to by the final local variable segs points to two DPJArray objects, each
of which points to a segment of the original array. The DPJArray objects are instantiated with owner RPLs segs:[0]:* and segs:[1]:* in their types. An owner RPL is like an RPL, except that it begins with a final local reference variable instead of Root. This allows different partitions of the same array to be represented. Again because of the tree structure of DPJ regions, segs:[0]:* and segs:[1]:* are disjoint region sets, and so the effects in lines and are noninterfering. Also, as in ownership type systems, region segs is under the region P bound to the first parameter of its type, so the effect summary writes P:* in line covers the effects of the method body.
Commutativity Annotations Sometimes, a parallel computation is deterministic, even if it has interfering reads and writes. For example, two inserts to a concurrent set can go in either order, and preserve determinism, even though there are interfering writes to the set object. DPJ supports this kind of computation with a commutativity annotation on methods (typically API methods for a concurrent data structure). For example: interface Set
Deterministic Parallel Java
This annotation is provided by a library or framework programmer and trusted by the compiler (it is not checked by the DPJ effect system). It means that cobegin { add(e1); add(e2); } isequivalentto doingthe add operationsinsequence(isolation) and that both sequence orders are equivalent (commutativity). When the compiler encounters a method invocation, it generates an invocation effect that records both the method that was invoked, and the read-write effects of that method: foreach (int i in 0, n) { /* invokes Set.add with writes R */ set.add(A[i]); }
The compiler uses the commutativity annotations to check interference of effect. For example, the effect invokes Set.add with writes R is noninterfering with itself (because add is declared commutative). However, the same effect interferes with invokes Set. size with reads R.
Expressivity and Limitations DPJ can express a wide variety of algorithms that use a fork-join style of parallelism and access data through a combination of () read-only sharing; () disjoint updates to arrays and trees; () commutative operations on concurrent data structures like sets and maps; and () accesses to task-private heap objects. DPJ has been used to write the following parallel algorithms, among others: IDEA encryption, sorting, k-means clustering, an n-body force computation, collision detection algorithm from a game engine, and a Monte Carlo financial simulation. The main advantages of the DPJ approach are () a compile-time guarantee of determinism in all executions; and () no overhead from checking determinism at runtime, as the region information is erased before parallel code generation. Further, the effect annotations provide checkable documentation and enable modular analysis, and the defaults allow incremental transformation of sequential to parallel programs. DPJ does have some limitations. One limitation, mentioned above, is that currently only fork-join parallelism can be expressed. A second limitation is that
D
to ensure soundness, the language disallows some patterns that are in fact deterministic, but cannot be proved deterministic by the type system. One example is that the type system disallows reshuffling of the elements in an index-parameterized array. This limitation poses a problem for the Barnes-Hut n-body force computation, as rearranging the array of bodies is required for good locality. A workaround for this limitation is to recopy the bodies on reinsertion in to the array, but this is somewhat clumsy and could degrade performance. A third limitation is that some programmer effort is required to write the DPJ annotations, although this effort is repaid by the benefits discussed above.
Ongoing Work The foregoing describes the state of the public release of DPJ as of October . The following subsections describe ongoing work that is expected to be part of future releases.
Controlled Nondeterminism DPJ is being extended to support controlled nondeterminism [], for algorithms such as branch and bound search, where several correct answers are equally acceptable, and it would unduly constrain the schedule to require the same answer on every execution. The key new language features for controlled nondeterminism are () foreach_nd and cobegin_nd statements that operate like foreach and cobegin, except they allow interference among their parallel tasks; () atomic statements supported by software transactional memory (STM); and () atomic regions and atomic effects for reasoning about which memory regions may have interfering effects, and where effects occur inside atomic statements. Together, these features provide several strong guarantees for nondeterministic programs. First, the extended language is deterministic by default: the program is guaranteed to be deterministic unless the programmer explicitly introduces nondeterminism with foreach_nd or cobegin_nd. Second, the extended language provides both strong isolation (i.e., the program behaves as if it is a sequential interleaving of atomic statements and unguarded memory accesses) and data race freedom. This is true even if
D
D
Deterministic Parallel Java
the underlying STM provides only weak isolation (i.e., it allows interleavings between unguarded accesses and atomic statements). Third, foreach and cobegin are guaranteed to behave as isolated statements (as if they are enclosed in an atomic statement), even inside a foreach_nd or cobegin_nd. Finally, the extended type and effect system allows the compiler to boost the STM performance by removing unnecessary synchronization for memory accesses that can never cause interference.
Object-Oriented Parallel Frameworks DPJ is being extended to support object-oriented parallel frameworks. Current support focuses on two kinds of frameworks: () collection frameworks like Java’s ParallelArray [] that provide parallel operations such as map, reduce, filter, and scan on their elements via user-defined methods; and () a framework for pipeline parallelism, similar to the pipeline framework in Intel’s Threading Building Blocks []. The key challenge is to prevent the user’s methods from causing interference when invoked by the parallel framework. For example, a user-defined map function must not do an unsynchronized write to a global variable. The OO framework support incorporates several extensions to the DPJ type and effect system for expressing generic types and effects in framework APIs, with appropriate constraints on the effects of user-defined methods. Frameworks represent one possible solution to the problem of array reshuffling and other operations that the type system cannot express. Such operations can be encapsulated in a framework. The type and effect system can then check the uses of the framework, while the framework internals are checked by more flexible (but more complex and/or weaker) methods such as program logic or testing. This approach separates concerns between framework designer and user, and it fosters modular checking. It also enables different forms of verification with different trade-offs in complexity and power to work together. In addition, frameworks provide a natural way to extend the fork-join model of DPJ by adding other parallel control abstractions, including finer-grain synchronization (e.g., pipelined loops) or domain-specific abstractions (e.g., kd-tree querying and sparse matrix computations).
Inferring DPJ Annotations Research is underway to develop an interactive tool that helps the user write DPJ programs by partially inferring annotations. A tool called DPJizer has been developed [] that uses interprocedural constraint solving to infer method effect summaries for DPJ programs annotated with the other type, region, and effect information. DPJizer simplifies DPJ development by automating one of the more tedious, yet straightforward, aspects of writing DPJ annotations. The tool also reports the effect of each program statement, helping the programmer to isolate and eliminate effects that are causing unwanted interference. The next step is to infer region information, given a program annotated with parallel constructs. This is a more challenging goal, because there are many more degrees of freedom. The current strategy is to have the programmer provide partial information (e.g., partition the heap into regions and put in the parallel constructs), and then have the tool infer types and effects that guarantee noninterference. Runtime Checks Research is underway to add runtime checking as a complementary mode for guaranteeing determinism, in addition to the compile-time type and effect checking. The advantage of runtime checking is that it is more precise. The disadvantages are () it adds overhead, so it is probably useful only for testing and debugging; and () it provides only a fail-stop check, instead of a compiletime guarantee of correctness. One place where runtime checking could be particularly useful is in relaxing the prohibition on inconsistent type assignments (e.g., in reshuffling an index-parameterized array). In particular, if the runtime can guarantee that a reference is unique, then a safe cast to a different type can be performed at runtime. Using both static and runtime checks, and providing good integration between the two, would offer the user a powerful set of options for managing the trade-offs among the expressivity, performance, and correctness guarantees of the different approaches.
Related Entries Determinacy Formal Methods–Based Tools for Race, Deadlock, and Other Errors Transactional Memories
Distributed Memory Computers
Bibliographic Notes and Further Reading For further information on the DPJ project and its goals, see the DPJ web site [] and the position paper presented at HotPar []. The primary reference for the DPJ type and effect system is the paper presented at OOPSLA []. A paper on the nondeterminism work will appear at POPL []. The Ph.D. thesis of Robert Bocchino [] presents the full language, together with a formal core language and effect system, soundness results for the core system, and proofs. A paper presented at ASE [] describes the work on inferring method effect summaries. The main influence on DPJ has been the prior work on type and effect systems, including FX [] and the various effect systems based on ownership types [, ]. Other influences include work on design-by-contract for framework APIs [], work on type and effect inference [], and the vast body of literature on transactional memory [].
Bibliography . http://gee.cs.oswego.edu/dl/jsr/dist/extraydocs/index.html? extray/package-tree.html . http://dpj.cs.uiuc.edu . Bocchino RL Jr () An effect system and language for deterministic-by-default parallel programming. PhD thesis, University of Illinois, Urbana-Champaign . Bocchino RL Jr, Adve VS, Adve SV, Snir M () Parallel programming must be deterministic by default. In: HOTPAR ’: USENIX workshop on hot topics in parallelism, Berkeley, March . Bocchino RL Jr, Adve VS, Dig D, Adve SV, Heumann S, Komuravelli R, Overbey J, Simmons P, Sung H, Vakilian M () A type and effect system for deterministic parallel Java. In: OOPSLA ’: Proceedings of the th ACM SIGPLAN conference on object-oriented programming systems, languages, and applications, New York, pp – . Bocchino RL Jr, Heumann S, Honarmand N, Adve S, Adve V, Welc A, Shpeisman T, Ni Y () Safe nondeterminism in a deterministic-by-default parallel language. In: POPL ’: Proceedings of the th ACM SIGACT-SIGPLAN symposium on principles of programming languages, New York . Cameron NR, Drossopoulou S, Noble J, Smith MJ () Multiple ownership. In: OOPSLA ’: Proceedings of the nd ACM SIGPLAN conference on object-oriented programming systems and applications, New York, pp – . Clarke D, Drossopoulou S () Ownership, encapsulation and the disjointness of type and effect. In: OOPSLA ’: Proceedings of the th ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, New York, pp –
D
. Harris T, Larus J, Rajwar R () Transactional memory (Synthesis lectures on computer architecture), nd edn. Morgan & Claypool Publishers, San Rafael . Lucassen JM, Gifford DK () Polymorphic effect systems. In: POPL ’: Proceedings of the th ACM SIGPLAN-SIGACT symposium on principles of programming languages, New York, pp – . Reinders J () Intel threading building blocks: outfitting C++ for multi-core processor parallelism. O’Reilly Media Sebastopol . Talpin J-P, Jouvelot P () Polymorphic type, region and effect inference. J Funct Program :– . Thomas P, Weedon R () Object-oriented programming in Eiffel, nd edn. Addison-Wesley Longman Publishing Co., Inc., Boston . Vakilian M, Dig D, Bocchino R, Overbey J, Adve V, Johnson R () Inferring method effect summaries for nested heap regions. In: ASE ’: Proceedings of the th IEEE/ACM International Conference on Automated Software Engineering, Auckland, pp –
Direct Schemes Dense Linear System Solvers
Distributed Computer Clusters Distributed-Memory Multiprocessor Hypercubes and Meshes
Distributed Hash Table (DHT) Peer-to-Peer
Distributed Logic Languages Logic Languages
Distributed Memory Computers Clusters Disturbed-Memory Multiprocess Hypercubes and Meshes
D
D
Distributed Process Management
Distributed Process Management Single System Image
Distributed Switched Networks Networks, Direct Networks, Multistage
Distributed-Memory Multiprocessor Marc Snir University of Illinois at Urbana-Champaign, Urbana, IL, USA
Synonyms Distributed computer; Distributed Memory Computers; Massively parallel processor (MPP); Multicomputers; Multiprocessors; Server farm
Definition A computer system consisting of a multiplicity of processors, each with its own local memory, connected via a network. Load or store instructions issued by a processor can only address the local memory, and different mechanisms are provided for global communication.
Discussion Terminology A Distributed-Memory Multiprocessor (DMM) is built by connecting nodes, which consist of uniprocessors or of shared memory multiprocessors (SMPs), via a network, also called Interconnection Network (IN) or Switch. While the terminology is fuzzy, Cluster generally refers to a DMM mostly built of commodity components, while Massively Parallel Processor (MPP) generally refers to a DMM built of more specialized components that can scale to a larger number of nodes and is used for large, compute-intensive tasks – in particular, in scientific computing. A server farm is a cluster
used as a server, in order to provide more performance or better cost-performance than a shared memory multiprocessor server. A high-availability (HA) cluster is a cluster with software and possibly hardware that supports fast failover so as to tolerate component failures. Finally, a cloud is a (generally very large) cluster that supports dynamic provisioning of computing resources, usually virtualized, to remote users, via the Internet.
Historical Background High-performance computing was focused until the late s on SMPs and vector computers built of bipolar gates. The fast evolution of VLSI technology indicated the possibility of achieving much better costperformance by assembling a larger number of less performing, but much cheaper, microprocessors. The node Cosmic Cube built at Caltech in the early s was an early demonstration of this technology []. Several companies started manufacturing such systems in the mid-s; a , node nCUBE system was installed at Sandia National Lab in the late s and a Transputer nodes system was running at Edinburgh in . The success of these systems, which was documented in the influential talk “The Attack of the Killer Micros” by Eugene D. Brooks III at the Supercomputing Conference, led to a flurry of commercial and governmental activities. Companies such as Thinking Machines (CM-), Intel (iPSC, Paragon), and Meiko (CS-) started offering Distributed-Memory Multiprocessor (DMM) products in the mid-s, followed by Fujitsu (AP), IBM (SP, SP), and Cray (TD, TE) in the early s. DSMs have dominated the TOP list of fastest supercomputers since the early s. The use of clusters evolved in parallel, and largely independently, in the commercial world, driven by several trends: Clusters were used in the early s for server consolidation, replacing multiple, distinct servers with one cluster, thus reducing operating costs. The explosive growth of the Internet in the late s led to a need for large clusters with a small footprint (to fit in downtown buildings) that can be quickly expanded, to serve Internet companies. Web requests can easily be served on a DMM, as each TCP/IP request is a small, independent action and request failures can be tolerated.
Distributed-Memory Multiprocessor
DMMs were also used to extend SMP architectures beyond the point where hardware could conveniently support coherent shared memory. Thus, IBM introduced in the parallel sysplex architecture that allowed coupling of multiple S/ mainframes into a larger cluster. The use of such clusters became more widespread as software vendors developed distributed computing firmware and adapted applications to run on clusters. Clusters have become increasingly used since the late s for providing fault tolerance. Rather than using very specialized hardware, such as in the early Tandem computers, High Availability (HA) Clusters provide fault tolerance through hardware replication that eliminates single points of failure, and suitable software. Many vendors (IBM, Microsoft, HP, Sun, etc.) offer HA cluster software solutions. Utility computing, namely the provisioning of computing resources (CPU time and storage) using a payper-use model, has been pursued by multiple companies over decades. Changes in technology (low-cost, highbandwidth networking, decline in hardware cost and relative increase in operation cost, increased use of offthe-shelf software and improved support for hardware virtualization) and in market needs (very rapid and unpredictable growth of the compute needs of Internet companies) have led to the recent success of this model, under the name of cloud computing. Cloud computing platforms are large clusters with software that supports the dynamic provisioning of virtualized compute and storage resources (hardware as a service – HaaS) and, possibly, firmware and application software (software as a service – SaaS)
Architecture At its simple most, a DMM can be assembled by connecting personal computers or workstations with a commercial network such as Ethernet, as shown in Fig. . The Beowulf system built in is an early example []. A more compact and more maintainable cluster can be assembled from rack-mounted servers, as shown in Fig. . In addition to the network that is used for communication between nodes, such a cluster will have a control network that is used for controlling and monitoring each node from a centralized console. The network of a DMM can be (a) a commodity Local Area Network (LAN), such as Ethernet;
D
D
Distributed-Memory Multiprocessor. Fig. Beowulf cluster
(b) a higher performance, third party switched network, such as Infiniband or Myrinet; or (c) a vendorspecific network, such as available on IBM and Cray systems. Communication over Ethernet will typically use the TCP/IP protocols; communication over nonEthernet fabrics can support lower latency protocols. Adapters for networks of type (a) and (b) will connect to a standard I/O bus, such as PCIe; vendor-specific networks can connect to the (nonstandard) memory bus, thus achieving higher bandwidth and lower latency. The basic communication protocol supported by any of these networks is message passing: Two nodes communicate when one node sends a message to the second node and the second node receives the message. Networks of type (b) and (c) often support remote direct memory access (rDMA): A node can directly access data in the memory of another node, using a get operation, or update the remote memory, using a put operation.
D
Distributed-Memory Multiprocessor
Network Attached Storage (NAS) technologies. Some commercial clusters, such as the IBM Parallel Sysplex, provide clustering hardware that accelerates data sharing and synchronization across nodes.
Software DMMs reuse, on each node, software designed for node-sized uniprocessors or SMPs. This includes the operating system, programming languages, compilers, libraries, and tools. An additional layer of parallel software is built atop the node software, in order to integrate the multiple nodes into one system. This includes, for compute clusters: ●
Distributed-Memory Multiprocessor. Fig. Rackmounted cluster
Support for such operations in user space requires an adapter that can translate virtual addresses to physical addresses. Oftentimes, such an adapter will also support read-modify-write operations on operands in remote memory, such as increment; this avoids the need for a round-trip for such operations and facilitates their atomic execution. Networks of type (c) may have hardware support for collective operations, such as barrier, broadcast, and reduce. The topology of the interconnection network impacts its performance: Networks with a low-dimension mesh topology (D or D) can be packaged so that only short wires are needed; this enables the use of cheaper copper cables while maintaining high bandwidth. Radixnetwork topologies, such as butterfly or fat-tree, reduce the average number of hops between nodes, but require longer wires []. Optical cables are needed for highspeed transmission over long distances. Commercial clusters (server farms and clouds) typically use Ethernet networking technology and communicate using TCP/IP protocols. The nodes often share storage, using Storage Area Networks (SAN) or
●
●
●
●
●
Software for system initialization, monitoring and control. This enables the control of a large number of nodes from one console; concurrent booting of a large number of nodes; network firmware initialization; error detection; etc. Scheduler and resource manager for allocating resources to parallel jobs, loading them and starting them. Parallel jobs often run on dedicated partitions; the scheduler has to allocate nodes in large chunks, while ensuring high node utilization and reducing waiting time for batch jobs. The resource manager has to reduce job-loading time (e.g., by using a parallel tree broadcast algorithm) and ensure that all nodes of a job start correctly. Parallel file system. Such a system integrates a large number of disks into one system and provides scalable file coherence protocols. Files are striped across many nodes so as to support parallel access to different parts of the same file and ensure load balancing. Software RAID schemes may be used to enhance the reliability of such a large file system. Checkpoint/restart facilities. On large DMMS, the mean time between failures is often shorter than the running time of large jobs. To ensure job completion, jobs periodically checkpoint their state; if a failure occurs, then job execution is restarted from the last checkpoint. A checkpoint facility supports coordinated checkpoint or restart by many nodes. Message passing libraries for supporting communication across nodes. MPI is the de facto standard for message passing on large DMMs. Tools for debugging and performance monitoring of parallel jobs. These are often built atop node
Distributed-Memory Multiprocessor
level debugging and performance monitoring; they enable users to control many nodes in debug mode, and to aggregate information from many nodes in debug or monitoring nodes. The use of conventional software on the nodes of DMMs is limiting; in particular, the use of a conventional operating systems on each node results in jitter: Background OS activities may slow down nodes at random times; applications where barriers are used frequently can slow down significantly as all threads wait at each barrier to the slowest thread. Some systems, e.g., the IBM Blue Gene system, are using on their compute node–specialized light-weight kernels (LWK) that provide only critical services and offload to remote server nodes heavier or more asynchronous system services. Commercial clusters use TCP/IP for inter-node communication, and distributed file systems for file sharing. They leverage firmware and application software designed for distributed systems, such as support for remote procedure calls (or remote method invocations), distributed transaction processing monitors, messaging frameworks, and “shared-nothing” databases. A critical component of server farms that handle Web requests is a load balancer that distributes TCP/IP requests coming on one port to distinct nodes. HA clusters have software that monitor the health of the system components and its applications (heartbeat) and report failures; software to achieve consensus after failures and to initiate recovery actions; and applicationspecific recovery software to restart each application in a consistent state. Disk state is preserved, either though mirroring, or through the sharing of highly reliable RAID storage via multiple connections. Clouds often use virtual machine software to virtualize processors and storage, thus facilitating their dynamic allocation. They need a resource management infrastructure that can allocate virtual machines on demand.
D
on DMMs in the late s was the conference on Hypercube Concurrent Computers and Applications). The abstract model used was of unit time communication on each edge, with communications either being limited to one incident edge per node at a time, or occurring concurrently on all edges incident to a node. For example, the problem of sorting on a hypercube is treated by several tens of publications. As indirect networks became more prevalent, significant amount of research focused on the topology of routing networks and cost-performance tradeoffs, and routing protocols for different topologies that are efficient and deadlock free, and that can handle failures. Modern DMMs largely hide the network topology from the user and may not provide mechanisms for controlling the mapping of processes and communications onto the network nodes and edges. Therefore, the topology is not part of the programming model. The abstract programming models used to represent such systems assume that communication time between two nodes is a function of the size of the message communicated. Two of the most frequently used models are the postal model [] and the logP model []. System research on DMMs has focused on communication protocols, operating systems, file systems, programming environments, and applications. Some of the larger efforts in this area are the Network of Workstation (NOW) effort at Berkeley [], and the SHRIMP project at Princeton [].
Related Entries Checkpointing Clusters Connection Machine Cray TE Distributed-Memory Multiprocessor Ethernet Hypercubes and Meshes IBM RS/ SP InfiniBand Meiko
Research
Moore’s Law
Early DMMs had directly connected networks; the network topology was part of the programming model. Theoretical research focused on the mapping of parallel algorithms to specific topologies; in particular, hypercubes and related networks (the primary conference
MPI (Message Passing Interface) Myrinet Network Interfaces Network of Workstations Networks, Direct
D
D
Ditonic Sorting
Networks, Multistage Parallel Computing PCI Express Routing (Including Deadlock Avoidance)
Bibliographic Notes and Further Reading The book of Fox et al. [] covers the early history of MPPs, their hardware and software, and the applications enabled on them. The book of Culler and Singh [] is a good introduction to MPP architectures and their networks, with a detailed description of the Cray TD and IBM SP networks. The book of Pfister [] is a general introduction to clusters, their design, and their use, while the two books of Buyya [, ] provide in-depth coverage of cluster hardware, software, and applications. The book of Dally and Towles [] provides a very good coverage of IN theory and practice. Finally, the book of Leighton [] is an excellent introduction to the theory results that are relevant to DMMs and INs.
Bibliography . Anderson TE, Culler DE, Patterson DA, the NOW team () A case for NOW (Networks of Workstations). IEEE Micro ():– . Bar-Noy A, Kipnis S () Designing broadcasting algorithms in the postal model for message-passing systems. In: Proceedings of the fourth annual ACM symposium on parallel algorithms and architectures SPAA ’. San Diego, California, United States, June –July , . ACM, New York, pp – . Blumrich MA, Li K, Alpert R, Dubnicki C, Felten EW, Sandberg J () Virtual memory mapped network interface for the SHRIMP multicomputer. In: Sohi GS (ed) years of the international symposia on computer architecture (selected papers) (ISCA ’). ACM Press, New York, pp – . Buyya R () High performance cluster computing: architectures and systems, vol . Prentice-Hall, New Jersey . Buyya R () High performance cluster computing: programming and applications, vol , Prentice-Hall, New Jersey . Culler DF, Singh JP () Parallel computer architecture: a hardware/software approach. Morgan Kaufman, San Francisco . Culler D, Karp R, Patterson D, Sahay A, Schauser EK, Santos E, Subramonian R, von Eicken T () LogP: towards a realistic model of parallel computation. In: Proceedings of the fourth ACM SIGPLAN symposium on principles and practice of parallel programming, San Diego, California, United States, – May , pp – . Dally WJ, Towles BP () Principles and practices of interconnection networks. Morgan Kaufmann, San Francisco . Fox GC, Williams RD, Messina PC () Parallel computing works, Morgan Kaufman, San Francisco
. Leighton FT () Introduction to parallel algorithms and architectures: arrays, trees, hypercubes. Morgan Kaufman, San Francisco . Pfister GF () In search of clusters, nd edn, Prentice Hall, New Jersey . Sterling T, Becker DJ, Savarese D, Dorband JE, Ranawake UA, Packer CV () BEOWULF: a parallel workstation for scientific computation. In: Proceedings of the th international conference on parallel processing. Oconomowoc, Wisconsin, pp –
Ditonic Sorting Bitonic Sort
DLPAR Dynamic Logical Partitioning for POWER Systems
Doall Loops Loops, Parallel
Domain Decomposition Thomas George , Vivek Sarin IBM Research, Delhi, India Texas A&M University, College Station, TX, USA
Synonyms Functional decomposition; Grid partitioning
Definition Domain decomposition, in the context of parallel computing, refers to partitioning of computational work among multiple processors by distributing the computational domain of a problem, in other words, data associated with the problem. In the scientific computing literature, domain decomposition mainly refers to techniques for solving partial differential equations (PDE) by iteratively solving subproblems corresponding to smaller subdomains. Although the evolution of these
Domain Decomposition
D
techniques is motivated by PDE-based computational simulations, the general methodology is applicable in a number of scientific domains not dominated by PDEs.
Introduction One of the key steps in parallel computing is partitioning the problem to be solved into multiple smaller components that can be addressed simultaneously by separate processors with minimal communication. There are two main approaches for dividing the computational work. The first one is domain decomposition, which involves partitioning the data associated with the problem into smaller chunks and each parallel processor works on a portion of the data. The second approach, functional decomposition, on the other hand, focuses on the computational steps rather than the data and involves processors executing different independent portions of the same program concurrently. Most parallel program designs involve a combination of these two complementary approaches. In this entry, we focus on domain decomposition techniques, which are based on data parallelism. Figure shows a -D finite element mesh model of the human heart [] used for simulation of the human cardiovascular system and in particular, the flow of blood through the various chambers and valves (http://www.andrew.cmu.edu/user/jessicaz/ medical_data/Heart_Valve_new.htm). Computation is performed repeatedly for each point on the mesh using Darcy’s law [], which describes the flow of a fluid through a porous medium and is widely used in blood flow simulations. Figure also illustrates decomposition of this mesh model into subdomains (indicated by different colors) that are relatively easier to model due to their simpler geometry. Domain decomposition is vital for high-performance scientific computing. In addition to providing enhanced parallelism, it also offers a practical framework for solving computational problems over complex, heterogeneous and irregular domains by decomposing them into homogeneous subdomains with regular geometries. The subdomains are each modeled separately using faster, simpler computational algorithms, and the solutions are then carefully merged to ensure consistency at the boundary cases. Such decomposition is also useful when the modeling equations are different across subdomains as in the case of singularities and
D
Domain Decomposition. Fig. Tetrahedral meshing of the heart. The figure on the right shows the mesh partitioned into subdomains indicated with different colors
anomalous regions. Lastly, it is also useful for developing out-of-core (or external memory) solution techniques for large computational problems with billions of unknowns by facilitating the partitioning of the problem domain into smaller chunks, each of which can fit into main memory.
Domain Decomposition for PDEs Scientific computing applications such as weather modeling, heat transfer, and fluid mechanics involve modeling complex physical systems via partial differential equations. Though there exist many different classes of PDEs, the most widely used ones are second-order elliptic equations, for example, the Poisson’s equation, which manifests in many different forms in a number of applications. For example, the Fourier law for thermal conduction, Darcy’s law for flows in porous media and Ohm’s law for electrical networks, all arise as special cases of the Poisson’s equation. Example . For the sake of illustration, we consider a problem that requires solving the Poisson equation over a complex domain Ω with Γ denoting the boundary ∇ u = f in Ω, u = uΓ on Γ.
D
Domain Decomposition
In practice, the domain is discretized using either finite difference or finite element methods, which yields a large sparse symmetric positive definite linear system, Au = f where A is determined by the discretization and u is a vector of unknowns. Domain decomposition for this problem involves partitioning the domain Ω into subdomains {Ω i }si= , which may be overlapping or nonoverlapping depending on the nature of the problem. The partitioning into subdomains is sometimes referred to as the coarse grid with the original discretization being described as the fine grid. Given a discretized computational problem as in the example, there are four main steps that need to be addressed in order to obtain an efficient parallel solution. ●
The first step is to identify the preferred characteristics of subdomains for the given problem. This includes choosing the number of subdomains, the type of partitioning (element/edge/vertex-based), and the level of overlap between the subdomains (with disjoint subdomains being the extreme case). Each of these choices depend on a variety of factors such as size, type, and geometry of the problem domain, the number of parallel processors, and the PDE/linear system being solved. ● The next step is to partition the problem domain into the desired subdomains. This can be achieved by using a variety of geometric and graph partitioning algorithms. For certain applications involving adaptive mesh refinement, one may chose to iteratively refine the partitioning based on the load imbalance. ● Given the subdomains, the next step involves solving the PDE for the entire problem domain. This consists of : – Solving the PDE over each subdomain, that is, ∇ u = f in Ω i , for ≤ i ≤ s, or, equivalently the linear systems Ai ui = fi , for ≤ i ≤ s, where Ai , ui , and fi are restrictions of A, u, and f to Ω i , respectively – Ensuring consistency at the interfaces (or the overlap in case of overlapping regions) of the subdomains, which is referred to as the coarse solve ● Depending on the overlap between the subdomains, there are two classes of solution approaches.
Specifically, Schur’s complement techniques are suitable for subdomains with no overlap and Schwarz alternating techniques are suited for overlapping subdomains. Since these solution techniques iteratively solve the constituent subproblems, these subproblems need not be solved accurately in each step and different choices of approximation over the subproblems lead to variants of the overall algorithms. ● Given a parallel architecture, the final step is to map the subdomains and the interfaces to the individual processors and translate the domaindecomposition-based solution approach to a parallel program that can be implemented efficiently on this architecture. This mapping depends on the relative computational and communication requirements for solving the interfaces. Each of the above steps will be discussed in the following sections.
Nature of Subdomains The effectiveness of any domain decomposition approach critically depends on the relative suitability of the subdomains both with respect to the computational problem (PDE/linear system) being solved and the parallel architecture being employed. Below, we discuss certain key characteristics of subdomains that influence the overall performance as well as some general guidelines proposed in literature for choosing them for a given problem. Number of Subdomains
The choice of the number of subdomains has a significant effect on both the computational time and the quality of the final solution. In general, a large number of subdomains, or alternately a small size for the coarse grid, results in a better, but more expensive solution since a large number of subdomain problems need to be solved and there is much more communication. On the other hand, choosing a small number of subdomains has the opposite effect. Given a fine grid, that is, a discretized problem, it has been observed empirically [] that there exists an optimal choice for the number of subdomains that minimizes the overall computational time. However, theoretical results [] exist only for fairly simple scenarios
Domain Decomposition
where the same solver is used for all the subdomain problems as well as the original problem, and the convergence rate is independent of the coarse grid size. For example, for a uni-processor setting on a structured nd grid with fine grid scale h and a solver with complexity O(nα ), the optimal choice for the coarse grid size is given by α−d α α Hopt = ( ) h α−d . α−d Practical scientific computing problems, however, often involve fairly complex domains as in Fig. and this requires taking into account a number of other considerations. Firstly, it is preferable to partition the domain into subdomains with regular geometry on which faster solvers can be deployed. Secondly, in a parallel setting, an ideal choice is to assign each subdomain to an individual processor to minimize communication costs. Such a choice might, however, lead to load imbalance in case the number of subdomains is less than that of the available processors or there is vast difference in the relative sizes of the subdomains. For a parallel scenario, there exist limited theoretical results. For example, for a structured nd grid where all the subdomain solves are performed in parallel followed by an integration phase performed by one of the solvers, it can be shown that the optimal number of processors is nd/ , but this analysis only considers load distribution and ignores the communication costs. For a real-world application, one needs to empirically determine an appropriate partitioning size that optimizes both the load balance as well as the communication costs. Type of Partitioning
There are three main types of partitioning depending on the constraints imposed on subdomain interfaces. The first type is element-based partitioning, common in case of finite-element discretizations, where each element is solely assigned to a single subdomain. The second type is edge-based partitioning, more suitable for finite-volume methods, where each edge is solely assigned to a single subdomain so as to simplify the computation of fluxes across edges. The last type is vertex-based partitioning, which is the least restrictive and only requires each vertex to be assigned to a unique subdomain. The choice of partitioning type determines how the subdomain interface values are computed and can have a significant effect on the complexity and
D
convergence of the algorithm, for example, Schur complement method can be implemented much more efficiently for edge-based partitionings than the other two types []. For most problems, the appropriate type of partitioning is determined by the nature of the computational problem (PDE) and the discretization itself, but where there is flexibility, it is important to make a judicious choice in order to optimize the computation time for the coarse solve. Level of Overlap
Overlap between the subdomains is another important characteristic that determines the solution approach. In particular, Schur complement–based approaches are applicable only in case of nonoverlapping subdomains whereas Schwarz alternating procedures (SAP) are applicable in case of overlapping subdomains. The appropriate level of overlap is often dependent on the geometry of the problem domain and the PDE to be solved. A decomposition into subdomains with regular geometry (e.g., circles, rectangles) is often preferable irrespective of the level of overlap. In general, SAPbased techniques are easier to implement in addition to providing close to optimal convergence rate and being more robust. However, extra computational effort is required for the regions of overlap and they are not suitable for handling discontinuities.
Partitioning Algorithms For scientific computing tasks, the problem domain is invariably discretized making it natural to use a graph (or in certain cases hypergraph) representation. Decomposing the problem domain into subdomains, therefore, becomes a graph partitioning problem. For parallel computing with homogeneous processors, it is desirable to have a partitioning such that the resulting subdomains require comparable amounts of computational effort, that is, balanced load, and there are minimal communication costs for handling the subdomain interfaces. In other words, the subdomains need to be of similar size with few edges between them. In case of structured and quasi-uniformly refined grids, such a partitioning can be obtained with relative ease. However, for unstructured grids resulting from most realworld applications, it is an NP-complete problem []. A number of algorithms have been proposed in literature to address this problem and [, , ] provides
D
D
Domain Decomposition
a detailed survey of these algorithms. Some of the key approaches are discussed below. Geometric Approaches
Geometric partitioning approaches rely mainly on the physical mesh coordinates to partition the problem domain into subdomains corresponding to regions of space. Such algorithms are preferable for applications such as crash and particle simulations, where geometric proximity between grid points is much more critical than the graph connectivity. Recursive bisection methods form an important class of geometric approaches that involve computing a separator that partitions a region into two subregions in a desired ratio (usually equal). This bisection process is performed recursively on the subregions until the desired number of partitions are obtained. The simplest of these techniques is the Recursive Coordinate Bisection (RCB) [], which uses separators orthogonal to coordinate axes. A more effective technique is Recursive Inertial Bisection (RIB) [] that employs separators orthogonal to principal inertial axes of the geometry. Over the last decades, a number of extensions [] as well as hybrid techniques [] that combine RIB with local search methods such as the Kernighan-Lin’s method, in order to simultaneously minimize the edge cut between the subdomains have been proposed. Space-filling curve (SFC)-based partitioning approaches [] are another class of geometric techniques. In these methods, a space-filling curve maps an n-dimensional space to a single dimension, which provides a linear ordering of all the mesh points. The mesh is then partitioned by dividing the ordered points into contiguous sets of desired sizes, which correspond to the subdomains. The explicit use of geometric information and the incremental nature of the above techniques makes them highly beneficial for applications where geometric locality is critical and there are frequent changes in the geometry requiring dynamic load balancing. However, these techniques have limited applicability for computational problems where one needs to consider graph connectivity and communication costs.
These techniques are widely used in a number of scientific computing applications since they can provide better control of communication costs and do not require an embedding of the problem in a geometric space. There currently exist a large number of such algorithms [], but most of them have been derived from combinations of three basic techniques: (a) Recursive Graph Bisection, (b) Recursive Spectral Bisection, and (c) Greedy approach. Of these, the recursive graph bisection method [] begins by finding a pseudoperipheral vertex, (i.e., one of a pair of vertices on the longest graph geodesic), computing the graph distance to all the other vertices from this vertex. This distance is then used to sort the vertices, which are then split into two groups in the desired ratio (usually equal) such that one group is closer to the peripheral vertex and the other one is away, and this process is repeated recursively till the desired number of partitions are obtained. The second technique, that is, recursive spectral bisection method [] makes use of the Fielder vector, which is defined as the eigenvector corresponding to the second lowest eigenvalue of the Laplacian matrix of the graph. The size of this vector is the same as the number of vertices in the graph and the difference between the Fielder coordinates is closely related to the graph distance between the corresponding vertices. The Fielder vector, thus, provides a linear ordering of the vertices, which can be used to divide the vertices into two groups and recursively deployed as in the case of SFC-based partitioning algorithms. The greedy approach typically uses a breadth first search and uses graph growing heuristics to create partitions []. There are also a number of fast multilevel techniques that are especially suitable for large problem domains. These techniques typically consist of three phases: coarsening, partitioning, and refining. In the first phase, a sequence of increasingly coarse graphs is constructed by collapsing vertices that are tightly connected to each other. The coarsened graph is then partitioned into the desired number of subdomains in the second phase. The graph is then progressively uncoarsened and refined using local search methods such as the Kernighan-Lin method [], which improves the edge cut via a series of vertex-pair exchanges, in the third phase.
Coordinate-Free Approaches
Coordinate-free partitioning approaches include spectral and graph-theory-based techniques that rely only on the connectivity structure of the problem domain.
Dynamic Approaches
When the computational structure is relatively invariant through out a simulation, it is sufficient to consider a
Domain Decomposition
static partitioning of the problem domain. However, for applications where the structure changes dynamically (e.g., in crash simulations with geometric deformations or in case of adaptive mesh refinement methods), it is critical to dynamically repartition the problem domain to ensure load balance. Such a repartitioning needs to account for not only the load distribution and communication costs, but also the costs of data redistribution across the partitionings []. Furthermore, in case of large problems, since it is expensive to load the entire mesh onto a single processor, these partitionings themselves need to be computed in parallel. Currently, there exist a number of dynamic partitioning algorithms such as [, , ] which can viewed as extensions of existing static partitioning algorithms that incorporate an additional objective term based on the data redistribution costs. These techniques often involve a clever remapping of subdomain labels to reduce the redistribution costs. In case of multilevel methods such as LocallyMatched Scratch-Remap (LMSR) [], the coarsening phase is similar to that in the static case, the partitioning (i.e., repartitioning) phase leads to tentative new subdomains, and the final uncoarsening/refinement phase is adapted so as to additionally optimize the data redistribution costs.
D
expressed as ⎡ ⎤ ⎡ int ⎢ B ⎢ D ⎥ ⎢ ⎥ ⎢ u ⎢ ⎥⎢ ⎢ ⎥ ⎢ int ⎢ ⎢ B D ⎥ ⎢ ⎥ ⎢ u ⎢ ⎥⎢ ⎢ ⎥⎢ ⎢ ⎢ ⋱ ⋮ ⎥ ⎢ ⎥⎢ ⋮ ⎢ ⎥⎢ ⎢ ⎥ ⎢ int ⎢ ⎢ Bs D s ⎥ ⎢ ⎥ ⎢ us ⎢ ⎥⎢ ⎢ ⎥⎢ ⎢ E E ⋯ Es C ⎥ ⎢ ubnd ⎣ ⎦⎣
⎤ ⎡ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥=⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎦ ⎣
⎤ fint ⎥ ⎥ ⎥ ⎥ fint ⎥ ⎥ ⎥ ⎥ , ⋮ ⎥ ⎥ ⎥ ⎥ fsint ⎥ ⎥ ⎥ bnd ⎥ ⎥ f ⎦
()
D
where each fiint denotes the vector of unknowns interior to subdomain Ω i and ubnd denotes the vector of interface unknowns. Figure shows the nonzero structure for such a linear system. This can be equivalently expressed in a block form as ⎛B D ⎜ ⎜ ⎝E C
⎞ ⎛ uint ⎟⎜ ⎟⎜ ⎠ ⎝ ubnd
⎞ ⎛ f int ⎟=⎜ ⎟ ⎜ ⎠ ⎝ f bnd
⎞ ⎟, ⎟ ⎠
()
where B, D, E, f int , and uint are aggregations over the corresponding subdomain specific variables. From Eq. , by substituting for uint , we obtain a reduced system (C − EB− D)ubnd = f bnd − EB− f int .
PDE Solution Techniques There are two main classes of approaches for solving PDEs over a partitioned problem domain [] depending on the overlap, namely, (a) Schur complement– based approaches and (b) Schwarz alternating procedures, which we discuss below.
Schur Complement–Based Approaches
This class of approaches is based on the notion of a special matrix called the Schur complement, which enables one to solve for the subdomain interface values independent of the interior unknowns. The computation of the unknowns in the interior of each of the subdomains is then performed in parallel independently. Consider a computational problem corresponding to a second-order elliptic PDE as in Example . As mentioned earlier, a discretization leads to the linear system Au = f , u ∈ Ω. In case of nonoverlapping subdomains, that is, Ω i ⋂ Ω j = /, and edge-based partitioning on a finite difference mesh, this linear system can be
Domain Decomposition. Fig. Nonzero structure of the resulting linear system
D
Domain Decomposition
Γ3,0
Algorithm Schur complement method . Form the reduced system based on Schur complement matrix . Solve the reduced system for the interface unknowns . Solve the decoupled systems for interior unknowns by back-substituting the interface unknowns
The matrix S = (C − EB− D) is called the Schur complement matrix. This matrix can be computed directly from the original coefficient matrix A given the partitioning and can be used to compute the interface values ubnd . Back-substitution of ubnd is then used to solve smaller int decoupled linear systems Bi uint − Di ubnd . i = fi In the case of vertex-based and element-based partitionings, the coefficient matrix A exhibits a more complex block structure unlike in Eq. . However, even for these cases, one can construct a reduced system based on a Schur complement matrix assembled from local (subdomain specific) Schur complements and the interface coupling matrices, which can be used to solve for the interface unknowns. Algorithm describes the key steps for this class of techniques irrespective of the partitioning type. Depending on the choice of the methods (e.g., direct vs. iterative methods, exact vs. approximate solve) used for forming the Schur complement and solving the reduced systems, there are multiple possible variants. Schwarz Alternating Procedures
Schwarz alternating procedures refer to techniques that involve alternating between overlapping subdomains and in each case, performing the corresponding subdomain solve assuming boundary conditions based on the most recent solution. Consider the discretized elliptic PDE problem in Example for the case where each subdomain Ω i overlaps with its neighboring subdomains (as in Fig. ). Let Γij denote the boundary of Ω i that is included in Ω j . The original solution method proposed by Schwarz [] in was a sequential method where each step consists of solving the restriction of Au = f over the subdomain Ω i assuming the boundary conditions over Γij , ∀j. Since the approach is iterative and sweeps through each subdomain multiple times, it suffices to perform
Ω3 Γ1,3
Γ3,1 Γ2,1
Γ1,0
Γ1,2
Ω2
Γ2,0
Ω1
Domain Decomposition. Fig. An L-shaped domain showing overlapping subdomains
an approximate subdomain solve or in the extreme case, one iteration of the subdomain solve. Thus, each step can be viewed as an additive update of u. This update step can be concretely specified using the notion of a restriction operator. Let Ri ∈ {, }∣Ω i ∣×∣Ω∣ be a binary matrix that denotes the restriction of Ω to Ω i , that is, the jth component of Ri is iff the jth unknown belongs to Ω i . Given {Ri }si= , one can define the restriction of A to Ω i as the matrix Ai = Ri ARTi and the update is then given by u = u + RTi A− i Ri ( f − Au). Since the updates over each subdomain are applied sequentially, it is equivalent to a multiplicative composition and this procedure is referred to as Multiplicative Schwarz Alternation. To make this approach amenable to parallelization, an alternate procedure Additive Schwarz Alternation [] was proposed, which involves using the same residual ( f − Au) to compute the incremental updates over each subdomain in a single sweep and the incremental updates are then aggregated to obtain the new solution. Algorithm shows the main steps for the basic additive Schwarz alternation and each of the δ i computations in a single iteration are independent of each other and can be performed in parallel. Algorithm corresponds to the most basic version of additive Schwarz. Real applications, however
Domain Decomposition
Algorithm Additive Schwarz method Choose an initial guess for u repeat for i = ⋯s do Compute δ i = RTi A− i R i ( f − Au) unew = u + ∑si= δ i until convergence
employ variants of this approach with similar independent additive updates, but are based on more complex preconditioners [].
Mapping to Parallel Processors A natural way to map a domain decomposition algorithm to a parallel architecture is to assign each subdomain to a separate processor. In this case, the subdomain solves can be performed independently in parallel, but the coarse solve (handling of interface values) is nontrivial since the relevant interface data is scattered among all the processors. For example, in Schur complement approaches, solving the reduced system based on the Schur complement, in the general case, requires access to data from all the subdomains. There are three common ways to address the above problem. The first is to keep the data in place and perform the solve in parallel with the necessary communication. Such an approach would require an intelligent grouping of the interface unknowns with minimal dependencies across groups, which can then be assigned to different subdomains so as to minimize the communication cost. The second option is to collect all the data on a single processor, perform the coarse solve, and broadcast the output while the third approach is to replicate the relevant data on all the processors and perform the coarse solve in parallel. There have been empirical studies [] that evaluated the relative benefits of these approaches, but the appropriate choice typically depends heavily on the characteristics of the specific problem and the parallel architecture.
Advanced Methods More recent work on domain decomposition has focused on efficient techniques for specific topics. These include advanced methods for graph partitioning and for solving PDEs.
D
In graph partitioning, researchers have developed hyper-graph algorithms [, ] that can be used for graphs with edges that involve more than two nodes. Such graphs arise naturally in many applications such as higher-order finite elements, and can also be constructed from regular graphs. Empirical evidence points to improvement in the efficiency of partitioning algorithms as well as in the quality of partitions. It has also been recognized for a while that large graph problems that require domain decomposition techniques often arise in a parallel setting, and as a result, work has focused on developing parallel partitioning methods. These methods are, in general, straightforward parallelization schemes, with additional care taken to ensure quality of the partitions. Another important area of work involves adaptive partitioning and refining [, ] for applications where the graph changes frequently during the simulation. A key component of these algorithms is the ability to repartition quickly and efficiently starting from an initial partition. The new partitioning must ensure good load balance without increasing communication requirement or introducing large overhead. The domain decomposition step has implications on the numerical procedure used to solve the PDE. In particular, the rate of convergence of iterative solvers can be improved significantly by considering domain decomposition as a “preconditioning” step that reduces the number of iterations. In this regard, there has been some work on improving the Schur-complement approach by using Schwarz-type methods for the reduced linear system. This can be useful when the Schur complement is complex and large enough to require a parallel solver. Another approach that is motivated by solving PDEs in parallel is the Finite Element Tearing and Integration (FETI) [, ] method in which splitting of the domain into subdomains is accompanied by modification of the linear operator to achieve faster convergence of the iterative solver. After splitting the domain, one needs to enforce continuity across subdomain boundaries using Lagrange multipliers. The solution process determines these Lagrange multipliers via an iterative method that requires repeated subdomain solves that can be done in parallel.
Application Areas Domain decomposition has been used in a wide variety of disciplines. These include aerospace, nuclear,
D
D
Domain Decomposition
mechanics, electrical, materials, petroleum, and others. The benefits of the method are most apparent when used for applications that require large systems to be solved. In such cases, the solver phase can consume up to % or more of the total running time of the application. The domain decomposition concept can be applied in a fairly straightforward manner once the subdomains are identified. However, it may be important to finetune the parameters for each application in order to get the best performance.
Future Directions The advent of multicore processors and the availability of large multiprocessor machines pose significant challenges to researchers and developers in the area of domain decomposition. Challenges that arise from large-scale compute capability involve the ability to use a large number of cores concurrently and efficiently. It will be much harder to achieve good partitioning and load balance on such platforms. The ability to perform asynchronous computation and to adapt to changing computational structure is equally important. In addition, it is imperative to build fault tolerance into the algorithmic structure to protect against inevitable architectural instabilities. One must also consider relaxing the numerical requirements on the computations in favor of obtaining an approximate but acceptable solution quickly. Another area of interest is the automatic tuning of parameters required by the algorithms to ensure good performance. It is also important to understand that domain decomposition is often used as a low-level computational tool in applications where the main goal is to optimize within a design space or to conduct sensitivity studies. A tighter integration of the domain decomposition step with other computational modules is necessary to realize the full benefit of this technique for an application.
Related Entries Graph Partitioning
Bibliography . Berger MJ, Bokhari SH () A partitioning strategy for nonuniform problems on multiprocessors. IEEE Trans Comput ():– . Çatalyürek UV, Boman EG, Devine KD, Bozdag D, Heaphy RT, Riesen LA () A repartitioning hypergraph model for dynamic load balancing. J Parallel Distrib Comput ():–
. Chan TF, Shao JP () Parallel complexity of domain decomposition methods and optimal coarse grid size. Parallel Comput ():– . Devine KD, Boman EG, Heaphy RT, Hendrickson BA, Teresco JD, Faik J, Flaherty JE, Gervasio LG () New challenges in dynamic load balancing. Appl Numer Math (–):– . Devine KD, Boman EG, Heaphy RT, Bisseling RH, Çatalyürek UV () Parallel hypergraph partitioning for scientific computing. In: Proceedings of the th parallel and distributed processing symposium , Rhodes Island, Greece, pp . Dryja M, Widlund OB () Domain decomposition algorithms with small overlap. SIAM J Sci Comput ():– . Farhat C, Lesoinne M () Automatic partitioning of unstructured meshes for the parallel solution of problems in computational mechanics. Int J Numer Methods Eng ():– . Farhat C () A simple and efficient automatic fem domain decomposer. Comput Struct ():– . Farhat C, Pierson K, Lesoinne M () The second generation feti methods and their application to the parallel solution of large-scale linear and geometrically non-linear structural analysis problems. Comput Methods Appl Mech Eng (–): – . Farhat C, Roux F-X () A method of finite element tearing and interconnecting and its parallel solution algorithm. Int J Numer Meth Eng :– . Fjällström P-O () Algorithms for graph partitioning: a survey, vol , part . Linköping University Electronic Press, Sweden . Gropp WD, Keyes DE () Domain decomposition on parallel computers. IMPACT Comput Sci Eng ():– . Gropp WD, Smith BF () Experiences with domain decomposition in three dimensions: overlapping schwarz methods. In: Proceedings of the th international symposium on domain decomposition, AMS, Providence, , pp – . Hendrickson B, Kolda TG () Graph partitioning models for parallel computing. Parallel Comput ():– . Karypis G, Kumar V () Parallel multilevel k-way partitioning scheme for irregular graphs. In: Proceedings of the th ACM/IEEE conference on supercomputing (CDROM) (Super computing ’), Article , Pittsburgh, PA, USA . Karypis G, Kumar V () Multilevel -way hypergraph partitioning. In: Proceedings of the th design automation conference, New Orleans, LA, USA, pp – . Kernighan BW, Lin S () An efficient heuristic procedure for partitioning graphs. Bell Syst Tech J ():– . Leland R, Hendrickson B () An empirical study of static load balancing. In: Proceedings of scalable high performance computing conference, pp – . Neuman SP () Theoretical derivation of Darcy’s law. Acta Mech ():– . Oliker L, Biswas R () Plum: parallel load balancing for adaptive unstructured meshes. J Parallel Distrib Comput ():– . Pothen A, Simon HD, Liou K () Partitioning sparse matrices with eigenvectors of graphs. SIAM J Matrix Anal Appl (): – . Saad Y () Iterative methods for sparse linear systems, nd edn. SIAM, Philadelphia, PA
Dynamic Logical Partitioning for POWER Systems
. Schamberger S, Wierum J () Partitioning finite element meshes using space-filling curves. Future Gen Comp Syst ():– . Schloegel K, Karypis G, Kumar V () Wavefront diffusion and LMSR: Algorithms for dynamic repartitioning of adaptive meshes. IEEE Transactions on Parallel and Distributed Systems ():– . Schwarz HA () Uber einen Grenzübergang durch alternierendes Verfahren. Vierteljahrsschrift der Naturforschenden Gesellschaft in Zürich, :– . Simon HD () Partitioning of unstructured problems for parallel processing. Comput Syst Eng (–):– . Sohn A, Simon H () Jove: a dynamic load balancing framework for adaptive computations on an sp- distributed multiprocessor. Technical report, CIS, New Jersey Institute of Technology, NJ . Sun K, Zhou Q, Mohanram K, Sorensen DC () Parallel domain decomposition for simulation of large-scale power grids. In: ICCAD, pp – . Zhang Y, Bajaj C () Finite element meshing for cardiac analysis. Technical Report –, ICES, University of Texas at Austin, TX
DPJ Deterministic Parallel Java
DR Dynamic Logical Partitioning for POWER Systems
Dynamic Logical Partitioning for POWER Systems Joefon Jann T. J. Watson Research Center, IBM Corp., Yorktown Heights, NY, USA
Synonyms DLPAR; DR; Dynamic LPAR; Dynamic reconfiguration
Definition Dynamic Logical Partitioning (DLPAR) is the nondisruptive reconfiguration of the real or virtual resources made available to the operating system (OS) running in a logical partition (LPAR). Nondisruptive means that no OS reboot is needed.
D
Discussion Introduction A Logical PARtition (LPAR) is a subset of hardware resources of a computer upon which an operating system instance can run. One can define one or more LPARs in a physical server, and an OS instance running in an LPAR has no access and visibility to the other OS instances running in other LPARs of the same server. In October , IBM introduced the DLPAR technology on POWER-based servers with the general availability of AIX .. This DLPAR technology enables resources such as memory, processors, disks, network adaptors, and CD/DVD/tape drives to be moved nondisruptively (i.e., without requiring a reboot of the source and target operating systems) between LPARs in the same physical server. In simple terms, one can perform the following basic nondisruptive dynamic operations with DLPAR: ● ● ●
Remove a unit of resource from an LPAR. Add a unit of resource to an LPAR. Move a unit of resource from one LPAR to another in the same physical server.
For POWER systems, the granularity/unit of resource for a DLPAR operation was CPU for processors, Logical Memory Block (LMB) of size MB for memory, and IO-slot for I/O (disks, network, and media) adapters. Those units of processor, memory, and I/Oslots that are not assigned to any LPAR reside in a “free pool” of the physical server, and can be allocated to a new or existing LPARs via DLPAR operations later on. Similarly, when a unit of resource is DLPAR-removed from an LPAR, it goes into the free pool. The fundamentals of DLPAR are described in []. AIX has supported DLPAR removal of memory since /; however, DLPAR removal of memory was not available for Linux on POWER until , with the availability of SUSE SLES . On POWER and follow-on POWER systems, the minimum size of an LMB (unit of DLPAR removal or add) can be as small as MB. This minimum size is a function of the total amount of physical memory in the physical server. Since the availability of POWER, all POWER servers are partitioned into one or more LPARs. These LPARs are defined via the POWER HYPervisor firmware (PHYP). An LPAR can be a dedicatedprocessor LPAR or a Shared-Processor LPAR (SPLPAR). A dedicated-processor LPAR is an LPAR which contains
D
D
Dynamic Logical Partitioning for POWER Systems
an integral number of CPUs. SPLPAR was introduced with AIX . on POWER systems in /. One can define a pool of CPUs to be shared by a set of LPARs, which will be called shared-processor LPARs (SPLPARs). The PHYP controls the time-slicing and allocation of CPU resources across the SPLPARs sharing the pool. An SPLPAR has the following attributes: (a) Minimum, assigned, and maximum number of Virtual Processors (VPs). The number of VPs is the degree of processorconcurrency made available to the OS instance. (b) Minimum, assigned, and maximum number of processor-units of Capacity. Capacity is the amount of cumulative real-CPUequivalent resources allocated to the OS instance in a reasonable interval of time (e.g., ms). (c) Capped or uncapped mode. A capped SPLPAR cannot use excess CPU resources in the pool beyond its maximum capacity. (d) If uncapped, an SPLPAR has an uncapped-weight value (currently, an integer between and ), which is used for prioritizing the SPLPARs in a pool. The DLPAR technology was then enhanced to be able to dynamically move units of a SPLPAR’s CPU-resources nondisruptively from one SPLPAR to another in the same SPLPAR pool. More precisely, DLPAR technology can nondisruptively and dynamically ●
Remove or add an integral number of Virtual Processors from or to an SPLPAR, or move such from one SPLPAR to another ● Remove or add Capacity (in processor-units) from or to an SPLPAR, or move capacity from one SPLPAR to another ● Change the processor Mode of an SPLPAR to Capped or Uncapped ● Change the Weight of an Uncapped SPLPAR (between a value of and )
DLPAR Technology DLPAR Removal of a CPU DLPAR enables the dynamic removal of a CPU from a running OS instance in one LPAR, without requiring a reboot of the OS instance.
At a very high level, the following sequence of operations initiated from the OS is used to accomplish a DLPAR removal of a CPU: . Notify all applications and kernel extensions that have registered to be notified of CPU DLPAR events, so that they may remove dependencies on the CPU to be removed. This typically involves unbinding their threads that are bound to the CPU being removed. . Migrate threads that are bound to the CPU being removed to another CPU in the same LPAR. . Reroute existing and future hardware-interrupts that are directed to the CPU being removed. This involves changing the interrupt controller data structures. . Migrate the timers and threads from the CPU being removed to another CPU in the same LPAR. . Notify the hypervisor/firmware to complete the removal task.
DLPAR Addition of a CPU Dynamic addition of a CPU involves the following tasks initiated from the OS: . Create a process (waitproc) to run an idle loop on the incoming CPU, before it starts to do real work. . Set up various hardware registers (e.g., GPR for the kernel stack, GPR for the kernel Table Of Contents) of the incoming CPU. . Allocate and/or initialize the processor-specific kernel data structures (e.g., runqueue, per-processor data area, interrupt stack) for the incoming CPU. . Add support for the incoming CPU to the interrupt subsystem. . Notify the DLPAR-registered applications and kernel extensions that a new CPU has been added. Dynamic addition and removal of processors to and from an LPAR allow it to adapt to varying workloads. Figure shows that the throughput (in requests-persecond) of the WebSphere Trade benchmark scales almost linearly with the number of CPUs in a dedicatedprocessor-LPAR, as CPUs were removed and added to the LPAR with DLPAR operations [].
Dynamic Logical Partitioning for POWER Systems
900 8
7
6
5
4
<---Number of CPUs--> 3 2 1 2 3
4
5
6
D
8
7
800 700
D
Request per second
600 500 400 300 200 100 0
12
24
36
48
60 72 84 96 108 120 132 Elapsed time in hundreds of seconds
144
156
168
180
Dynamic Logical Partitioning for POWER Systems. Fig. WebSphere Trade benchmark: throughput (in requests-per-second) as a function of the number of CPUs (the upper x-axis labels), and as a function of elapsed time (the lower x-axis labels)
DLPAR Addition of a Memory Block (LMB) When a DLPAR memory-add request arrives at the OS, the latter has to perform two tasks: . Allocate and initialize software page descriptors that will hold metadata for the incoming memory. . Distribute the incoming memory among the framesets of a mempool (memopools are described two paragraphs below). The challenges encountered in implementing these two tasks in AIX and how they were resolved are described as follows. Prior to implementing DLPAR, software page descriptors could be accessed in translation-off mode (where virtual address is used as real address) while trying to reload a page mapping into the hardware page table. If page descriptors are allowed to be accessed in translation-off mode, then the memory allocated for the incoming new descriptors has to be physically contiguous to the memory for the existing descriptors; which implies that memory has to be reserved at boottime for enough descriptors for the maximal amount
of memory that the OS instance can grow into. This reservation can incur much memory wastage and performance degradation, particularly if not utilized. We avoided this wastage by changing the AIX kernel so that software page descriptor data structures are always accessed in translation-on mode. The challenge of distributing the incoming memory across different page replacement daemons, so that each daemon handles a roughly equal load was resolved as follows: In AIX, memory is hierarchically represented by the data structures vmpool, mempool, and frameset. A vmpool represents an affinity domain of memory. A vmpool is divided into multiple mempools, and each mempool is managed by a single page replacement LRU (least recently used) daemon. Each mempool is further subdivided into one or more framesets that contain the free-frame lists, so as to improve the scalability of free-frame allocators. When new memory is added to AIX, the vmpool that it should belong to is defined by the physical location of the memory chip. Within that vmpool, we wanted to distribute the
D
Dynamic Logical Partitioning for POWER Systems
memory across all the available mempools to balance the load on page replacement daemons. However, the kernel assumed that a mempool consisted of physically contiguous memory. Thus, to be able to break up the new memory (LMB) into several parts and distribute them across different mempools, the kernel was modified to allow mempools to be made up of discontiguous sections of memory.
DLPAR Removal of a Memory Block (LMB) This is by far the hardest-to-implement of all the DLPAR operations, particularly when there are ongoing DMA operations in the OS instance. For the purpose of DLPAR memory removal, we classify the memory pageframes in AIX into five categories: unused, page-able, pinned, DMA-mapped, and translation-off memory. The approach taken to remove a page-frame (, bytes) in each of these categories is as follows: . A page-frame containing a free/unused page is simply removed from its free-list. . A page-frame containing a page-able page can be made to either page out its contents to disk or to migrate its contents to a different free page-frame. . A page-frame containing a pinned page will have its contents migrated to a different page-frame; also the page-fault reload handler had to be made to spin during the migration. . A page-frame containing a DMA-mapped page cannot be removed nor have its contents migrated until all the accesses to the page are blocked. Here, the term “DMA-mapped page-frame” is used generically to mean a page-frame whose physical address is subject to read or write by an external (to kernel) entity such as a DMA engine. The contents of a DMA-mapped page-frame are migrated to a different page-frame with a new hypervisor call (h_migrate_dma) that will selectively suspend the bus traffic while it is modifying the TCEs (translation control entries) in system memory used by the bus unit controller for DMA accesses. Other page-frames whose physical addresses are exposed to external entities are handled by invoking preregistered DLPAR callback routines and then waiting for completion of the removal of their dependencies on the page. . A page-frame containing translation-off pages will not be removed by DLPAR in AIX .. Fortunately,
there is just a small amount of these page-frames, and they are usually collocated in low memory. The design and implementation of DLPAR memory removal has manifested itself as three modular functions, such that one can mix and match these functions in several possible ways, adapting to the state of the system at the time of memory removal, thus achieving the desired end result with the most optimal path. These three modular functions perform, respectively, the following three tasks on the memory (LMB) being removed: (a) Remove its free and clean pages from the regular use of VMM (Virtual Memory Manager) (b) Page out its page-able dirty pages (c) Migrate the contents of each remaining page-frame in the LMB to a free page-frame outside the LMB Memory removal can be implemented with any one of the sequences: abc, ac, bc, or just c. For example, the decision to either invoke page-out (task b) or to migrate (task c) all the pages depends on the load in the LPAR at that particular time. If the LMB being removed contains a lot of dirty pages that belong to highly active threads, then it does not make sense to invoke task b, because these pages will be paged back in almost immediately, hence negatively impacting the efficiency of the system. As a second example, if there are not enough free frames in other LMBs to migrate the pages to, then the memory removal procedure can invoke task b before invoking task c, so that there will be far fewer pages left that need to be migrated in task c.
DLPAR of I/O Slots The methods to dynamically configure or unconfigure a device have been introduced as early as AIX version . The changes required in the kernel design for the DLPAR of I/O slots were not in the same scale as those for the DLPAR of processors, and particularly as those for the DLPAR of memory. The reason is that the onus of configuring or unconfiguring a device lies with the device driver software, which operates in the kernel extension environment. The kernel just acts as a provider of serialization mechanisms for devices accessing common resources, and as an intermediary between the applications and the device drivers.
Dynamic Logical Partitioning for POWER Systems
Values and Uses of DLPAR DLPAR is the foundation of many advanced technologies for ensuring scalability, reliability, and full utilization of the resources in POWER Systems servers. It opens up a whole set of possibilities for great flexibility in dealing with dynamically changing workload demands, server consolidations of workloads from different time zones, and high availability for applications.
Some Obvious Values Enabled by DLPAR ●
●
●
●
●
Ability to dynamically move processors and memory from a test LPAR to a production LPAR in periods of peak production workload demand, then move them back as demand decreases. Ability to programmatically and dynamically move processors and memory from less loaded LPARs to busy LPARs, whenever the need arises []. This is particularly useful for economically providing resources to LPARs that service workloads in different time zones or even different continents. Ability to dynamically move an infrequently used I/O device between LPARs, such as a CD/DVD/tape drives for installations, applying updates, and for backups. Occasionally, one may also want to move Ethernet adaptors and disk drives dynamically from LPAR to another. Ability to dynamically release a set of processor, memory, and I/O resources into the “free pool,” so that a new LPAR can be created using the resources in the free pool. Providing high availability for the applications in an LPAR. For example, you can configure a set of minimal LPARs on a single system to act as “failover” backup LPARs to a set of primary LPARs, and also keep some set of resources free. If one of the associated primaries fails, then its backup LPAR can pick up the workload, and resources can be dynamically added to the backup LPAR as needed.
Some Non-Obvious Values Enabled by DLPAR ●
Dynamic CPU Guard, which is the automatic and graceful de-configuration of a processor which has intermittent errors. ● Dynamic CPU Sparing, which is the Dynamic CPU Guard feature enhanced with automatic enablement of a spare CPU to replace a defective CPU.
D
●
CUoD (Capacity Upgrade on Demand), which allows a customer to activate preinstalled but inactive and unpaid for processors as resource needs arise, as soon as IBM is notified and enables the corresponding license keys. Many enterprises with seasonally varying business activities extensively utilize this feature in their IT infrastructure. ● Hot repair of PCI devices, and possibly future hot repair of MCM (Multi-Chip Modules). ● The failover of an LPAR to another LPAR in another physical server, used with the IBM HACMP LPP (Licensed Program Product). This is detailed in the -page IBM Redpaper titled “HACMP v., Dynamic LPAR, and Virtualization” []. HACMP is acronym for High Availability Cluster Multiprocessing. It is IBM’s solution for high-availability clusters on the AIX and Linux POWER platforms.
DLPAR-Safe, DLPAR-Aware, and DLPAR-Friendly Programs/Applications A DLPAR-safe program is one that does not fail as a result of DLPAR operations. Its performance may suffer when resources are taken away, and performance may not scale up when new resources are added. However, by default, most applications are DLPAR-safe, and only a few applications are expected to be impacted by DLPAR. A DLPAR-aware program is one that has code that is designed to adjust its use of system resources commensurate with the actual resources of the LPAR, which is expected to vary over time. This may be accomplished in two ways. The first way is by regularly polling the system to discover changes in its LPAR resources. The second way is by registering a set of DLPAR code that is designed to be executed in the context of a DLPAR operation. At a minimum, DLPAR-aware programs are designed to avoid introducing conditions that may cause DLPAR operations to fail. They are not necessarily concerned with the impact that DLPAR may have on their performance. A DLPAR-friendly program is a DLPAR-aware application or middleware that automatically tunes itself in response to the changing LPAR resources. AIX provides a set of DLPAR scripts and APIs for applications to dynamically resize. Notable vendor applications
D
D
Dynamic LPAR
that are DLPAR-friendly include IBM DB [], Lotus Domino [], Oracle [], etc. More information about these scripts and APIs can be found in chapter (Dynamic Logical Partitioning) of the online IBM manual “AIX Version . General Programming Concepts” []. To maintain expected levels of application performance when memory is removed, buffers may need to be drained and resized. Similarly when memory is added, buffers may need to be dynamically increased to gain performance. Similar treatment needs to be applied to threads, whose numbers, at least in theory, need to be dynamically adjusted to the changes in the number of online CPUs. However, thread-based adjustments are not necessarily limited to CPU-based decisions; for example, the best way to reduce memory consumption in Java programs may be to reduce the number of threads, since this should reduce the number of active objects that need to be preserved by the Java Virtual Machine’s garbage collector.
Future Directions Future uses of DLPAR are already being explored; among these are: ●
Power/energy conservation for the server by dynamically DLPAR-remove of unused CPUs and memory ● Dynamic hot repair of server components, even MCM (Multi-Chip Modules)
Related Entries
. Shah P () DB and dynamic logical partitioning. http://www. ibm.com/developerworks/eserver/articles/db_dlpar.html . Bassemir R, Faurot G () Lotus Domino and AIX DLPAR. http://www.ibm.com/developerworks/aix/library/ au-DominoandDLPARver.html . Shanmugam R () Oracle database and Oracle RAC gR on IBM AIX. http://www.ibm.com/servers/enable/site/peducation/ wp/a/a.pdf . IBM manual “AIX Version .: General Programming Concepts: Writing and Debugging Programs” SC--, / pp – http://publib.boulder.ibm.com/infocenter/aix/vr/ topic/com.ibm.aix.genprogc/doc/genprogc/genprogc.pdf . Quintero D, Bodily S, Pothier P, Lascu O () HACMP v., dynamic LPAR, and virtualization. IBM Redpaper, http://www. redbooks.ibm.com/redpapers/pdfs/redp.pdf . Bodily S, Killeen R, Rosca L () PowerHA for AIX cookbook. IBM Redbook, http://www.redbooks.ibm.com/redbooks/ pdfs/sg.pdf . Matsubara K, Guérin N, Reimbold S, Niijima T () The complete partitioning guide for IBM eServer pSeries servers. IBM Redbook, SG--, http://portal.acm.org/citation. cfm?id= ISBN: . Irving N, Jenner M, Kortesniemi A () Partitioning implementation for IBM eServer p servers. IBM Redbook, SG-, http://www.redbooks.ibm.com/redbooks/pdfs/sg.pdf . Hales C, Milsted C, Stadler O, Vågmo M () PowerVM virtualization on IBM system p: introduction and configuration, th edn. IBM Redbook, http://www.redbooks.ibm.com/ redbooks/pdfs/sg.pdf . Dimmer I, Haug V, Huché T, Singh AK, Vågmo M, Venkataraman AK () Ch : DLPAR. In: PowerVM virtualization managing & monitoring. IBM RedBook, http://www.redbooks.ibm. com/redbooks/pdfs/sg.pdf . Jann J, Burugula RS, Dubey N IBM DLPAR tool set for pSeries for P & P systems. http://www.alphaworks.ibm.com/tech/dlpar (Contact [email protected]) . Wikipedia entry on DLPAR: http://en.wikipedia.org/wiki/ DLPAR
IBM Power Architecture Multi-Threaded Processors
Bibliography . Jann J, Browning L, Burugula RS () Dynamic reconfiguration: basic building blocks for autonomic computing in IBM pSeries servers. IBM Syst J (Spec Issue Auton Comput) () . Jann J, Dubey N, Pattnaik P, Burugula RS () Dynamic reconfiguration of CPU and WebSphere on IBM pSeries servers. Softw Pract Exp J (SPE Issue) ():– . Lynch J Dynamic LPAR – the way to the future. http://www. ibmsystemsmag.com/aix/junejuly/administrator/p.aspx
Dynamic LPAR Dynamic Logical Partitioning for POWER Systems
Dynamic Reconfiguration Dynamic Logical Partitioning for POWER Systems
E Earth Simulator Ken’ichi Itakura Japan Agency for Marine-Earth Science and Technology (JAMSTEC), Yokohama, Japan
Synonyms ES
Definition The Earth Simulator is a highly parallel vector supercomputer system consisting of processor nodes and an interconnection network.
Discussion The Earth Simulator, which was developed, as a national project, by three governmental agencies, the National Space Development Agency of Japan (NASDA), the Japan Atomic Energy Research Institute (JAERI), and Japan Marine Science and Technology Center (JAMSTEC), was led by Dr. Hajime Miyoshi. The ES is housed in the Earth Simulator Building (approximately m × m × m). The fabrication and installation of the ES at the Earth Simulator Center of JAMSTEC by NEC was completed at the end of February in . It was first on the Top list for two and a half years starting in June (Fig. ).
Hardware The ES is a highly parallel vector supercomputer system of the distributed-memory type, and consisted of processor nodes (PNs) connected by × single-stage crossbar switches. Each PN is a system with a shared memory, consisting of eight vector-type arithmetic processors (APs), a -GB main memory system (MS), a remote access control unit (RCU), and an I/O processor. The peak performance of each AP is Gflops.
The ES as a whole thus consists of , APs with TB of main memory and the theoretical performance of Tflops. Each AP consists of a four-way super-scalar unit (SU), a vector unit (VU), and main memory access control unit on a single LSI chip. The AP operates at a clock frequency of MHz with some circuits operating at GHz. Each SU is a super-scalar processor with KB instruction caches, KB data caches, and general-purpose scalar registers. Branch prediction, data prefetching, and out-of-order instruction execution are all employed. Each VU has vector registers, each of which has vector elements, along with eight sets of six different types of vector pipelines: addition/shifting, multiplication, division, logical operations, masking, and load/store. The same type of vector pipelines works together by a single vector instruction and pipelines of different types can operate concurrently. The VU and SU support the IEEE floatingpoint data format. This LSI chip technology is nm CMOS and eight layers copper interconnection. The die size is . mm × . mm. It has million transistors and , pins. The maximum power consumption is W. The overall MS is divided into , banks and the sequence of bank numbers corresponds to increasing addresses of locations in memory. Therefore, the peak throughput is obtained by accessing contiguous data which are assigned to locations in increasing order of memory address. The RCU is directly connected to the crossbar switches and controls internode data communications at . GB/s bidirectional transfer rate for both sending and receiving data. Thus the total bandwidth of internode network is about TB/s. Several data-transfer modes, including access to three-dimensional (D) sub-arrays and indirect access modes, are realized in hardware. In an operation that involves access to the data of a sub-array, the data is moved from one PN to
David Padua (ed.), Encyclopedia of Parallel Computing, DOI ./----, © Springer Science+Business Media, LLC
E
Earth Simulator
Disks Interconnectin network (IN) cabinets (65)
Processor node (PN) cabinets (320)
Air conditioning system
65 m 71 yd
50 m 55 yd
Double floor for cables
Power supply system
Earth Simulator. Fig. Bird’s-eye view of the earth simulator system
another in a single hardware operation, and relatively little time is consumed for this processing. The Interconnection Network (IN) cabinet has two switches of ×. The signal from PN changes from Serial to Parallel and is inputted into a switching circuit. The signal which came out from the switching circuit is performed Parallel/Serial conversion. Then, it is sent to PN cabinet. The number of Cables which connects PN cabinet and IN cabinet is × = , , and the total extension is , km.
MDPS In October , Mass Data Processing System (MDPS) was installed as a massively data storage system which renews the tape library system. It consists of four file service processors (FSPs), TB hard disk drives, and a currently used . PB cartridge tapes library (CTL). MDPS was adopted aiming to improve data transfer throughput and accessibility. . The transfer speed of saving and extracting the data between the ES and the storage became two to fivetimes faster. This improvement can be realized by expansion of transfer cable ability and replacement of a tape archive with disks.
. User view is a very large disk and usual UNIX commands and tools can access the MDPS data on the login server. Data I/O procedure becomes easy. . MDPS enables users to access the results computed by ES remotely, because it can transfer the data to a dedicated server out of ES LAN (ES-Network FTP Server).
Operating System The operating system running on ES is an enhanced version of NEC’s UNIX-based OS called “SUPER-UX” that is developed for NEC’s SX Series supercomputers. To support ultra-scale scientific computations, SUPER-UX was enhanced mainly in the following two points: . Extending scalability . Providing special features for ES Extending scalability up to the whole system ( PNs) is the major requirement for the OS of ES. All functions of the OS, such as process management, memory management, file management, etc., are fully optimized to fulfill the requirement. For example, any of the OS tasks costing order n, such as scattering data in sequence over all PNs, is replaced with the equivalent one costing order log n, such as binary-tree copy, if possible, where n is the number of PNs.
Earth Simulator
E
Crossbar network
CCS #0
CCS #1
PN #639
PN #625
PN #624
PN #31
PN #17
PN #16
PN #15
Back-end clusters
PN #1
PN #0
TSS cluster
CCS #39
Earth Simulator. Fig. Super cluster system of ES: a hierarchical management system is introduced to control the ES. Every nodes are collected as a cluster system and therefore, there are sets of cluster in total. A set of cluster is called an “S-cluster” which is dedicated for interactive processing and small-scale batch jobs. A job within one node can be processed on the S-cluster. The other sets of cluster is called “L-cluster” which are for medium-scale and large-scale batch jobs. Parallel processing jobs on several nodes are executed on some sets of cluster. Each cluster has a cluster control station (CCS) which monitors the state of the nodes and controls electricity of the nodes belonged to the cluster. A super cluster control station (SCCS) plays an important role of integration and coordination of all the CCS operations. PN#0
PN#1
PN#2
PN#n
AP
AP
AP
AP
FAL
FAL
FAL
FAL
File server
File server
File server
Request File server
Read/write
Parallel file
Earth Simulator. Fig. Parallel file system (PFS): A parallel file, i.e., file on PFS is striped and stored cyclically in the specified blocking size into the disk of each PN. When a program accesses to the file, the File Access Library (FAL) sends a request for I/O via IN to the File Server on the node that owns the data to be accessed
On the other hand, the OS provides some special features which aim for efficient use or administration of such a large system. The features include internode high-speed communication via IN, global address space among PNs, super cluster system (Fig. ), batch job environment, etc.
Parallel File System If a large parallel job running on PNs reads from/writes to one disk installed in a PN, each PN accesses to the disk in sequence, and performance degrades terribly. Although local I/O in which each PN
reads from or writes to its own disk solves the problem, it is a very hard work to manage such a large number of partial files. Therefore, parallel I/O is greatly demanded in ES from the point of view of both performance and usability. The parallel file system (PFS) provides the parallel I/O features to ES (Fig. ). It enables handling multiple files, which are located on separate disks of multiple PNs, as logically one large file. Each process of a parallel program can read/write distributed data from/to the parallel file concurrently with one I/O statement to achieve high performance and usability of I/O.
E
E
Earth Simulator
Job Scheduling ES is basically a batch-job system. Network Queuing System II (NQSII) is introduced to manage the batch job (Fig. ). There are two type queues. One is L batch queue and the other is S batch queue. S batch queue is aimed at being used for a pre-run or a post-run for large-scale
batch jobs (making initial data, processing results of a simulation, and other processes), and L batch queue is for a production run. Users choose an appropriate queue for their jobs. S batch queue is designed for single-node batch jobs. In S batch queue, Enhanced Resource Scheduler (ERSII) is used as a scheduler and does a job scheduling based
Running
Login server
L-system
Waiting Batch job entry (qsub)
Multi-node batch job
Node number J1
J2 J3
J2 J1
J3 Time
Current
L batch queue
S-system
AP number
S batch queue Single-node batch job in S-system Current
Time
Earth Simulator. Fig. Shows the queue configuration of ES
For single-node batch jobs Nodes for single-node batch jobs
For multi-node batch jobs Nodes for multi-node batch jobs
Interactive nodes
•••••••
•••••••••
S-disk Stage-in/out HDME disk
Submitting jobs (single-node jobs)
DATA disk
Work disk
Work disk
••••••••
Work disk
Stage-in/out automatically before/after job execution Submitting jobs (multi-node jobs)
NFS mount MDPS (M-Disk)
User
Batch script
NFS mount MT
MT
Login server Remote login
Tape library
Earth Simulator. Fig. Job execution. The user writes a batch script and submits the batch script to ES. The node scheduling, the file staging, and other processing are automatically processed by the system
Earth Simulator
on CPU time. On the other hand, L batch queue is for multi-node batch jobs. In this queue, the customized scheduler for ES is used as the scheduler. We have been developing this scheduler with following strategies: . The nodes allocated to a batch job are used exclusively for that batch job. . The batch job is scheduled based on elapsed time instead of CPU time. Strategy () enables to estimate the job termination time and to make it easy to allocate nodes for the next batch jobs in advance. Strategy () contributes to an efficiency
Earth Simulator. Table Programming model on ES
Inter-PN
“Hybrid”
“Flat”
HPF/MPI
HPF/MPI
Intra-PN
Microtasking/openMP
AP
Automatic vectorization
CPU
CPU
job execution. The job can use the nodes exclusively and the processes in each node can be executed simultaneously. As a result, the large-scale parallel program is able to be executed efficiently. PNs of L-system are prohibited from access to the user disk to ensure enough disk I/O performance. Therefore, the files used by the batch job are copied from the user disk to the work disk before the job execution. This process is called “stage-in.” It is important to hide this staging time for the job scheduling. Main steps of the job scheduling are summarized as follows: . Node allocation . Stage-in (copies files from the user disk to the work disk automatically) . Job escalation (rescheduling for the earlier estimated start time if possible) . Job execution . Stage-out (copies files from the work disk to the user disk automatically)
CPU
CPU
Microtask/OpenMp
CPU
CPU
Microtask/OpenMp HPF/MPI
Hybrid parallelization
CPU
CPU
CPU
CPU
HPF/MPI
CPU
HPF/MPI HPF/MPI
Flat parallelization
Earth Simulator. Fig. Two types of parallelization of ES
E
CPU
E
E
Eden
When a new batch job is submitted, the scheduler searches available nodes (Step ). After the nodes and the estimated start time are allocated to the batch job, stage-in process starts (Step ). The job waits until the estimated start time after stage-in process is finished. If the scheduler finds the earlier start time than the estimated start time, it allocates the new start time to the batch job. This process is called “Job Escalation” (Step ). When the estimated start time has arrived, the scheduler executes the batch job (Step ). The scheduler terminates the batch job and starts stage-out process after the job execution is finished or the declared elapsed time is over (Step ) (Fig. ).
Programming Model in ES The ES hardware has a three-level hierarchy of parallelism: vector processing in an AP, parallel processing with shared memory in a PN, and parallel processing among PNs via IN. To bring out high performance of ES fully, you must develop parallel programs that make the most use of such parallelism. As shown in Table , the three-level hierarchy of parallelism of ES can be used in two manners, which are called hybrid and flat parallelization, respectively (Fig. ). In the hybrid parallelization, the internode parallelism is expressed by HPF or MPI, and the intranode by microtasking or OpenMP, and you must, therefore, consider the hierarchical parallelism in writing your programs. In the flat parallelization, the both inter- and intranode parallelism can be expressed by HPF or MPI, and it is not necessary for you to consider such complicated parallelism. Generally speaking, the hybrid parallelization is superior to the flat in performance and vice versa in ease of programming. Note that the MPI libraries and the HPF runtimes are optimized to perform as well as possible both in the hybrid and flat parallelization.
Bibliography . Sato T, Kitawaki S, Yokokawa M () Earth simulator running In: th international supercomputer conference ISC, Hedielberg . Kitawaki S () The development of the earth simulator. In: LACSI symposium (Keynote speech), Santa Fe . Habata S, Kitawaki S, Yokokawa M () The earth simulator system. NEC Res Dev :():– . Inasaka J, Ikeda R, Umezawa K, Ko Y, Yamada S, Kitawaki S () Hardware technology of the earth simulator. NEC Res Dev ():–
Eden Rita Loogen Philipps-Universität Marburg, Marburg, Germany
Definition Eden is a parallel functional programming language that extends the non-strict functional language Haskell with constructs for the definition and instantiation of parallel processes. The programmer is freed from managing synchronization and data exchange between processes while keeping full control over process granularity, data distribution, and communication topology. Eden is geared toward distributed settings, that is, processes do not share any data. Common and sophisticated parallel communication patterns and topologies, that is, algorithmic skeletons, are provided as higher-order functions in a skeleton library written in Eden. Eden is implemented on the basis of the Glasgow Haskell Compiler GHC [], a mature and efficient Haskell implementation. While the compiler frontend is almost unchanged, the backend is extended with a parallel runtime system (PRTS) []. This PRTS uses suitable middleware (currently PVM or MPI) to manage parallel execution.
Discussion Introduction Functional languages are promising candidates for parallel programming, because of their high level of abstraction and, in particular, because of their referential transparency. In principle, any sub-expression could be evaluated in parallel. As this implicit parallelism would lead to too much overhead, modern parallel functional languages allow the programmers to specify parallelism explicitly. The underlying idea of Eden is to enable programmers to specify process networks in a declarative way. Processes map input to output values. Inputs and outputs are transferred via unidirectional one-to-one channels. A comprehensive definition of Eden including a discussion of its formal semantics and its implementation can be found in the journal paper []. We describe only the essentials here.
Eden
Basic Eden Constructs The basic Eden coordination constructs are process abstraction and instantiation:
p r o c e s s :: ( T r a n s a , (a → b) ( # ) :: ( T r a n s a , Process a
Trans b) → Process Trans b) b → a →
⇒ a b ⇒ b
The function process embeds functions of type a→b into process abstractions of type Process a b, while the instantiation operator ( # ) takes such a process abstraction and input of type a to create a new process which consumes the input and produces output of type b. Thus, the instantiation operator leads to a function application, with the side effect that the function is evaluated by a remote child process. Its argument is concurrently, that is, by a new thread, evaluated in the parent process and sent to the child process which, in turn, fully evaluates and sends back the result of the function application. Both processes are using implicit : communication channels established between child and parent process on process instantiation. The type context (Trans a, Trans b) states that both a and b must be types belonging to the type class Trans of transmissible values. In general, Haskell type classes provide a structured way to define overloaded functions. Trans defines implicitly used communication functions which by default transmit normal form values in a single piece. The overloading is used twofold. Lists are communicated element-wise as streams and tuple components are evaluated concurrently. An independent thread will be created for each component of an output tuple and the result of its evaluation will be sent on a separate channel. The connection points of channels to processes are called inports on the receiver side and outports on the sender side. There is a oneto-one correspondence between the threads and the outports of a process while data that is received via the inports is shared by all threads of a process. Analogously, several threads will be created in a parent process for tuple inputs of a child process. During its lifetime, an Eden process can thus contain a variable number of threads. The demand-driven (lazy) evaluation of Haskell is an obstacle for parallelism. A completely demand-
E
driven evaluation creates a parallel process only when its result is already needed to continue the main evaluation. This suppresses any real parallel evaluation. Although Eden overrules the lazy evaluation, for example, by evaluating inputs and outputs of processes eagerly and by sending those values immediately to the corresponding recipient processes, it is often necessary to use explicit demand control in order to start processes speculatively before their result values are needed by the main computation. Evaluation strategies [] are used for that purpose. We will not go into further details. In the following, we will use the (predefined) Eden function spawn to eagerly and immediately instantiate a finite list of process abstractions with their corresponding inputs. Appropriate demand control is incorporated in this function. Neglecting demand control, spawn would be defined as follows: s p a w n :: ( T r a n s a , T r a n s b ) ⇒ [ Process a b] → [a] → [b] -- d e f i n i t i o n w i t h o u t d e m a n d c o n t r o l spawn = z i p W i t h (#)
The variant spawnAt additionally locates the created processes on given processor elements (identified by their number). s p a w n A t :: ( T r a n s a , T r a n s b ) ⇒ [ Int ] → [ P r o c e s s a b ] → [ a ] → [ b ]
Example:
A
parallel
variant
of
the
function
map :: (a→b) → [a] → [b] which creates a
process for each application of the parameter function to an element of the input list can be defined as follows: parMap
:: ( T r a n s a , T r a n s b ) ⇒ (a → b) → [a] → [b]
parMap f = spawn ( repeat ( process
f ))
The Haskell prelude function repeat :: a → [a] yields a list infinitely repeating the parameter element.
⊲
To model many-to-one communication, Eden provides a nondeterministic function merge that merges a list of streams into a single stream.
E
E
Eden
m e r g e :: T r a n s a ⇒
[[ a ]]
→
[a]
This function can, for example, be used by the master in a master–worker system to receive the results of the workers as a single stream in the time-dependent order in which they are provided. Although merge is of great worth, because it is the key to specify many reactive systems, one has to be aware that functional purity and its benefits are lost when merge is being used in a program. Fortunately, functional purity can be preserved in most portions of an Eden program. In particular, it is, for example, possible to use sorting in order to force a particular order of the results returned by a merge application and thus to encapsulate merge in a deterministic context. Example: A simple master–worker system with the following functionality can easily be defined in Eden as shown in Fig. . The master process distributes tasks to worker processes, which solve the tasks and return the results to the master: tasks
results master
masterWorker np pf f toWs
tag_map f worker 1
fromWs
...
tag_map f worker np
The Eden function masterWorker (evaluated by the “master” process) takes four parameters: np specifies the number of worker processes that will be spawned, prefetch determines how many tasks will initially be sent by the master to each worker process, the function f describes how tasks have to be solved by the workers, and the final parameter tasks is the list of tasks that have to be solved by the whole system. The auxiliary pure Haskell function distribute :: Int -> [a] -> [Int] -> [[a]] (code not shown) is used to distribute the tasks to the workers. Its first parameter determines the number of output lists, which become the input streams for the worker processes. The third parameter is the request list reqs which guides the task distribution. The same list is used to sort
the results according to the original task order (function orderBy :: [[b]] -> [Int] -> [b], code not shown). Initially, the master sends as many tasks as specified by the parameter prefetch in a round-robin manner to the workers (see definition of initReqs). Further tasks are sent to workers which have delivered a result. The list newReqs is extracted from the worker results tagged with the corresponding worker id which are merged according to the arrival of the worker results. This simple master–worker definition has the advantage that the tasks need not be numbered to reestablish the original task order on the results. Moreover, worker processes need not send explicit requests for new work together with the result values.
Defining Non-hierarchical Process Networks in Eden With the Eden constructs introduced up to now, communication channels are only established between parent and child processes during process creation. This results in purely hierarchical process topologies. Eden further provides functions to create and use explicit channel connections between arbitrary processes. These functions are rather low level and it is not easy to use them appropriately. Therefore, we will not show them here, but explain two more abstract approaches to define non-hierarchical process networks in Eden: remote data and graph specifications. Remote data of type a is represented by a handle of type RD a with the following interface functions []: release :: a -> RD a fetch :: RD a -> a. The function release yields a remote data handle that can be passed to other processes, which will in turn use the function fetch to access the remote data. The data transmission occurs automatically from the process that releases the data to the process which uses the handle to fetch the remote data. Example: We show a small example where the remote data concept is used to establish a direct channel connection between sibling processes. Given functions f and g, one can calculate (g ○ f) a in parallel creating a process for each function.
Eden
E
m a s t e r W o r k e r :: ( T r a n s a , T r a n s b ) ⇒ Int → Int → ( a → b ) → [ a ] → [ b ] m a s t e r W o r k e r np p r e f e t c h f t a s k s = o r d e r B y f r o m W s reqs where fromWs = s p a w n w o r k e r s toWs workers = [ p r o c e s s ( t a g _ m a p f ) ∣ n ← [1.. np ]] toWs = d i s t r i b u t e np t a s k s reqs newReqs = m e r g e [[ i ∣ r ← rs ] ∣ (i , rs ) ← zip [1.. np ] f r o m W s ] reqs = i n i t R e q s ++ n e w R e q s i n i t R e q s = c o n c a t ( r e p l i c a t e p r e f e t c h [1.. np ])
Eden. Fig. Eden definition of a simple master–worker system
main
main
inp
f
a
inp
g Indirect connection
release.f g.fetch Direct connection
b
Eden. Fig. A simple process graph
Simply replacing the function calls by process instantiations (process g # (process f # inp))
leads to the process network in Fig. a where the process evaluating the above expression is called main. Process main instantiates a first child process calculating g, thereby creating a concurrent thread to evaluate the input for the new child process. This thread instantiates the second child process for calculating f. It receives the remotely calculated result of the f process and passes it to the g process. The drawback of this approach is that the result of the f process will not be sent directly to the g process. This causes unnecessary communication costs. In the second implementation, we use remote data to establish a direct channel connection between the child processes: process (g o fetch) # (process (release of) # inp) It uses function release to produce a handle of type RD a for data of type a. Calling fetch with remote data returns the value released before. The use of remote
data leads to a direct communication of the actual data between the processes evaluating f and g (see Fig. b). The remote data handle is treated like the original data in the first version, that is, it is passed via the main process from the process computing f to the one computing g. ⊲ The remote data concept enables the programmer to easily build complex topologies by combining simpler ones [].
Grace
(Graph-based communication in Eden) is a library that allows a programmer to specify a network of processes as a graph, where the graph nodes represent processes and the edges of communication channels []. The graph is described as a Haskell data structure ProcessNetwork a, where a is the type of the result computed by the process network. A function build is used to define a process network, while a function start instantiates such a network and automatically sets up the corresponding process topology, that is, the processes are created and the necessary communication channels are installed.
E
E
Eden
b u i l d :: f o r a l l f a g r p n e . -- type c o n t e x t ( Placeable f a g r p, Ord e , Eq n ) ⇒ -- main node (n , f ) → [ Node n ] → -- node list [ Edge n e ] → -- edge list ProcessNetwork r s t a r t :: ( T r a n s a ) ⇒ ProcessNetwork a → a
The function build transforms a graph specification into a value of type ProcessNetwork r. The graph is defined by a list of nodes of type [Node n] and a list of edges of type [Edge n e]. Type variables n and e represent the types of node and edge labels, respectively. To allow functions with different types to be associated with graph nodes, the type f of node functions is existentially quantified and only explicitly given for the main node, because the function placed on the main node determines the result type r of the process network. This is also the reason why the main node is not a member of the node list but provided as a separate parameter. The function build uses multiparameter classes with functional dependencies and explicit quantification to achieve flexibility in the specification of graphs, for example, by placing arbitrary functions on nodes and nevertheless using standard Haskell lists to represent node lists. The type context Ord e and Eq n ensures that edges can be ordered by their label and that nodes can be identified by their label. Placeable is a multiparameter type class with dependent types, which is used by the Grace implementation to partition user supplied function types into their parts, for example,
parameter and result types. This is needed to create individual channels for these parts. Suitable instances will be derived automatically for every possible function. In the type context Placeable f a g r p the function’s type f determines the other types: a is the type of the function’s first argument, g is the remaining part of the function’s type without the first argument. The final result type of the function after applying all parameters is r. Finally, p is a type level list of all the parameters. The main benefit of Grace is a clean separation between coordination and computation. The network specification encapsulates the coordination aspects. The graph nodes are annotated with functions describing the computations of the corresponding processes. Example: We consider again a simple process network which is similar to the one in Fig. b and show its specification in Grace. The behavior of the three processes is specified by functions f, g, and root. The communication topology is defined by a graph consisting of three nodes (a root node and two additional nodes) and three edges. The specification is shown in Fig. . ⊲
Algorithmic Skeletons Algorithmic skeletons [] capture common patterns of parallel evaluations like task farms, pipelines, divideand-conquer schemes, etc. The application programmer only needs to instantiate a skeleton appropriately, thereby concentrating on the problem-specific matters and trusting on the skeleton with respect to all parallel details. The small introductory example functions parMap and masterWorker shown above are examples for skeleton definitions in Eden. In Eden and other parallel functional languages, skeletons are no more than polymorphic higher-order functions which can be applied
example = start network where -- fct. spec. of process behaviour f, g, root :: (Trans a ) => [a] -> [a] ... -- process graph specification network = build ("root", root) nodes edges nodes = [N "nd_f" f , N "nd_g" g ] edges = [E "nd_f" "nd_g" 0 nothing, E "nd_g" "root" 0 nothing, E "root" "nd_f" 0 nothing]
Eden. Fig. Grace specification of a simple process graph
root
f
g
Eden
with different types and parameters. Thus, programming with skeletons follows the same principle as programming with higher-order functions. An important issue is that skeletons can be both used and implemented in Eden. In other skeletal approaches, the creation of new skeletons is considered as a system programming task, or even as a compiler construction task. Skeletons are implemented by using imperative languages and/or parallel libraries. Therefore, these systems offer a closed collection of skeletons which the application programmer can use, but without the possibility of creating new ones, so that adding a new skeleton usually implies a considerable effort. The Eden skeleton library contains a big collection of different skeletons. Various kinds of skeleton definitions in Eden together with cost models to predict the execution time of skeleton instantiations have been presented in a book chapter []. Parallel map implementations have been analyzed in []. An Eden implementation of the large-scale mapand-reduce programming model proposed by Google [] has been investigated in []. Hierarchical master– worker schemes with several layers of masters and submasters can elegantly be created by nesting a simple single-layer master–worker scheme. More elaborate hierarchical master–worker systems have been discussed in []. Example: (A regular fixed-branching divide and conquer skeleton) As a nontrivial example of a skeleton definition we present a dynamically unfolding divide-and-conquer skeleton []. The essence of a divide-and-conquer algorithm is to decide whether the input is trivial and, in this case, to solve it, or else to decompose nontrivial input into a number of subproblems, which are solved recursively, and to combine the output. A general skeleton takes parameter functions for this functionality, as shown here:
type D i v i d e C o n q u e r a = ( a → Bool ) → -(a → b) → -( a → [ a ]) → -([ b ] → b ) → -a → b --
b trivial ? solve split combine input / result
E
The resulting structure is a tree of task nodes where successor nodes are the sub-problems, the leaves representing trivial tasks. A fundamental Eden skeleton which specifies a general divide and conquer algorithm structure can be found in []. Here, we show a version where every nontrivial task is split into a fixed number of sub-tasks. Moreover, the process tree is created in a distributed fashion: One of the tree branches is processed locally, the others are instantiated as new processes, as long as PEs are available. These branches will recursively produce new parallel subtasks, resulting in a distributed expansion of the computation. In the following binary tree of task nodes, the boxes indicate which task nodes will be evaluated by the eight processes:
In this setting, explicit placement of processes is essential to ensure that processes are not placed on the same processor element while leaving others unused. In [], this kind of skeleton is compared with a flat expansion version, where the main process unfolds the tree up to a given depth, usually with more branches than available processor elements (PEs). The resulting subtrees can then be evaluated by a farm of parallel worker processes, the main process combines the results of the subprocesses. A master–worker system (as shown above) can be used to implement this version. Figure shows the distributed expansion divideand-conquer skeleton for k-ary task trees. Besides the standard parameter functions, the skeleton takes the branching degree, and a ticket list with PE numbers to place newly created processes. The left-most branch of the task tree is solved locally, other branches are instantiated using the Eden function spawnAt, which instantiates a collection of processes (given as a list) with respective input, on explicitly specified PEs. Results are combined by the combine function. Explicit Placement via Tickets: The ticket list is used to control the placement of newly created processes. First, the PE numbers for placing the immediate child
E
E
Eden
dcN :: (Trans a, Trans b) ⇒ Int → [Int] → -- branch degree/tickets DivideConquer a b dcN k tickets trivial solve split combine x |null tickets = seqDC x |trivial x = solve x |otherwise = ... -- code for demand control omitted combine (myRes:childRes ++ localRess) where -- sequential computation seqDC x = if trivial x then solve x else combine (map seqDC (split x)) -- child process generation childRes = spawnAt childTickets childProcs procIns childProcs = map (process ◦ rec_dcN) theirTs rec_dcN ts = dcN k ts trivial solve split combine -- ticket distribution (childTickets, restTickets) = splitAt (k-1) tickets (myTs: theirTs) = unshuffle k restTickets -- input splitting (myIn:theirIn) = split x (procIns, localIns) = splitAt (length childTickets) theirIn -- local computations myRes = ticketF myTs myIn localRess = map seqDC localIns
Eden. Fig. Distributed expansion divide and conquer skeleton for k-ary task trees
processes are taken from the ticket list. Then, the remaining tickets are distributed to the children in a round-robin manner using the function unshuffle :: Int -> [a] -> [[a]] which unshuffles a given list into as many lists as the first parameter tells. Child computations will be performed locally when no more tickets are available. The explicit process placement via ticket lists is a simple and flexible way to control the distribution of processes as well as the recursive unfolding of the task tree. If too few tickets are available, computations are performed locally. Duplicate tickets can be used to allocate several child processes on the same PE. ⊲
Implementation Eden’s implementation follows a microkernel approach. The kernel implements a few, general control mechanisms and provides them as primitive operations (see Fig. ). The more complex high-level language constructs are implemented in libraries using the primitive base constructs. Apart from simplifying the implementation, this approach has important advantages with respect to productivity and maintainability of the system.
Details on Eden’s implementation, especially on primitives provided by the interface of the parallel runtime system (PRTS) and the implementation of the Eden module can be found in [, ].
EdenTV – The Eden Trace Viewer Tool The Eden trace viewer tool (EdenTV) [] provides a postmortem analysis of program executions on the level of the computational units of the PRTS. The latter is instrumented with special trace generation commands activated by a runtime option. In the spacetime diagrams generated by EdenTV, machines (i.e., processor elements), processes, and threads are represented by horizontal bars, with time on the x-axis. The diagram bars have segments in different colors, which indicate the activities of the respective logical unit in a period during the execution. Bars are grey, when the logical unit is running, light grey, when it is runnable, but currently not running, and dark grey, when the unit is blocked. Messages between processes or machines are optionally shown by gray arrows which start from the sending unit bar and point at the receiving unit bar. The representation of messages is very important for programmers, since they can observe
Eden
Eden program
E
Language Haskell libraries
Eden module and libraries
System
Eden primitives Kernel
Sequential RTS
Parallel RTS Suitable middleware
E
Eden. Fig. Implementation layers
11.6 11.8
12
12.2 12.4 12.6 12.8
13
13.2 13.4 13.6 13.8
14
14.2 14.4 14.6 14.8
15
15.2 15.4 15.6 15.8
16
16.2 16.4 16.6
Zoom of the final communication phase of the hyperquicksort algorithm when sorting 5M elements on 17 machines of a Beowulf cluster. Messages are shown as black arrows.
Eden. Fig. Trace of hypercube communication topology of dimension
hot spots and inefficiencies in the communication during the execution as well as control communication topologies. Figure shows in grayscale (light grey = yellow, grey = green, dark grey = red) the trace of a recursive parallel quicksort in a hypercube of dimension with message traffic, exposing the well-known butterfly communication pattern imposed by the hypercube structure.
Future Directions Lines of future studies are ●
It is planned for the future to maintain a common parallel runtime environment for Eden, Glasgow parallel Haskell (GpH), and other parallel Haskells. Such a common environment is highly desirable and
would improve the availability and acceptance of parallel Haskells. ● Recent results [] show that the Eden system, although tailored for distributed memory machines, behaves equally well on workstation clusters and on multi-core machines. It needs to be investigated whether the Eden system can be optimized for multi-core machines by exploiting the shared memory and thereby avoiding unnecessary communications. ● We plan to design and investigate further skeletons. The flexibility of defining individual skeletons for several algorithm classes has not been completely exploited yet. Much more experience in the design and use of skeletons is necessary. ●
Recent studies on remote data and the Grace system have shown that high-level parallel programming notations simplify parallel programming a lot. More
E
Eden
research is necessary to find even more abstract parallel programming constructs. ● Supporting tools like profiling, analysis, visualization, and debugging tools are essential for real-world parallel program development. Much more work is needed to further develop existing, and to design new powerful tools which help the programmers to analyze and tune parallel programs.
Related Entries Concurrent ML Functional Languages Glasgow Parallel Haskell (GpH) Multilisp
receive functions are provided for channel communication. First-class synchronous operations, called events, however, allow to hide channels and complex communication and synchronization protocols behind appropriate abstractions. In contrast to Eden and other parallel functional languages, Concurrent ML has been designed for concurrent programming, that is, with a focus on structuring software and not with the goal of speeding up computations. Recently, however, a shared-memory parallel implementation has been developed []. Manticore is a strict parallel functional language that supports different kinds of parallelism. In particular, it combines explicit concurrency in the style of Concurrent ML with various implicitly parallel constructs which provide fine-grain data parallelism.
NESL Parallel Skeletons Sisal
Glasgow parallel Haskell (GpH) is another parallel Haskell extension that is less explicit about parallelism than Eden. It provides a simple parallel combinator par to annotate sub-expressions that should be evaluated in parallel if free processing elements were available. In contrast to Eden, GpH assumes a global shared memory (at least virtually), and the parallel runtime system decides whether an annotated sub-expression will be evaluated in parallel or not. NESL is a strict data-parallel functional language that supports nested parallel array comprehensions. Parallelism is hidden and concentrated in data-parallel operations. SISAL (Streams and Iteration in a Single Assignment Language) and MultiLisp are early strict parallel functional languages. SISAL is a first-order functional language with implicit parallelism, a rich set of array operations and stream support. It has been designed for numerical applications. MultiLisp extends the LISP dialect Scheme with explicit parallel constructs, in particular so-called futures. Futures initiate parallel evaluations without forcing the main computation to wait for the result. In this respect, futures lead to a special form of laziness within a strict computation language. Concurrent ML extends the strict functional language SML (Standard ML) with concurrency primitives like thread creation and synchronous message passing over explicit channels. Blocking send and
Bibliographic Notes and Further Reading The seminal book on research directions in parallel functional programming [] edited by Hammond and Michaelson covers not only fundamental issues but also provides summaries about selected research areas. Various parallel and distributed variants of the functional programming language Haskell are discussed in a journal paper by Trinder et al. []. A comprehensive overview on patterns and skeletons for parallel and distributed programming can be found in a book edited by Gorlatch and Rabhi []. The three parallel functional languages, Glasgow parallel Haskell, PMLS (a parallel version of ML), and Eden, have been compared with respect to programming methodology and performance in []. Comprehensive and up-to-date information on Eden is provided on its web site http://www.mathematik.uni-marburg.de/∼eden Basic information on its design, semantics, and implementation as well as the underlying programming methodology can be found in []. Details on the parallel runtime system and Eden’s concept of implementation can best be found in [, ]. The technique of layered parallel runtime environments has been further developed and generalized by Berthold et al. [, ]. The Eden trace viewer tool EdenTV is available on Eden’s web site. A short introductory description is given in []. Another tool for analyzing the behavior of Eden programs has been developed by de la Encina et al. [, ]
Eden
by extending the tool Hood (Haskell Object Observation Debugger) for Eden. Extensive work has been done on skeletal programming in Eden. An overview on various skeleton types have been presented as a chapter in the mentioned book by Gorlatch and Rabhi []. Definitions and applications of specific skeletons can, for example, be found in the following papers: parallel map [], topology skeletons [], adaptive skeletons [], Google map-reduce [], hierarchical master–worker systems [], divide-and-conquer schemes []. Special skeletons for computer algebra algorithms are developed with the goal to define the kernel of a computer algebra system in Eden []. An operational and a denotational semantics for Eden have been defined by Ortega-Mallén and Hidalgo-Herrero [, ]. These semantics have been used to analyze Eden skeletons [, ]. A non-determinism analysis has been presented by Segura and Peña []. Eden is intensively being used in teaching parallel programming and algorithms and as a platform for investigating high-level parallel programming concepts, techniques, methodology, and parallel runtime systems. Up to now, its main use in practice has been the implementation of parallel computer algebra components [, ].
Acknowledgments The author is grateful to Yolanda Ortega-Mallén, Jost Berthold, Mischa Dieterle, Thomas Horstmeyer, and Oleg Lobachev for their helpful comments on previous versions of this essay.
Bibliography . Berthold J () Towards a generalised runtime environment for parallel Haskells. Computational Science – ICCS’, LNCS . Springer (Workshop on practical aspects of High-level parallel programming – PAPP ) . Berthold J, Dieterle M, Lobachev O, Loogen R () Distributed memory programming on many-cores - a case study using Eden divide-&-conquer skeletons. ARCS , Workshop on ManyCores. VDE Verlag . Berthold J, Dieterle M, Loogen R () Implementing parallel google map-reduce in Eden. Europar’, LNCS , Springer, pp – . Berthold J, Dieterle M, Loogen R, Priebe S () Hierarchical master–worker skeletons. Practical Aspects of Declarative Languages (PADL ), LNCS , Springer, pp –
E
. Berthold J, Klusik U, Loogen R, Priebe S, Weskamp N () High-level process control in Eden. EuroPar – Parallel Processing, LNCS , Springer, pp – . Berthold J, Loogen R () Skeletons for recursively unfolding process topologies. Parallel Computing: Current & Future Issues of High-End Computing, ParCo , NIC Series, vol , pp – . Berthold J, Loogen R () Parallel coordination made explicit in a functional setting. Implementation and Application of Functional Languages (IFL ), Selected Papers, LNCS , Springer, pp – (awarded best paper of IFL’) . Berthold J, Loogen R () Visualizing parallel functional program runs – case studies with the Eden trace viewer. Parallel Computing: Architectures, Algorithms and Applications, ParCo , NIC Series, vol , pp – . Berthold J, Marlow S, Zain AA, Hammond K () Comparing and optimising parallel Haskell implementations on multicore. rd International Workshop on Advanced Distributed and Parallel Network Applications (ADPNA-). IEEE Computer Society . Berthold J, Zain AA, Loidl H-W () Scheduling light-weight parallelism in ArtCoP. Practical Aspects of Declarative Languages (PADL ), LNCS , Springer, pp – . Cole M () Algorithmic skeletons: structured management of parallel computation. MIT Press, Cambridge . Dean J, Ghemawat S () MapReduce: simplified data processing on large clusters. Commun ACM ():– . Dieterle M, Horstmeyer T, Loogen R () Skeleton composition using remote data. Practical Aspects of Declarative Programming (PADL ), LNCS , Springer, pp – . Encina A, Llana L, Rubio F, Hidalgo-Herrero M () Observing intermediate structures in a parallel lazy functional language. Principles and Practice of Declarative Programming (PPDP ), ACM, pp – . Encina A, Rodríguez I, Rubio F () pHood: a tool to analyze parallel functional programs. Implementation of Functional Languages (IFL’), Seton Hall University, New York, pp –. Technical Report, SHU-TR-CS--- . Hammond K, Berthold J, Loogen R () Automatic skeletons in template Haskell. Parallel Process Lett ():– . Hammond K, Michaelson G (eds) () Research directions in parallel functional programming. Springer, Heidelberg . Hidalgo-Herrero M, Ortega-Mallén Y () An operational semantics for the parallel language Eden. Parallel Process Lett ():– . Hidalgo-Herrero M, Ortega-Mallén Y () Continuation semantics for parallel Haskell dialects. Asian Symposium on Programming Languages and Systems (APLAS ), LNCS , Springer, pp – . Hidalgo-Herrero M, Ortega-Mallén Y, Rubio F () Analyzing the influence of mixed evaluation on the performance of Eden skeletons. Parallel Comput (–):– . Hidalgo-Herrero M, Ortega-Mallén Y, Rubio F () Comparing alternative evaluation strategies for stream-based parallel functional languages. Implementation and Application of Functional
E
E . .
.
.
.
. .
. .
. . . .
Eigenvalue and Singular-Value Problems
Languages (IFL ), Selected Papers, LNCS , Springer, pp – Horstmeyer T, Loogen R () Graph-based communication in Eden. In: Trends in Functional Programming, vol . Intellect Klusik U, Loogen R, Priebe S, Rubio F () Implementation skeletons in Eden – low-effort parallel programming. Implementation of Functional Languages (IFL ), Selected Papers, LNCS , Springer, pp – Lobachev O, Loogen R () Towards an implementation of a computer algebra system in a functional language. AISC/Calculemus/MKM , LNAI , pp – Loidl H-W, Rubio Diez F, Scaife N, Hammond K, Klusik U, Loogen R, Michaelson G, Horiguchi S, Pena Mari R, Priebe S, Portillo AR, Trinder P () Comparing parallel functional languages: programming and performance. Higher-Order Symbolic Computation ():– Loogen R, Ortega-Mallén Y, Peña R, Priebe S, Rubio F () Parallelism abstractions in Eden. In [], chapter , Springer, pp – Loogen R, Ortega-Mallén Y, Peña-Marí R () Parallel functional programming in Eden. J Funct Prog ():– Peña R, Segura C () Non-determinism analysis in a parallelfunctional language. Implementation of Functional Languages (IFL ), LNCS , Springer Rabhi FA, Gorlatch S (eds) Patterns and skeletons for parallel and distributed computing. Springer, Heidelberg Reppy J, Russo CV, Yiao Y () Parallel concurrent ML. International Conference on Functional Programming (ICFP) , ACM The GHC Developer Team. The Glasgow Haskell Compiler. http://www.haskell.org/ghc Trinder P, Hammond K, Loidl H-W, Peyton Jones S () Algorithm + strategy = parallelism. J Funct Prog ():– Trinder PW, Loidl HW, Pointon RF () Parallel and distributed Haskells. J Funct Prog ( & ):– Zain AA, Berthold J, Hammond K, Trinder PW () Orchestrating production computer algebra components into portable parallel programs. Open Source Grid and Cluster Conference (OSGCC)
either eigenvalues: these are the roots λ of the nthdegree characteristic polynomial: det(A − λI) = .
()
The set of all eigenvalues is called the spectrum of A: Λ(A) = {λ , . . . , λn }. Eigenvalues can be real or complex, whether the matrix is real or complex. or eigenpairs: in addition to an eigenvalue λ, one seeks for the corresponding eigenvector x ∈ Cn − {} which is the solution of the singular system (A − λI)x = .
()
When the matrix A is real symmetric or complex hermitian, the eigenvalues are real. Given a matrix A ∈ Rm×n (or A ∈ Cm×n ), the singular-value decomposition (SVD) of A consists of computing orthogonal matrices U ∈ Rm×m and V ∈ Rn×n (or unitary matrices U ∈ Cm×m and V ∈ Cn×n ) such that the m × n matrix ˜ = U T AV Σ
()
˜ = U H AV) has zero entries except for its diago(or Σ nal leading p × p real submatrix Σ = diag(σ , . . . , σp ) with σ ≥ . . . ≥ σp ≥ . The first p columns of U = [u , . . . , um ], and of V = [v , . . . , vn ], are called the left and right singular vectors, respectively. The singular triplets are (σi , ui , vi ) for i = , . . . , p.
Discussion Mathematical Fundamentals Eigenproblems. Equation has n complex roots, some
Eigenvalue and Singular-Value Problems Bernard Philippe , Ahmed Sameh Campus de Beaulieu, Rennes, France Purdue University, West Lafayette, IN, USA
Definition Given a matrix A ∈ Rn×n or A ∈ Cn×n , an eigenvalue problem consists of computing eigen-elements of A which are:
of them possibly multiple. For every eigenvalue λ ∈ C, the dimension of the eigenspace Xλ = ker(A − λI) is at least . The algebraic multiplicity of λ (its multiplicity as root of ( )) is larger or equal to its geometric multiplicities (the dimension of Xλ ). When all the geometric multiplicities are equal to the algebraic ones, a basis X = [x , . . . , xn ] ∈ Cn×n of eigenvectors (say right eigenvectors) can be built by concatenating bases of the invariant subspaces X λ where λ ∈ Λ(A). In such a situation, the matrix A is said to be diagonalizable since by expressing the same operator in the basis X, D = X − AX = diag(λ , . . . , λn ).
()
Eigenvalue and Singular-Value Problems
By denoting X −H = Y = [y , . . . , yn ], Eq. becomes D = Y H AX with Y H X = I. For i = , . . . , n, the vector yi is the left eigenvector corresponding to the eigenvalue λi . The well-known Jordan normal form generalizes the diagonalization when Eq. does not hold. However, the problem of determining such a decomposition is ill-posed. Transforming it into a well-posed problem can be done by proving the existence of matrices with an assigned block structure in a neighborhood of the matrix. For more details see Kagström and Ruhe’s contribution in []. A decomposition which is numerically computable is called the Schur decomposition, in which a unitary matrix U exists such that the complex matrix
E
matrix, while the augmented symmetric matrix ⎛ A⎞ ⎟ Aaug = ⎜ ⎜ ⎟ T ⎝A ⎠
()
has eigenvalues ±σ , . . . , ±σn , corresponding to the eigenvectors ⎛ ⎜ ui √ ⎜ ⎝ ±v i
⎞ ⎟, ⎟ ⎠
E
i = , . . . , n.
Every method for computing singular values of A is based on computing the eigenpairs of B or Aaug .
The Symmetric Eigenvalue Problem ⎛ λ τ ⋯ τ n ⎜ ⎜ ⎜ ⋱ ⋱ ⋮ ⎜ T = U H AU = ⎜ ⎜ ⎜ ⋱ τ n−,n ⎜ ⎜ ⎜ λn ⎝
⎞ ⎟ ⎟ ⎟ ⎟ ⎟, ⎟ ⎟ ⎟ ⎟ ⎟ ⎠
Three class of methods are available for solving problems () and () when A is symmetric: ()
is upper triangular. Eigenvectors can then be computed from T. When the matrix A is Hermitian – or real symmetrics – so is the matrix T which becomes diagonal and real. The symmetric eigenvalue problem deserves a lot of attention since it arises frequently in practice. When the matrix A is real unsymmetric, pairs of complex conjugate eigenvalues may exist. In order to avoid complex arithmetic in that case, the real Schur decomposition extends the Schur decomposition by allowing × diagonal blocks with pairs of conjugate eigenvalues.
SVD problems. Since the complex case is similar to the
real one and since the SVD of AT is obviously obtained from the SVD of A, the study is restricted to the situation A ∈ Rm×n with m ≤ n. Let A have the singular-value decomposition ( ), then, the symmetric matrix
Jacobi iterations: The oldest method, introduced by Jacobi in [], consists of annihilating successively off-diagonal entries of the matrix via orthogonal similarity transformations: A = A and Ak+ = RTk Ak Rk where Rk is a plane rotation. The scheme is organized into sweeps of n(n−)/ rotations to annihilate every off-diagonal pairs of symmetric entries once. One sweep involves n + O(n ) operations when symmetry is exploited in the computation. The method was abandoned due to high computational cost but has been revived with the advent of parallelism. However, depending on the parallel architecture, it remains more expensive than other solvers. QR iterations: The aim of the QR algorithm relies on the following iteration: A = A and Q = I ⎧ ⎪ ⎪ For k ≥ , ⎪ ⎪ ⎪ ⎪ () ⎪ ⎪ ⎨ (Qk+ , Rk+ ) = qr(Ak ) (QR Factorization) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ A = Rk+ Qk+ , ⎪ ⎩ k+ where the
QR factorization
is defined
in
()
[linear least squares and orthogonalization 12]. Under weak assumptions, matrix Ak
has eigenvalues σ ≥ . . . ≥ σn ≥ , with corresponding eigenvectors (vi ), (i = , . . . , n). B is called the normal
approaches a diagonal matrix for sufficiently large k. A direct implementation of ( ) would involve O(n ) arithmetic operations at each iteration. If,
B = AT A ∈ Rn×n ,
E
Eigenvalue and Singular-Value Problems
however, one obtains the tridiagonal matrix T = QT AQ via orthogonal similarity transformations at a cost of O(n ) arithmetic operations, the number of arithmetic operations of a QR iteration is reduced to O(n ) or even O(n) if only the eigenvalues are needed. To accelerate convergence, a shifted version of the QR algorithm is given by T = T and Q = I ⎧ ⎪ ⎪ For k ≥ , ⎪ ⎪ ⎪ ⎪ () ⎪ ⎪ ⎨ (Qk+ , Rk+ ) = qr(Tk − μk I) (QR Factorization) ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ T = Rk+ Qk+ + μ k I, ⎪ ⎩ k+ where μk is an approximation of the smallest eigenvalue of Tk . Sturm sequence evaluation: The Sturm sequence in λ ∈ R is defined by p (λ) = and for k = , . . . , n by pk (λ) = det(Ak − λ Ik ) where Ak and Ik are the principal matrices of order k of A and of the identity matrix. The number N (λ) of sign changes in the sequence is equal to the number of eigenvalues of A which are smaller than λ. N (λ) can also be computed as the number of negative diagonal entries of the matrix U in the LU factorization of the matrix A − λI = LU. As for the QR iteration, the algorithm is applied to the tridiagonal matrix T, so that each Sturm sequence computation has complexity O(n). Sturm sequence allows one to partition a given interval [a, b] as desired to obtain each of the eigenvalues in [a, b] to a desired accuracy. When needed, the eigenvectors are then obtained by inverse iterations. Many references exist which describe the theory underlying the three methods outlined above, e.g., see []. A parallel version of the cyclic Jacobi algorithm was given by Sameh []. It is obtained by the simultaneous annihilation of several off-diagonal elements by a given orthogonal matrix Uk , rather than only one rotation as is done in the serial version. For example, let A be of order (see Table ) and consider the orthogonal matrix Uk as the direct sum of four independent plane rotations simultaneously determined. An example of such a matrix is
Parallel Jacobi algorithms.
Uk = Rk (, ) ⊕ Rk (, ) ⊕ Rk (, ) ⊕ Rk (, ),
Eigenvalue and Singular-Value Problems. Table Annihilation scheme as in [] (first regime) x
x
x
x
x
x
x
x
where Rk (i, j) is that rotation which annihilates the (i, j) off-diagonal element (⊕ indicates that the rotations are assembled in a single matrix and extended to order n by the identity). Let one sweep be the collection of such orthogonal similarity transformations that annihilate the element in each of the n(n − ) off-diagonal positions (above the main diagonal) only once, then for a matrix of order , the first sweep will consist of seven successive orthogonal transformations with each one annihilating distinct groups of maximum four elements simultaneously as described in the Table . For the remaining sweeps, the structure of each subsequent transformation Uk , k > , is chosen to be the same as that of Rj where j = + (k − ) mod . In general, the most efficient annihilation scheme consists of (r − ) similarity transformations per sweep, where r = ⌊ (n + )⌋, in which each transformation annihilates different ⌊ n⌋ off-diagonal elements (see []). Several other annihilation schemes are possible which are based on round-robin techniques. Luk and Park [] have demonstrated that various parallel Jacobi rotation ordering schemes are equivalent to the sequential row-ordering scheme, and hence share the same convergence properties. The above algorithms are well-suited for shared memory computers. While they can also be implemented on distributed memory systems, their efficiency on such systems may suffer due to communication costs. In order to increase the granularity of the computation (i.e., to increase the number of floating point operations between two communications), block algorithms are considered. A parallel Block-Jacobi algorithm is given by Gimenez et al. in []. This algorithm takes advantage of the symmetry of the matrix.
Eigenvalue and Singular-Value Problems
Tridiagonalization of a symmetric matrix by Householder reductions. The goal is to compute a symmetric tridiago-
nal matrix T, orthogonaly similar to A: T = QT AQ where Q = H ⋯Hn− combines Householder reductions. These transformations are defined in [linear least square and orthogonalization 12]. The transformation H is chosen such that n − zeros are introduced in the first column of H A, the first row being inchanged: ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ H A = ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝
⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆
⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟. ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠
Therefore, by symmetry, applying H on the right results in the following pattern: ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ A = H AH = ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝
⋆ ⋆ ⎞ ⎟ ⎟ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆⎟ ⎟ ⎟ ⎟ ⋆ ⋆ ⋆ ⋆ ⋆⎟ ⎟ ⎟ ⎟ ⋆ ⋆ ⋆ ⋆ ⋆⎟ ⎟ ⎟ ⎟ ⋆ ⋆ ⋆ ⋆ ⋆⎟ ⎟ ⎟ ⎟ ⋆ ⋆ ⋆ ⋆ ⋆⎠
()
In order to take advantage of the symmetry, the reduction A = H AH can be implemented by a symmetric rank-two update which is a routine of the Level BLAS []. The tridiagonal matrix T is obtained by repeating successively similar reductions on columns to n − . The total procedure involves n / + O(n ) arithmetic operations. Assembling the matrix Q = H . . . Hn− requires n / + O(n ) additional operations. As indicated in [linear least square and orthogonalization 12], successive applications of Householder reductions can be compacted into blocks which allow use of routines of Level BLAS []. Such reduction is implemented in routine DSYTRD of LAPACK [] which takes advantage of the symmetry of the process.
E
A parallel version of the algorithm is implemented in routine PDSYTRD of ScaLAPACK []. Parallel QR for the symmetric tridiagonal eigenvalue problem. The QR scheme is the method of choice when
all the eigenvalues and the eigenvectors are sought. A Divide-and-Conquer algorithm was introduced by Dongarra and Sorensen in []. It consists of partitioning the triadiagonal matrix by rank-one tearings. A tridiagonal matrix T of order n can be partitioned ⎛T ⎞ ⎟ + ρff T where T and T as follows: T = ⎜ ⎟ ⎜ ⎝ T ⎠ are two tridiagonal matrices of order n and n such that n = n + n and where f is a vector with zero entries except in position n and n + . If the Schur decompositions T = Q D QT and T = Q D QT are already known, the problem is reduced to computing ˜ = QT TQ = D + ρzz T where the eigenvalues of D ⎛Q ⎞ ⎛D ⎞ ⎟, D = ⎜ ⎟, and z = Qf . Q = ⎜ ⎜ ⎜ ⎟ ⎟ ⎝ Q ⎠ ⎝ D ⎠ The eigenvalues of T are interleaved with the diagonal entries of D and satisfy a simple secular equation obtained from the characteristic polynomial of ˜ The eigenvectors are then explicitely known but D. they must be expressed back in the original basis by pre-multiplying them by Q. The back transformation may even include the initial orthogonal transformation which reduces the original symmetric matrix to the tridiagonal form. By p recursive tearings, the problem is reduced to p independent eigenvalue problems of order m = n/p . Then, the technique described above is recursively applied in parallel to update the eigendecompositions. The total number of arithmetic operations is O(n ). Parallel Sturm sequences. Sturm sequences are often used when only the part of the spectrum, in an interval [a, b], is sought. Several authors discussed the parallel computation of Sturm sequences for a tridiagonal matrix T, and among them are Lo et al. in []. It is hard to parallelize the LU factorization of (T − λI) which provides the number of sign changes in the Sturm sequences, because the diagonal entries of U are obtained by a nonlinear recurrence. One
E
E
Eigenvalue and Singular-Value Problems
possible approach is to consider the linear three-term recursion which computes the Sturm sequence terms pk (λ) = det(Tk − λ Ik ), for k = , . . . , n. Since this computation corresponds to solving a triangular banded system of three diagonals, it can be parallelized, as for instance, by the algorithm described by Chen et al. in []. An analysis of the situation is given in []. Because the sequence (pk (λ)),n could suffer from overflow or underflow, Zhang proposed the computation of a scaled sequence in [] and provided the backward error of the sequence. To parallelize the computation, a more beneficial approach is to consider simultaneous computation of Sturm sequences by replacing bisection of intervals by multisection. However, multisections are efficient only when most of the created subintervals contain eigenvalues. Therefore, in [] a strategy in two steps is proposed: () isolating all the eigenvalues with disjoint intervals, () extracting each eigenvalue from its interval. Multisections are used for step () and bisections or other root finders are used for step (). This approach proved to be very efficient. When the eigenvectors are needed, they are computed independently by Inverse Iterations. A difficulty could arise if one wishes to compute all the eigenvectors corresponding to a cluster of very poorly separated eigenvalues. Demmel et al. discussed in [] the reliability of the Sturm sequence computation in floating point arithmetic where the sequence is no longer monotonic. Therefore, in very rare situations, it is possible to obtain a wrong answer with regular bisection. A robust algorithm, called MRRR developed by Dhillon et al. [], is implemented via LAPACK with routine DSTEMR for the computation of high-quality eigenvectors. Several approaches are discussed in [] for possible parallel implementations of the scheme in DSTEMR.
Also, successive applications of Householder reductions can be compacted into blocks which allow use of Level BLAS []. The algorithm is implemented via routine DGEHRD of LAPACK []. A parallel version of the algorithm is implemented via routine PDGEHRD of ScaLAPACK [].
Parallel solution of the Hessenberg eigenproblem. Sev-
eral attempts for generalizing the Divide-and-Conquer approach used in the symmetric case have been considered, including Dongarra and Sidani in [], Adams and Arbenz in []. In these algorithms, the Hessenberg matrix is partitioned into a sum of a two-block uppertriangular matrix and a rank-one update. Compared to the symmetric case, there are two major drawbacks that limit the benefit of the tearing procedure: () the condition number of the eigenproblems defined by the two diagonal blocks of the first matrix can be much higher than that of the original eigenproblem as shown by Jessup in []. () the Newton iterations involved in the updating process include solving triangular or Hessenberg linear systems rendering the procedure rather more expensive than without tearing. Henry et al. discussed in [] existing approaches for realizing a parallel QR procedure and ended up considering an improved implicit multishifted strategy which allows the use of Level BLAS []. The algorithm is implemented in subroutine PDLAHQR of ScaLAPACK [].
The Singular-Value Problem As mentioned above, methods for solving the symmetric eigenvalue problems can be reconsidered as methods for solving SVD problems. In this section, the study is restricted to the case where A ∈ Rm×n with m ≥ n.
The Unsymmetric Eigenvalue Problem
Parallel Jacobi algorithms for SVD. In the special case
Reduction to the Hessenberg form. Similar to the sym-
where m ≫ n, one should first perform an initial “skinny” QR factorization of A = QR where Q ∈ Rm×n is orthogonal and R ∈ Rn×n is triangular. Next, the SVD T ˜ of R = UΣV provides the SVD of A = UΣV T with ˜ Computing the QR factorization in parallel U = QU. has been outlined in [linear least squares and orthogonalization 12]. The complete process is obtained at the cost of mn + O(mn + n ) arithmetic operations.
metric case, the reduced form is obtained by Householder reductions. But here, lack of symmetry implies that, unlike the picture in (), zeros are not introduced in the upper part of the matrix. Therefore, the process results in an upper Hessenberg matrix. The total procedure involves n / + O(n ) arithmetic operations. Assembling the matrix Q = H . . . Hn− requires n / + O(n ) additional operations.
Eigenvalue and Singular-Value Problems
One-sided and two-sided Jacobi algorithms. The first
one-sided algorithm was introduced by Hestenes in []. Applying a rotation R to both sides of B = AT A for annihilating the entry βij is equivalent to post-multiply A by R for making the columns i and j of AR orthogonal. Therefore, any parallel Jacobi algorithm for solving a symmetric eigenvalue problem can be considered as a parallel one-sided Jacobi algorithm for solving a SVD problem. Each sweep is a sequence of orthogonal transformations Vk made of a set of independent rotations that are successively applied on the right side by Ak+ = Ak V˜ kT (A = A). When the right singular vectors are needed, they are accumulated by Vk+ = Vk V˜k = (V = I). At convergence, the singular values are given by the norms of the columns of Ak and the normalized columns are the corresponding left singular vectors. Each sweep performs mn + O(mn) arithmetic operations. Brent and Luk [] and Sameh [] described possible implementations of this algorithm on multiprocessors. Charlier et al. [] demonstrate that an implementation of Kogbetliantz’s algorithm for computing the SVD of upper-triangular matrices is quite effective on a systolic array of processors. Therefore, the first step in a Jacobi SVD scheme is the QR factorization of A, followed by computing the SVD of ∈ Rn×n . The Kogbetliantz’s method for computing the SVD of a real square matrix A mirrors the method for computing the eigenpairs of symmetric matrices, in that the matrix A is reduced to the diagonal form by an infinite sequence of plane rotations Ak+ = Uk Ak VkT ,
k = , , . . . ,
()
Vk (i, j, ϕ kij ),
Uk = Uk (i, j, θ ijk ) where A ≡ A, and Vk = are plane rotations. It follows that Ak approaches the diagonal matrix Σ = diag(σ , σ , . . . , σn ), where σi is the ith singular value of A, and the products (Uk . . . U U ), (Vk . . . V V ) approach matrices whose ith column is the respective left and right singular vector corresponding to σi . Block Jacobi algorithms. The above algorithms are well-
suited for shared memory architectures. While they can also be implemented on distributed memory systems, their efficiency on such systems will suffer due to communication costs. In order to increase the granularity of the computation (i.e., to increase the number of floating
E
point operations between two communications), block algorithms are considered. For one-sided algorithms, each processor is allocated a block of columns instead of a single column. The computation remains the same as discussed above with the ordering of the rotations within a sweep is as given in []. For the two-sided version, the allocation manipulates -D blocks instead of single entries of the matrix. Beˇcka and Vajtešic propose in [] a modification of the basic algorithm in which one annihilates, in each step, two off-diagonal blocks by performing a full SVD on a small-sized matrix. While reasonable efficiencies is realized on distributed memory systems, this block strategy increases the number of sweeps needed to achieve convergence. We note that parallel Jacobi algorithms can only surpass the speed of the bidiagonalization schemes of ScaLAPACK when the number of processors available are much larger than the size of the matrix under consideration. Bidiagonalization of a matrix by Householder reductions.
The classical algorithm which reduces A into an upperbidiagonal matrix B, via the orthogonal transformation B = U T AV, was introduced by Golub and Kahan (see for instance []). The algorithms requires (mn −n /)+ O(mn) arithmetic operations. U and V can be assembled in (mn − n /) + O(mn) and n / + O(n ) additional operations, respectively. A parallel version of this algorithm is implemented in routine PDGEBRD of ScaLAPACK. The one-sided reduction for bidiagonalizing a matrix was proposed by Raha []. It appears to be better suited for parallel implementation. It consists of determining in a first step the orthogonal matrix V which would appear in the tridiagonalization of AT A: T = V T AT AV = BT B. The matrix V can be obtained by using Householder or Givens reductions, without first forming AT A explicitely. The second step performs a QR factorization of F = AV: F = QR. Since F T F = RT R = T is tridiagonal, its Cholesky factor R is upper-bidiagonal which means that any two nonadjacent columns of F are orthogonal. This allows a simplified QR factorization. Bosner and Barlow introduced in [] two adaptations of Raha’s approach: a block version which allows use of Level BLAS routines and a parallel version. Once the bidiagonalization is performed, all that remains is to compute the SVD of B. When only
E
E
Eigenvalue and Singular-Value Problems
the singular values are sought, it can be done by the iterative Golub and Kahan algorithm where the number of arithmetic operations is O(n) per iteration (see for instance []) which is performed only on uniprocessor. The whole Singular-Value Decomposition is implemented in routine PDGESVD of ScaLAPACK.
Bibliographic Notes and Further Reading Eigenvalue problems and singular-value decomposition: [] Algorithms: [] Linear algebra: [, , ] Parallel algorithms for dense computations: [, ]
Bibliography . Adams L, Arbenz P () Towards a divide and conquer algorithm for the real nonsymmetric eigenvalue problem. SIAM J Matrix Anal Appl ():– . Anderson E, Bai Z, Bischof C, Blackford S, Demmel J, Dongarra J, Du Croz J, Greenbaum A, Hammarling S, McKenney A, Sorensen D () LAPACK users’ guide, rd edn. Society for Industrial and Applied Mathematics, Philadelphia, Library available at http://www.netlib.org/lapack/ . Bai Z, Demmel J, Dongarra J, Ruhe A, van der Vorst H (eds) () Templates for the solution of algebraic eigenvalue problems – a practical guide. SIAM, Software–Environments–Tools, Philadelphia . Beˇcka M, Vajtešic M () Block-Jacobi SVD algorithms for distributed memory systems. Parallel algorithms and applications, Part I in , –, Part II in , – . Blackford LS, Choi J, Cleary A, D’Azevedo E, Demmel J, Dhillon I, Dongarra J, Hammarling S, Henry G, Petitet A, Stanley K, Walker D, Whaley RC () ScaLA-PACK users’ guide. Society for Industrial and Applied Mathematics, Philadelphia, Available on line at http://www.netlib.org/lapack/lug/ . Blackford LS, Demmel J, Dongarra J, Duff I, Hammarling S, Henry G, Heroux M, Kaufman L, Lumsdaine A, Petitet A, Pozo R, Remington K, Whaley RC () An updated set of basic linear algebra subprograms (BLAS). ACM Trans Math Soft –: – . Bosner N, Barlow JL () Block and parallel versions of onesided bidiagonalization. SIAM J Matrix Anal Appl :– . Brent RP, Luk FT () The solution of singular-value and symmetric eigenvalue problems on multiporcessor arrays. SIAM J Sci Stat Comput :– . Charlier J-P, Vanbegin M, Van Dooren P () On efficient implementations of kogbetliantz’s algorithm for computing the singular value decomposition. Numer Math :–
. Chen SC, Kuck DJ, Sameh AH () Practical parallel band triangular system solvers. ACM Trans Math Software :– . Demmel J, Dhillon I, Ren H () On the correctness of some bisection-like parallel eigenvalue algorithms in floating point arithmetic. ETNA :– . Demmel J () Applied numerical linear algebra. SIAM, Philadelphia . Dhillon D, Parlett B, Vömel C () The design and implementation of the MRRR algorithm. ACM Trans Math Software :– . Dongarra JJ, Duff IS, Sorensen DC, Van der Vorst H () Numerical linear algebra for high performance computers. SIAM, Philadelphia . Dongarra JJ, Sidani M () A parallel algorithm for the nonsymmetric eigenvalue problem. SIAM J Sci Comput :– . Dongarra JJ, Sorensen DC () A fully parallel algorithm for the symmetric eigenvalue problem. SIAM J Sci Stat Comput :s–s . Dongarra JJ, Van De Gein R () Parallelizing the QR algorithm for the unsymmetric algebraic eigenvalue problem: myths and reality. SIAM J Sci Comput :– . Gallivan KA, Plemmons RJ, Sameh AH () Parallel algorithms for dense linear algebra computations. In: Gallivan KA, Heath MT, Ng E, Ortega JM, Peyton BW, Plemmons RJ, Romine CH, Sameh AH, Voigt RG (eds) Parallel algorithms computations. SIAM, Philapdelphia . Gimenez D, Hernandez V, van de Gein R, Vidal AM () A Block Jacobi method on a mesh of processors. Conc Pract Exp –:– . Golub GH, Van Loan CF () Matrix computations, rd edn. Johns Hopkins University Press, Baltimore . Henry G, Watkins D, Dongarra J () A parallel implementation of the nonsymmetric QR algorithm for distributed memory architectures. SIAM J Sci Comput :– . Hestenes MR () Inversion of matrices by biorthogonalization and related results. J Soc Ind Appl Math :– . Jacobi CGJ () Uber ein Leiches Vehfahren Die in der theorie der Sacular-storungen Vorkom-mendern Gleichungen Numerisch Aufzuosen. Crelle’s J für reine und angewandte Mathematik :– . Jessup ER () A case against a divide and conquer approach to the nonsymmetric eigenvalue problem. J Appl Num Math : – . Kagström B, Ruhe A () An algorithm for numerical computation of the Jordan normal form of a complex matrix. ACM Trans Math Software :– . Lo SS, Philippe B, Sameh A () A multiprocessor algorithm for the symmetric tridiagonal eigenvalue problem. SIAM J Sci Stat Comput :s–s . Luk F, Park H () On parallel Jacobi orderings. SIAM J Sci Stat Comput :– . Parlett BN () The symmetric eigenvalue problem. SIAM, Classics in Applied Mathematics, Philadelphia . Ralha R () One-sided reduction to bidiagonal form. Linear Algebra Appl :–
EPIC Processors
. Sameh A () On Jacobi and Jacobi-like algorithms for a parallel computer. Math Comp :– . Sameh A () Solving the linear least squares problem on a linear array of processors. In: Snyder L, Gannon DB, Siegel HJ (eds) Algorithmically specialized parallel computers. Academic, H.J. Siegel . Trefethen LN, Bau D III () Numerical linear algebra. SIAM, Philadelphia . Zhang J () The scaled sequence computation. In: Proceedings of the SIAM Conference on Applied Linear Algebra. July –,
EPIC Processors David I. August, Arun Raman Princeton University, Princeton, NJ, USA
Definition Explicitly Parallel Instruction Computing (EPIC) refers to architectures in which features are provided to facilitate compiler enhancements of instruction-level parallelism (ILP) in all programs, while keeping hardware complexity relatively low. Using ILP-enhancement techniques such as speculation and predication, the compiler identifies the operations that can execute in parallel in each cycle and communicates a plan of execution to the hardware.
Discussion
E
EPIC processors combine the relative hardware simplicity of Very Long Instruction Word (VLIW) processors and the runtime adaptability of superscalar processors. VLIW processors require the compiler to specify a POE consisting of a cycle-by-cycle description of independent operations in each functional unit, and the hardware follows the POE to execute the operations. While VLIW considerably simplifies the hardware, it also results in machine-code incompatibility between implementations of a VLIW architecture. Due to the strict adherence to the compiler’s POE, code compiled for one implementation will not execute correctly on other implementations with different hardware resources or latencies. Superscalar processors discover ILP and construct a POE entirely in hardware. This means that code compiled for a particular superscalar architecture will execute correctly on all implementations of that architecture. However, ILP extraction in hardware entails significant complexity, resulting in diminishing performance returns and an adverse impact on clock rate. EPIC processors retain VLIW’s philosophy of statically exposing ILP and constructing the POE, thereby keeping the hardware complexity relatively low, but incorporate the ability of superscalar processors to cope with dynamic factors such as variable memory latency. Combined with techniques to provide binary compatibility for correctness, EPIC processors provide a means to achieve high levels of ILP across diverse applications while reducing the hardware complexity needed for the same.
Introduction
Design Principles
The performance of modern processors is dependent on their ability to execute multiple instructions per cycle. In the early s, instruction-level parallelism (ILP) was the only viable approach to achieve higher performance without rewriting software. Programs for ILP processors are written in a sequential programming model, with the compiler and hardware being responsible for automatically extracting the parallelism in the program. Explicitly Parallel Instruction Computing (EPIC) refers to architectures in which features are provided to facilitate compiler enhancements of ILP in all programs. These architectures allow the compiler to generate an effective plan of execution (POE) of a given program, and provide mechanisms to communicate the compiler’s POE to the hardware.
In EPIC architectures, the compiler is responsible for designing a POE. This means that the compiler must identify the inherent parallelism of a sequential program and map it efficiently to the parallel hardware resources in order to minimize the program’s execution time. The main challenge faced by the compiler in determining a good POE is on account of dynamic events that determine the program’s runtime behavior. Will a load-store pair access the same memory location? If yes, then the operations must be executed sequentially; otherwise, they may be scheduled in any order. Will a conditional branch be taken? If yes, then operations along the taken path may be scheduled in parallel with operations before the branch; otherwise, they must not be executed. In such situations, the compiler can
E
E
EPIC Processors
speculate that a load-store pair will not alias, or that a branch will be taken, with the compiler’s confidence in the speculation typically being determined by the outcomes of profiling runs on representative inputs. EPIC architectures support compiler speculation by providing mechanisms in hardware to ensure correctness if speculation fails. Finally, the architecture must be rich enough for the compiler to express its POE; specifically, which operations are to be issued in parallel and which hardware resources they must use. Section “ILP Enchancement” presents the major techniques used to enhance ILP of programs. Section “Enabling Architectural Features” describes the micro-architectural features that enable these and other ILP enhancement techniques.
ILP Enhancement The code example in Fig. is used to illustrate the main compiler transformations for ILP enhancement that are made possible by EPIC architectures. In the example, a condition is evaluated to alternatively update a memory location p4 with the contents of location p2 or p3. Outside the if-construct, the variable val is updated with the value at location p5, with the latter potentially aliasing with p4. The processor model assumed for illustration purposes, shown in Table , is a six-issue processor capable of executing one branch per cycle, with no further restrictions on the combination of operations that may be concurrently issued. Conditional branches require separate comparison and EPIC Processors. Table Assumed processor model to illustrate ILP enhancement techniques
control transfer operations. All operations are assumed to have a latency of one cycle, with the exception of memory loads that have a latency of two cycles. Figure a shows the scheduled, assembly code. The compiler conservatively respects all dependences, resulting in a rather sparse schedule with several slots going to waste. The execution time along either branch is ten cycles.
Speculation Compiler-controlled speculation refers to breaking inherent programmatic dependences by guessing the outcome of a runtime event at compile time. As a result, the available ILP in the program is increased by reducing the height of long dependence chains and by increasing the scheduling freedom among the operations. Control Speculation
Many general-purpose applications are branchintensive. The latency to resolve a branch directly affects ILP since potentially multiple issue slots may go to waste. It is crucial to overlap other instructions with branches. Control speculation breaks control dependences which occur between branches and other operations. An operation is control dependent on a branch if the branch determines whether the operation will be executed at runtime. By guessing the direction of a branch, the control dependence may be broken effectively making the operation’s execution independent of the branch. Control speculation allows operations to move across branches, thus reducing dependence height and resulting in a more compact schedule.
Issue width
Six instructions
Data Speculation
Latency of conditional branches
Two cycles
Latency of memory loads
Two cycles
Latency of other instructions
One cycle
Many general-purpose applications exhibit irregular memory access patterns. Often, a compiler is conservative because it respects a memory dependence that occurs infrequently or because it cannot determine that the dependence does not actually exist. In either case, data speculation breaks data flow dependences between memory operations. Two memory operations are flow dependent on one another if the first operation writes a value to a memory location and the second operation potentially reads from the same location. By guessing that the two memory operations will access different locations, the load operation may be hoisted above the
if (*p1 == 0) *p4 = *p2; else *p4 = *p3; val += *p5;
EPIC Processors. Fig. Example C code to illustrate ILP enhancement techniques
EPIC Processors
0
E
(1) r1 = MEM[p1]
1 2
(2) c1 = (r1 == 0)
3
(3) JUMP c1, ELSE
4
(4) r2 = MEM[p2]
5 6
(5) MEM[p4] = r2
(6) JUMP CONTINUE
ELSE: 0
E
(7) r3 = MEM[p3]
1 2
(8) MEM[p4] = r3 CONTINUE:
0
(9) r5 = MEM[p5]
1
a 0
2
(10) r4 = r4 + r5
(4) r2 = MEM[p2]
(1) r1 = MEM[p1]
(7) r3 = MEM[p3]
(9) r5 = MEM[p5]
1 2
(2) c1 = (r1 == 0)
3
(3) JUMP c1, ELSE
4
(4′) CHECK r2
5
(10) r4 = r4 + r5
(5) MEM[p4] = r2
(6) JUMP CONTINUE
ELSE: 0 1
(7′) CHECK r3 (8) MEM[p4] = r3 CONTINUE:
b
0
(9′) CHECK r5
0
(1) r1 = MEM[p1]
1 2
(11) pT, pF = (r1 == 0)
3
(4) r2 = MEM[p2]
(7) r3 = MEM[p3]
5
(5) MEM[p4] = r2
(8) MEM[p4] = r3
6
(9) r5 = MEM[p5]
4
7
c 0
(1) r1 = MEM[p1]
8
(10) r4 = r4 + r5
(4) r2 = MEM[p2]
(7) r3 = MEM[p3]
(9) r5 = MEM[p5]
1
d
2
(11) pT, pF = (r1 == 0)
3
(4′) CHECK r2
(7′) CHECK r3
4
(5) MEM[p4] = r2
(8) MEM[p4] = r3
5
(10) r4 = r4 + r5
(9′) CHECK r5
EPIC Processors. Fig. Predication and speculation have a synergistic relationship: applying speculation after predication results in the best improvement in ILP. (a) Sequential schedule. (b) Schedule using control and data speculation alone. (c) Schedule using predication alone. (d) Schedule using predication and speculation combined
E
EPIC Processors
store. EPIC provides for more aggressive code motion: operations dependent on the load may also be hoisted above potentially aliasing stores. Applying speculation to the code in Fig. results in the tighter schedule shown in Fig. b.
to detect if a dependence actually existed for this execution and initiates repair if required. In general, all data-speculative and all potentially excepting controlspeculative operations require checks. In Fig. b, operations ′ , ′ , and ′ are the previously discussed symbolic check operations.
Predication Predicated execution is a mechanism that supports conditional execution of individual operations based on Boolean guards, which are implemented as predicate register values. A compiler converts control flow into predicates by applying if-conversion []. If-conversion translates conditional branches into predicate defining operations and guards operations along alternative paths of control under the computed predicates. A predicated operation is fetched regardless of its predicate value. An operation whose predicate is TRUE is executed normally. Conversely, an operation whose predicate is FALSE is prevented from modifying the processor state. With if-conversion, complex nests of branching code can be replaced by a straight-line sequence of predicated code. Predication increases ILP by allowing separate control flow paths to be overlapped and simultaneously executed in a single thread of control. Figure c illustrates an if-conversion of the code segment from Fig. . The predicate for each operation is shown within angle brackets. For example, operation is predicated on pT. The absence of a predicate indicates that the operation is always executed. Predicates are computed using predicate define operations, such as operation . The predicate pT is set to if the condition evaluates to true, and to if the condition evaluates to false. The predicate pF is the complement of pT. Operations and , and operations and are executed concurrently with the appropriate ones taking effect based on the predicate values. The net result of applying predication is that the dependence height of the code segment is reduced to nine cycles. Combining Speculation and Predication Both speculation and predication provide effective means to increase ILP. However, the example shows that their means of improving performance are fundamentally different. Speculation allows the compiler to break control and memory dependences, while predication allows the compiler to restructure program control flow
EPIC Processors
and to overlap separate execution paths. The problems attacked by both techniques often occur in conjunction; therefore, the techniques can be mutually beneficial. Figure d illustrates the use of predication and speculation in combination. If-conversion removes the branch resulting in a stream of predicated operations. As before, data speculation breaks the dependences between operations and , and operations and , allowing the compiler to move operation to the top of the block. Even though no branches remain in the code, control speculation is still useful to break dependences between predicate definitions and guarded instructions. In this example, the control dependences between operations and , and operations and are eliminated by removing the predicates on operations and . This form of speculation in predicated code is called promotion. As a result of this speculation, the compiler can hoist operations and to the top of the block to achieve a more compact schedule of height six cycles. In summary, predication is only % faster and speculation is only % faster than the original code segment. As the example shows, speculation in the form of promotion can have a greater positive effect on performance after if-conversion than before. This synergistic relationship between predication and speculation yields a combined speedup of % over the original code segment.
Enabling Architectural Features Since EPIC delegates the responsibilities of enhancing ILP and planning an execution schedule to the compiler, the architecture must provide features: . To overcome the worst impediments to a compiler’s ability to enhance ILP, namely, frequent control transfers and ambiguous memory dependences . To specify an execution schedule
Speculation Support As discussed in Section “ILP Enhancement”, an architecture that supports speculation must provide mechanisms to detect potential exceptions on controlspeculative operations as they occur, to record information about data-speculative memory accesses as they occur, and then to check at an appropriate time whether an exception should be taken or data-speculative repair should be initiated.
E
First, speculative operations must be distinguished from nonspeculative operations, since the latter should report exceptions immediately. Also, loads that have not been speculated with regard to data dependence need not interact with memory conflict detection hardware. For this, each operation that can be speculated has an extra bit, called the S-bit, associated with it. This bit is set on operations that are either control speculated or are data dependent on a data-speculative load. Each load has an additional bit, called the DS-bit. This bit is set only on data-speculative loads. Second, to defer the reporting of exceptions arising from speculative operations until it is clear that the operation would have been executed in the original program, a -bit tag called the NaT (Not a Thing) bit is added to each register. When a speculative load causes an exception, the NaT bit is set instead of immediately raising the exception. Speculative instructions that use a register with its NaT bit set propagate the bit through their destination registers until a check operation determines whether speculation failed in which case the deferred exception is suppressed. Third, a mechanism must be provided to store the source addresses of data-speculative loads until their independence with respect to intervening stores can be established. This functionality can be provided by the Memory Conflict Buffer (MCB) [] or the Advanced Load Address Table (ALAT) []. The MCB temporarily associates the destination register number of a speculative load with the address from which the value in the register was speculatively loaded. Destination addresses of subsequent stores are checked against the addresses in the buffer to detect memory conflicts. The MCB is queried by explicit data speculation check instructions, which initiate recovery if a conflict was detected.
Predication Support A set of predicate registers are used to store the results of compare operations. For example, the Intel Itanium processor family has a set of (-bit) predicate registers. An EPIC processor must also implement some form of predicate squash logic, which prevents instructions with false predicates from committing their results. The IMPACT EPIC architecture [] also provides predication-aware bypass/interlock logic, which forwards results based on the predicate values associated with the generating instructions.
E
E
EPIC Processors
Static Scheduling Support EPIC provides many features for enabling the compiler to specify a high-quality Plan Of Execution (POE). These features include the ability to specify multiple operations per instruction, the notion of architecturally visible latencies, and an architecturally visible register structure. Multiple Operations per Instruction (MultiOp)
EPIC architectures allow the compiler to specify explicit information about independent operations in a program. A MultiOp is a specification of a set of operations that are issued simultaneously to be executed by the functional units. Exactly one MultiOp is issued per cycle of the virtual time. The virtual time is the schedule built by the compiler, and may differ from the actual time if the hardware inserts stalls due to dynamic events. For example, the Itanium architecture [] consists of instruction groups that have no read-after-write (RAW) or write-after-read (WAR) register dependencies, delimited by stops. The Itanium implementation consists of -bit instruction bundles comprising three -bit opcodes and a -bit template that includes intrabundle stop bits. The use of template bits greatly simplifies instruction decoding. The use of stops enables explicit specification of parallelism. Architecturally Visible Latencies
A lot of the hardware complexity of a superscalar processor arises from the need to maintain an illusion of sequential execution (i.e., each operation completes before another begins) in the face of non-atomicity of the operations. To avoid the complexity, EPIC processors expose operations as being nonatomic; instead, read-and-write events of an operation are viewed as the atomic events. There is a contractual specification of assumed latencies of operations that must be honored by both the hardware and the compiler. A MultiOp that takes more than one cycle to generate all the results is said to have non-unit assumed latency (NUAL). The guarantee of NUAL affords multiple benefits to EPIC processors. Since the hardware is certain that the read event of an operation will not occur before the write event of the operation that produces the value to be read, there is no need for stall/interlock capability. Since the compiler is certain that an operation will not write its result before its assumed latency has elapsed, it
can schedule other operations that are anti- or outputdependent on the operation in question, resulting in tighter schedules. As mentioned earlier, EPIC processors may still have interlocking to account for the deviation of actual latencies of operations from their assumed latencies. However, MultiOp and NUAL permit an in-order implementation of the architecture while still extracting substantial ILP. Register Structure
ILP processors require a large number of registers. Specifically in EPIC processors, techniques such as control and data speculation cause increased register pressure. Fundamentally, since multiple operations are executing in parallel, there is a demand for a larger store for temporary values. Superscalar processors architecturally expose only a small fraction of the physical registers, and rely on register renaming to generate parallel schedules. However, this requires complex hardware for dynamic scheduling. Since EPIC demands a schedule from the compiler in order to simplify the hardware, it becomes necessary to expose a larger set of architectural registers to generate the same parallel schedule as achieved through hardware register renaming. EPIC architectures also provide a mechanism to rotate the register file to speed up inner-loop execution. Speeding up loops involves executing instructions from multiple loop iterations in parallel. One way to do this is by using Modulo Scheduling []. Modulo scheduling schedules one iteration of the loop such that when this schedule is repeated at regular intervals, no intra- or inter- iteration dependence is violated, and no resource conflict arises between the dynamic operations within and across loop iterations. While generating code, it is important that false dependences do not arise owing to dynamic instances on different iterations of the same operation writing to the same registers. While the loop could be unrolled, followed by static register renaming, this typically results in substantial code growth. EPIC architectures implement a rotating register file that provides register renaming such that successive writes to the same virtual register actually go to different physical registers. This is accomplished through the use of a rotating register base (RRB) register. The physical register number accessed by each operation is the sum of the virtual register number specified in the instruction
EPIC Processors
and the number in the RRB register, modulo the number of registers in the rotating register file. The RRB is updated at the end of each iteration using special operations. In this manner, EPIC architectures provide compiler-controlled dynamic register renaming. Another interesting feature is the register stack that is used to simplify and speed up function calls. Instead of spilling and filling all the registers at a procedure call and return sites, EPIC processors enable the compiler to rename the virtual register identifiers. The physical register accessed is determined by renaming the virtual register identifier in the instruction through a base register. The callee gets a physical register frame that may overlap with the caller’s frame; the overlap constitutes the registers that are passed as parameters.
Cache Management EPIC architectures allow the compiler to explicitly control data movement through the cache hierarchy by exposing the hierarchy architecturally. Generating a high-quality schedule often requires the compiler to be able to accurately predict the level in the cache hierarchy where a load operation will find its data. This is normally achieved by profiling or analytical means, and the source-cache specifier is encoded as part of the load instruction. This informs the hardware of the expected location of the datum as well as the assumed latency of the load. EPIC architectures also provide a target cache specifier for load/store operations that allow a compiler to instruct the hardware of where the datum should be put in the cache hierarchy for subsequent references to that datum. This allows the compiler to explicitly control the content of the caches. The compiler could, for example, prevent pollution of the caches by excluding data with poor temporal locality from the higher-level caches, and remove data from the last level cache after the last use. Many streaming programs operate on data without accessing them again. To improve the latency of such accesses, EPIC architectures provide a data prefetch cache. Prefetch load instructions can be used to prefetch data with poor temporal locality into the data prefetch cache. This prevents the displacement of first-level data cache contents that may potentially have much better temporal locality.
E
Related Entries Cydra Dependences Instruction-Level Parallelism Modulo Scheduling and Loop Pipelining Multiflow Computer Parallelization, Automatic
E Bibliographic Notes and Further Reading The EPIC philosophy was primarily developed by HP researchers in the early s [, ]. As mentioned in the introduction, the EPIC philosophy was fundamentally influenced by the VLIW philosophy developed in the s in the Multiflow and Cydrome processors [, ]. EPIC inherited various features from Cydrome, most significantly: ● ●
Predicated execution Support for software pipelining of loops in the form of rotating register files ● A rudimentary MultiOp instruction format ● Hardware support for speculation including speculative opcodes, a speculative tag bit in each register, and deferred exception handling Many of the ideas on speculation and its hardware support were developed independently by researchers in the IMPACT group at the University of Illinois [] and IBM Research []. The most famous commercial EPIC processor is the Itanium processor family jointly developed by HP and Intel. The first processor in the family was released in , with successors having improved memory subsystems, multiple cores on the same die, and improved power efficiency. More information on the EPIC philosophy and implementation can be found in the technical reports and manuals on the topic [, ].
Bibliography . Allen JR, Kennedy K, Porterfield C, Warren J () Conversion of control dependence to data dependence. In: Proceedings of the th ACM Symposium on Principles of Programming Languages, pp –, January . Gallagher DM, Chen WY, Mahlke SA, Gyllenhaal JC, Hwu WW () Dynamic memory disambiguation using the memory conflict buffer. In: Proceedings of th International Conference on
E .
.
.
.
. .
.
.
.
Erlangen General Purpose Array (EGPA)
Architectural Support for Programming Languages and Operating Systems, pp –, October Intel Corporation () Intel Itanium architecture software developer’s manual. Application architecture, vol , Revision .. Santa Clara, CA August DI, Connors DA, Mahlke SA, Sias JW, Crozier KM, Cheng B, Eaton PR, Olaniran QB, Hwu WW () Integrated predication and speculative execution in the IMPACT EPIC architecture. In: Proceedings of the th International Symposium on Computer Architecture, pp –, June Rau BR () Iterative modulo scheduling: an algorithm for software pipelining loops. In: Proceedings of the th International Symposium on Microarchitecture, pp –, December Schlansker MS, Rau BR () “EPIC: An architecture for instruction-level parallel processors,” Technical Report HPL-, Compiler and Architecture Research HP Laboratories Palo Alto, February Schlansker MS, Rau BR () EPIC: Explicitly Parallel Instruction Computing. Computer ():– Fisher JA () Very long instruction word architectures and the ELI-. In: ISCA ’: Proceedings of the th Annual International Symposium on Computer Architecture, ACM, New York, pp – Beck GR, Yen DW, Anderson TL () The Cydra minisupercomputer: architecture and implementation. J Supercomput :– Mahlke SA, Chen WY, Hwu WW, Rau BR, Schlansker MS () Sentinel scheduling for VLIW and superscalar processors. In: Proceedings of the th International Conference on Architectural Support for Programming Languages and Operating Systems, pp –, October Ebcioglu K () Some design ideas for a VLIW architecture for sequential-natured software. In: Cosnard M, Barton MH, Vanneschi M (eds) Parallel Processing (Proc. IFIP WG . Working Conference on Parallel Processing, Pisa, Italy), pp –, North Holland,
developed, mainly by W. Händler. Such a system is a collection of nodes where each node consists of one processor and one shared memory unit at least. The nodes are organized in several hierarchically ordered planes (layers) and one top-node. Each plane is a collection of m identical nodes arranged in an m × m grid, where each node is connected to its four nearest neighbors via shared memory. At the border of the plane, the nodes are interconnected in such a way that the layer forms a torus. Each upper plane is smaller than the layer immediately below. It has a quarter of the nodes of the lower plane. Each node, which is not part of the lowest plane, has access to the shared memory of four child nodes of the layer immediately below. The base layer, called working plane, is that part of the system where most of the real work is to be done. The upper layers and the top-node feed the working plane with data and offer support functions. In larger systems, only some of the lower planes are realized. In this case, the top-node is connected to the highest realized plane via bus. A special EGPA system is MEMSY. In such a system, only the two lower planes and the top-node are realized. Furthermore, by adding a bus system, long distance communication via messages is possible. Consequently, several programming models, not only shared memory, are supported. From to , three such systems of the EGPA class have been realized: the Pilot Pyramid (+ nodes), the DIRMU Pyramid, and the MEMSY Prototype ( + + nodes both).
Discussion
Erlangen General Purpose Array (EGPA) Jens Volkert Johannes Kepler University Linz, Linz, Austria
Synonyms MEMSY
Definition During the s, the concept of the multiprocessor class Erlangen General Purpose Array (EGPA) was
Introduction In , W. Händler published first ideas [A] and proposed one year later together with F. Hofmann and H.J. Schneider a new subclass of multiprocessors []: the Erlangen General Purpose Array (EGPA, Erlangen is an university city near Nürnberg in Bavaria). (Many of the following text is taken from [, , ].) The essential features and design objectives of these MIMDcomputers are: . Homogeneity: There is only one identical type of element – the node often called PMM (processormemory-module). . Memory-coupling: Connection between nodes is via memory.
Erlangen General Purpose Array (EGPA)
. Restricted neighborhood: No node has access to the whole memory, and no memory block can be accessed by all nodes. . Hierarchy: The PMMs are arranged in several layers and these layers form a total order. . Regularity: The nodes of one layer are ordered like a grid. W. Händler explained these features as follows: The reasons for these demands are speed (memory coupling), programmability (regularity, homogeneity), control (hierarchy), technical limits (restricted neighborhood), and expandability (all). In the following years, several EGPA systems were built. The first one, the pilot pyramid, was assembled in and was used until . It consisted of four PMMs at the working plane level (A-processors) and one PMM at level (B-processor). From to , DIRMU (Distributed Reconfigurable Multiprocessor Kit []) was used for testing applications written for EGPA systems. DIRMU was a kit consisting of PMMs, based, where processor-ports of one PMM could be interconnected with memory-ports of other PMMs by cables. Using several of these PMMs, different topologies could be realized: rings, trees, and so on. Especially an EGPA pyramid with three layers ( + + PMMs) could be built. From to , a new EGPA system started running in Erlangen. It was called MEMSY (Modular Expandable Multiprocessor System). In this multiprocessor, three layers were realized. It consisted of nodes in the working plane and nodes in the next layer. These machines were used to demonstrate the feasibility and usefulness of EGPA systems. In the following, the principles of EGPA systems are discussed, and the pilot pyramid and MEMSY serve as examples.
Motivation EGPA systems are general purpose machines, but they were especially designed for numerical simulations. With the emergence of powerful computer systems, the interest in computational science was awakened. This computational approach in natural science and engineering sciences was and is very successful in context with fluid dynamics, condensed matter physics, plasma
E
physics, applied geophysics, and others. The corresponding models of physical phenomena are either continuum models or many-body models. Essential is that there is an inherent “locality” in these approaches. The designer of EGPA kept this fact in mind when they developed their machines according to the possibilities the computer technology offered at that time, the s.
Design Goals The EGPA architecture was defined with the following design goals in mind: Scalability The architecture should be scalable with no theoretical limit. The communication network should grow with the number of processing elements in order to accommodate the increased communication demands in larger systems. Portability Algorithms should run on EGPA systems of any size. Cost effectiveness As far as possible, off-the-shelf components should be used. Flexibility The architecture should be usable for a great variety of user problems. Efficiency The computing power of the system should be big enough to handle real problems which occur in scientific research.
System Topology An EGPA system consists of identical PMMs (nodes): one processing unit and one attached communication (multiport) memory form one PMM (Processor Memory Module). The processing element consists of one or more processors. Optionally, there can exist a coprocessor and/or a private memory only accessible by the processing unit itself. The PMMs are hierarchically arranged in several layers (planes). An example with four layers (A, B, C, D) is given in Fig. a. At the topmost layer, there is only a single PMM which serves as a front-end to the system. At each lower layer, each PMM is connected to exactly four neighbors at the same layer, e.g., each processor has access to the communication memories of its neighboring PMMs. Thus, at each plane, the hardware has a
E
E
Erlangen General Purpose Array (EGPA)
D
C
B
A
ProcessorMemory-Module (PMM) Symmetric multiport-memory connection Asymmetric multiport memory connection between PMMs of different hierarchical level I/O communication supported by I/Oprocessor
Child NE Child NW Child SW Child SE Neighbour E Neighbour N Neighbour W Neighbour S
1111... 1110... 1101... 1100... 1011... 1010... 1001... 1000...
Private memory
Shared Memory
0001... 0000...
Erlangen General Purpose Array (EGPA). Fig. (a) EGPA with PMMs in the working plane (b) address space of a node
grid-like structure. As in each row, the most left node is connected to the most right PMM, and in each column, the corresponding nodes are connected too, each layer forms a torus. (In Fig. a, for clarity, only two of these turn-around connections are drawn.) In addition to the horizontal accesses, each node, except the very lowest, has access to four PMMs at the next lower layer. Consequently, the address space of a processor looks like in Fig. b. Each access with an address lying in the address space part of a child or neighbor will be translated by hardware to a shared memory address of that child or neighbor and that location of the child or neighbor will be accessed (e.g., the address . . . causes an access to the cell . . . of the neighbor in the north). The vertical connections are equipped to broadcast data downward to all four of the lower PMMs simultaneously, say, to transmit code segments. On the other hand, though the lower nodes do not have memoryaccess to those at higher layers, they are able to interrupt their supervising PMM. These substructures, consisting of four nodes and their supervisor, are called elementary pyramid. Several of them are highlighted in Fig. a. Each elementary pyramid has an I/O processor with corresponding I/O devices, mainly disks. The I/O processor which has access to all of its elementary pyramid memories is controlled by the supervising PMM. The topmost PMM is connected to a network containing a software development system.
The overall arrangement of an EGPA system is pyramidal. Each pyramid can be extended downward to any size by adding new levels. Such an extension significantly increases the computing power of the system. An EGPA system is a distributed memory multiprocessor. Remote communication between nodes is done via chains of copies and not via messages as usual.
System Software Each node has its own local operating system that exactly renders those services the resources of which are available. Now, in case a process needs a system service, this service either will be rendered at once or the addressed operating system cannot perform it and now informs his superior processor, i.e., its operating system, of the order. In the pilot pyramid, this technique was realized as follows. For each A-processor, a shadow process existed in the B-processor. At the four A-processors, only a user interface to the operating system of the B-processor was implemented. If there was an order to the operating system of the B-processor, this interface communicated with its shadow process. Then this shadow process sent the request to the proper operating system which executed the desired function afterward. This approach can be extended to larger EGPA computer; since each such system is composed of elementary pyramids (see Fig. a). Each B-processor is
Erlangen General Purpose Array (EGPA)
part of an elementary pyramid one level higher and so on. Consequently, the technique of shadow processes could be used in upper elementary pyramid as well. The proposed user interface for programming such a multiprocessor was in addition to several tools for testing parallel programs the EGPA monitor. This monitor consists of an EGPA frame program and a pool of subroutines. If a parallel program is executed, the frame program runs first installing the necessary system processes and then starting the user routines. The subroutines of the monitor are tools for controlling user programs. They allow to distribute programs and data to the system, to control the processes and to execute coordinating tasks (see []). Two very important tools are EXECUTE (PROGNAME, PROC) (starts routine PROGNAME at the indicated processor – child or neighbor) and WAIT (PROGNAME, PROC) (wait until the routine has finished at the processor). The EGPA monitor was realized for the Pilot Pyramid. But the concept can be carried to larger systems. It supports only one parallel program at a time in contrast to MEMSOS (see later) used in the MEMSY pilot.
The Pilot Pyramid The pilot pyramid, equivalent to an elementary pyramid, was assembled in and was used until (see Fig. ). It consisted of four PMMs at the working plane level (A-processors) and PMM at level (B-processor). All nodes were commercially available computers AEG / which offered a multiport memory. The characteristics of these processors were internal processor cycles of , , and ns and a word length of bits. The access time to a word was – μs. Each A-processor had a memory of KB and the B-processor KB. For more details, see []. The operating system was an expansion of the original available operating system MARTOS. The EGPA monitor could be called from user programs written in FORTRAN or SL (a subset of ALGOL).
Application Programming with the EGPA Monitor EGPA was designed for mainly solving problems using data partitioning. The corresponding programming model is known as SPMD (Single Program Multi Data).
E
I/O I/O control
Memory
E Processor
Erlangen General Purpose Array (EGPA). Fig. Pilot Pyramid
Before implementing a parallel program, the user has to partition his problem domain in such a way that parallel work can be done as much as possible. Furthermore, he has to assign himself the data to the processing elements and, in contrast to a global shared memory systems, in EGPA, the user had to take into account that needed data should not only exist within the system but also in the right memory at the right time. The synchronization and the control of data were done by using the EGPA monitor. The allocation of data was effected by segments, and most of data transports happened under user control. There were two possibilities for synchronization: under system control using messages or under user control by special routines of the monitor. The latter was a factor thousand faster, but deadlocks were possible if the user made a mistake. Nevertheless, it was preferred by nearly all users and the gained speedups were much better. One very important area for using EGPA systems are partial differential equations. Under the assumption that the region of interest is a square (or a cube) represented by a grid, for numerical purposes, and a grid-point-wise relaxation method is performed in red black order, the partitioning is very simple. To each processor one part is assigned. Each processor handles its portion of the grid and needs only values produced by neighboring processors. In Fig. a, the access area of one processor is hatched.
E
Erlangen General Purpose Array (EGPA)
On an EGPA system, one iteration step performed at a processor p is as follows: . Relax red points which do not depend on points of any neighbor. . Wait until all neighbors have signaled that their values can be used, e.g., the western neighbor has set the variable “ready” to iteration number (p accesses this variable using the name “ready&western”). . Relax red points which depend on black points of neighbors. These are points at the border of p. . Relax black points. . Ready := Iteration number +; /∗ signal to neighbors own borders are ready for next iteration. For this simple case, speedups near the number of processors in the working layer were achieved. If it was possible to map a rectangle onto a non-square physical space (see, e.g., Fig. b), the results were equivalent. If the domain under investigation is not so optimal, the programmer had to search for other solutions. If, e.g., the domain is a circle, it is not possible to give all nodes the same load and consequently, the efficiency of the system is smaller. Sometimes it helps to divide the region into several blocks. Then the grid is so folded onto the multiprocessor that each processor operates on a part of several blocks and each block is calculated by several processors simultaneously (Fig. c). Thereby, the torus structure of the layer is necessary. In Fig. d, this is illustrated for a circle. Each square is distributed to all nodes of the working plane. In many cases, the application designer has to search for new algorithm to use the power of the system. Thereby, he has to take into account the interconnection structure of an EGPA system. Matrix multiplication (H = F.G) with m × m nodes in the working
a
b
c
plane shall serve as an example. At the first glance, the interconnection topology of an EGPA seems to be a serious restriction for calculation of any new element, since data stored in remote memories (in the sense of restricted neighborhoods) are needed and have to be transported using the communication memories. The following algorithm is based on an idea of Cannon []. At the beginning, all matrices are equally distributed over the lowest plan as well as possible and H is set to . Hij , Fij , and Gij are stored in node pij (see Fig. Start position for m = ). The start position is transformed by more or less rotation of the partial matrices Fij and Gij in rows respectively columns of the working plane until the data are distributed as depicted in Fig. step for m = . For example, p reads G and copies it into the communications memory of processor p , this data is read and written into its own shared memory by p . These data transports can be done in parallel by the hardware. Thereby, the toroidal connections are used. Step : Each processor pij multiplies the partial matrices Fik and Gkj stored at its memory at the beginning of step i and adds the resulting matrix to Hij . Subsequently, the processor writes the matrix Fik into the memory of the processor “west” of it and the matrix Gij into the memory of the processor “north” of it. After m steps, the result H is distributed on the working plane as shown in Fig. . Result: Parts of the program for demonstration purposes written in SL (subset of ALGOL). It is one SPMD program per layer: The same code runs at each processor of one layer. For that reason, relative names with the following syntax were used.
d
Erlangen General Purpose Array (EGPA). Fig. Data grid partitioning for pointwise relaxation method
Erlangen General Purpose Array (EGPA)
Start position:
Step 1:
Step 2:
F21 G21 F22 G22 F23 G23 F24 G24
F22 G21 F23 G32 F24 G43 F21 G14
F23 G31 F24 G42 F21 G13 F23 G24
F31 G31 F32 G32 F33 G33 F34 G34
F33 G31 F34 G42 F31 G13 F32 G24
F34 G41 F31 G12 F32 G23 F33 G34
F41 G41 F42 G42 F43 G43 F44 G44
F44 G41 F41 G12 F42 G23 F43 G34
F41 G11 F42 G22 F43 G33 F44 G44
Step 3:
Step 4:
Result: H11
H12
H13
H14
F24 G41 F21 G12 F22 G23 F23 G34
F21 G11 F22 G22 F23 G33 F24 G44
H21
H22
H23
H24
F31 G11 F32 G22 F33 G33 F34 G44
F32 G21 F33 G32 F34 G43 F31 G14
H31
H32
H33
H34
F42 G21 F43 G32 F44 G43 F41 G14
F43 G31 F43 G42 F41 G13 F42 G24
H41
H42
H43
H44
F11 G11 F12 G12 F13 G13 F14 G14
F13 G31 F14 G42 F11 G13 F12 G24
F11 G11 F12 G22 F13 G33 F14 G44
F14 G41 F11 G12 F12 G23 F13 G34
E
F12 G21 F13 G32 F14 G43 F11 G14
Erlangen General Purpose Array (EGPA). Fig. Matrix multiplication for m = = nodes: partial matrix distribution
indicators that the data needed for STEP i are already stored∗ / WHILE FSEM/= STEP OR GSEM/= STEP DO WAIT (T) OD;/∗ T for avoiding too much memory accesses, if coherent caches are used - not in the Pilot Pyramid, WAIT is not necessary ‘/ MULT (HMATRIX,FMATRIX,GMATRIX) /∗ local matrix multiplication followed by addition of the result to HMATRIX ∗ / FSEM := GSEM := -STEP/∗ indicator that the data are used and can be overwritten∗ / WHILE FSEM& WEST/= -STEP DO WAIT(T) OD; /∗ western neighbour has not finished∗ / COPY (FMATRIX,FMATRIX&WEST); /∗ subroutine which copies FMATRIX into FMATRIX of the western neighbour.∗ / FSEM&WEST := STEP + 1;/∗ indicator that data is transported ∗ / WHILE GSEM&NORTH/= -STEP DO WAIT (T) OD; COPY (GMATRIX, GMATRIX&NORTH); GSEM&NORTH := STEP + 1
Allocation of data at each A-processor: /∗ installation of a global (shared) segment with name “MATRIX” in its own shared memory∗ / &SEG (MATRIX,GLOBAL); /∗ consequently there exist segments “MATRIX” installed in the shared memory of all its neighbour. It needs only access to those segments of its western and northern neighbour. Therefore∗ / &SEG (MATRIX&WEST,GLOBAL); &SEG (MATRIX&NORTH,GLOBAL); /∗ declaration part∗ / GLOBAL (MATRIX) [] REAL FMATRIX, GMATRIX, HMATRIX, INT FSEM, GSEM; VALGLOB (MATRIX&WEST) REF [] REAL FMATRIX&WEST, REF INT FSEM&WEST; VALGLOB (MATRIX&NORTH) REF [] REAL GMATRIX&NORTH, REF INT GSEM&NORTH; OD
Essential part of the main program running at each A-processor: /∗ P x P is the number of A-processors∗ / P is the FSEM := GSEM := ; FOR STEP FROM 1 TO m DO /∗ FSEM = STEP and GSEM = STEP are
The codes for gaining the data distribution at the beginning of Step (see Fig. ) as well as for COPY and for MULT are not represented. These routines need the size of the matrices which depend on the position of
E
E
Erlangen General Purpose Array (EGPA)
C-plane
B-plane
A-plane
Memory sharing within a plane Memory sharing between planea Common bus
Erlangen General Purpose Array (EGPA). Fig. MEMSY prototype
the processor. Therefore, the EGPA monitor provides each processor with its position (column, row) in the working plane together with the value of m.
MEMSY One essential result of all investigations concerning EGPA was the following. For many kinds of computations, the main computing load was carried by the lowest layer. The higher levels were mainly used for distributing partial tasks and data and for the realization of communication mechanisms. The consequence was that the higher layers were left underutilized. For removing this problem, the first idea was to taper the pyramid so that each lower level in an elementary pyramid has or even , rather than merely four PMMs. Experiences with a test system showed that excessive tapering (more than nine nodes in the elementary pyramid) can lead to bottlenecks at the higher layers. During these investigations, the best solution was found: The plane B must be mighty enough (elementary pyramid with five nodes) that the upper levels can be removed if the topmost node is connected to the B-plane by a bus. Such a special EGPA system is called MEMSY.
A prototype with nodes in the A-plane and nodes in the B-plane had been realized. The theoretical peak performance was GMIPS or MFLOPS (double precision), the measured performance was up to MFLOPS [, ]. (Most of the following information presented in this chapter is taken from [].) The node structure of MEMSY differed from that one of the pilot pyramid. Each node consisted of a Motorola multiprocessor board MVEME ( MC, MHz clock rate) and additional hardware. It contained a local memory, an attached shared memory, and an I/O-bus with access to different devices. The programming model of MEMSY was defined as a set of library calls which could be called from C and C++. In contrast to the EGPA model, it was not only based on the shared memory paradigm of EGPA. Instead, the model defined a variety of different mechanisms for communication and coordination. From these mechanisms, the application programmer could pick those calls which were best suited for his particular problem. The use of the shared memory was based on the concept of “segments,” very much like the original shared memory mechanism provided by UNIX System V. A process which wants to share data with another process (possibly on another node) first has to create a shared memory segment of the needed size. To have the operating system, select the correct location for this memory segment; the process has to specify with which neighboring nodes this segment needs to be shared. After the segment has been created, other processes may map the same segment into their address space by the means of an “attach” operation. The segments may also be unmapped and destroyed dynamically. For coordination purposes, three mechanisms had been provided: short (two word) messages, semaphores, and spinlocks. There was a second message type for transports of high-volume and fast data transfer between any two processors of the system. To express parallelism within a node, the programmer had to create multiple processes on each processing element by a special variant of the system call “fork.” Parallelism between nodes was handled by a configuration description, which forced the application environment to start one initial process on each node that the application should run on. In this starting phase, the application was consistent with the SPMD
Erlangen General Purpose Array (EGPA)
model. Through the execution of fork-calls, a configuration could be established which is beyond this model. The operating system of the MEMSY prototype was MEMSOS, it was based on UNIX V/ Release of Motorola, which was adapted to the multiprocessor architecture of the processor board []. In MEMSOS, most of the users’ needs were supported by the implementation of the “application concept.” An application was defined as the set of all processes belonging to one single user program. A process, belonging to an application, could be identified by (a) a globally unique application id, (b) a task id which was assigned to each process (called task) of an application and which was locally unique for this application and processor node and (c) a globally unique node id which identified any node in the system. This concept allowed a very simple steering and controlling of applications. MEMSOS could distinguish different applications and was able to run several of them concurrently in “space sharing” as well as in “time sharing” style. Additionally, a very important new concept, called “gang scheduling,” had been implemented. The purpose of this was to assure the concurrent execution of tasks of one application on all nodes at the same time. This was achieved by creating a runqueue for each application on each allocated node. A global component, called applserv, retrieved information about the nodes and the applications started and running there. It established a ranking of applications which were to be scheduled.
Exemplary Applications In order to validate the concepts of EGPA, certain applications from different scientific areas were adapted to the programming models of the EGPA monitor and of MEMSOS. In some cases, it was necessary to develop new parallel algorithms. The corresponding programs were written a lot of years ago. The listings of them are no more available. Therefore, it is not possible to give an impression of the amount for writing such a program, e.g., in lines of code. Only the matrix multiplication (see page ) gives an idea: lines (an estimation of the not shown part is included). The authors of such parallel programs wrote in their papers mainly about the principles of the underlying parallel algorithms and the runtimes. In many cases, they only published the maximal speedups received by
E
examples which were well-dimensioned for the used system. This means the problem size is maximal and, e. g., in case of the relaxation, the region under investigation is a square and not a circle or something else (see Fig. d). In a few cases, the dependency of the speedup on the problem size is depicted (see Gauss-Seidel and Table ). Using the Pilot pyramid, the maximum speedup was . Here are some of the measured speedups. The example “text formatting” demonstrates that not all types of tasks will run well on an EGPA system. Linear algebra [, ]: Matrix inversion: Gauss-Jordan (.), Column Substitution (.) Matrix multiplication (.) Gauss-Seidel iteration (). Speedup/dimension of the dense matrix: /, ./, ./, ./, ./ Differential equations []: Relaxation (.) Image processing and graphics []: Topographical representation (.) Illumination of topographical model (.), Vectorizing a grey level matrix (.), distance transformation – Each processor is working on a fixed part of data (.), – Dynamic assignment of varying parts of data (.). Nonlinear programming []: Search for a minima of a multi-dimensional objective function (.) Graph theory Network flow with neighborhood aid (.) Text formatting [] (.) Erlangen General Purpose Array (EGPA). Table Results with processors in the working plane [] Poisson equation with (/), (./), Dirichlet conditions: (./) (speedup/number of cells in finest grid) Steady state Stokes equation with staggered grid: (speedup/number of cells in finest grid)
(./), (./), (./)
E
E
Erlangen General Purpose Array (EGPA)
In order to have a larger system, the DIRMU (Distributed Reconfigurable Multiprocessor based on with coprocessor ) was used for building an EGPA with three layers. Among other things, that multiprocessor was used to investigate multigrid methods which were established in the industry at that time. Though in coarser grids the work per processor is very small, the results were very good (see Table ). In this context, it is worth mentioning that on a system, which only realized a working plane consisting of DIRMU-PMMs, adaptive multigrid approaches were implemented with high efficiency []. On the MEMSY Prototype, several algorithms were realized with great success. The measured efficiency was mostly greater than %. This is true for all the abovementioned algorithms which were ported from pilot or DIRMU to MEMSY [], but also for the following real world applications []: ● ● ●
Calculation of the electronic structure of polymers Parallel molecular dynamic simulation Parallel processing of seismic data by wave analogy of the common depth point ● Mapping parallel program graphs into MEMSY by the use of a hierarchical parallel. Algorithm ● Parallel application for calculating one-/two- and ecp-electron integrals for polymers Only for sparse matrix algorithms, efficiencies near % were measured.
Outlook At the same time when the mentioned EGPA systems were realized in Japan, a very similar systems, named PAX [], has been built. Also in Japan, the System HAP represented a lot of the principles of EGPA systems []. In the s, large commercial multiprocessors came into the market. Distributed memory MIMDs were now available which consisted of much more processors than, e.g., the MEMSY prototype. Even global shared memory systems with cache coherence were larger with respect to the number of processors. Especially the latter multiprocessors were easier to program than EGPA systems with their restricted neighborhoods. Consequently, the research on EGPA machines was stopped even in Erlangen.
Related Entries Computational Sciences Distributed-Memory Multiprocessor Metrics SPMD Computational Model Topology Aware Task Mapping
Bibliographic Notes and Further Reading Further papers dealing with the EGPA pilot are [, , , , , , ]. Publications concerning MEMSY can be found on the page: www.informatik.uni-erlangen.de/ Projects/MEMSY/papers.html
Bibliography . Bode A, Fritsch F, Händler W, Henning W, Hofmann F, Volkert J () Multi-grid oriented computer architecture. ICPP, – . Cannon LE () A cellular computer to implement the Kalman filter algorithm. PhD thesis, Montana State University . Dal Cin M, Hohl W, Dalibor S, Eckert T, Grygier A, Hessenauer H, Hildebrand U, Hönig J, Hofmann F, Linster C.-U, Michel E, Pataricza A, Sieh V, Thiel T, Turowski S () Architecture and realization of the modular expandable multiprocessor system MEMSY. In: Proceedings of parallel systems fair of the th international parallel processing symposium (IPPS ‘), Cancun, pp – . Finnemann H, Brehm J, Michel E, Volkert J () Solution of the neutron diffussion equation through multigrid methods implemented on a memory-coupled -processor system. Parallel comput (–):– . Finnemann H, Volkert J () Parallel multigrid algorithms implemented on memory-coupled multiprocessors. Nucl Sci Eng :– . Fritsch G, Jens Volkert () Multiprocessor systems for large numerical applications. Parcella – . Fritsch G, Henning W, Hessenauer H, Klar R, Linster CU, Oelrich CW, Schlenk P, Volkert J () Distributed sharedmemory architecture MEMSY for high performance parallel computations. Comput Archit News ():– . Fritsch G, Müller H () Paralleliation of a minimation problem for multiprocessor system. In: CONPAR . Lecture Notes Computer Science, vol . Springer, Berlin/Heidelberg/New York, pp – . Geus L, Henning W, Vajtersic M, Volkert J () Matrix inversion algorithms for the a pyramidal multiprocessor system. Comput Artif Intell (Nr ):– . Gossmann M, Volkert J, Zischler H () Matrix image processing and graphics on EGPA. EGPA-Bericht, IMMD, Universität Erlangen-Nürnberg
Ethernet . Hä¨ndler W () A unified associative and von-Neumann processor EGPP and EGPP array. In: Sagamore Computer Conference, pp – . Händler W., Klar R. () Fitting processors to the needs of a general purpose array (EGPA). MICRO : – . Hä¨ndler W, Herzog U, Fridolin Hofmann, Hans Jü¨ rgen Schneider: Multiprozessoren fü¨ r breite Anwendungsbereiche: Erlangen General Purpose Array. ARCS :– . Händler W, Bode A, Fritsch G, Henning W, Volkert J () A tightly coupled and hierarchical multiprocessor architecture. Comput Phy Commun :– . Händler W, Fritsch G, Volkert J () Applications implemented on the erlangen general purpose array. In: Parcella’. AkademieVerlag, Berlin, pp – . Händler W, Herzog U, Hofmann F, Schneider HJ () A general purpose array with a broad spectrum of applications. In: Informatik Fachberichte vol . Springer, Berlin/Heidelberg/New York, pp – . Händler W, Maehle E, Wirl K () Dirmu multiprocessor configurations. In: Proceedings of the international conference on parallel processing, pp – . Henning W, Volkert J () Programming EGPA Systems. In: Proceedings of the th International conference on distributed computing systems, Denver, pp – . Henning W, Volkert J () Multi grid algorithms implemented on EGPA multiprocessor. ICPP, pp – . Hercksen U, Klar R, Kleinöder W () Hardwaremeasurements of storage access conflicts in the processor array EGPA. In: ISCA ’, Proceedings of the th annual symposium on computer architecture, ACM Press, New York, La Baule, pp – . Hofmann F, Dal Cin M, Grygier A, Hessenauer H, Hildebrand U, Linster CU, Thiel T, Turovski S () MEMSY – a modular expandable multiprocessor system. In: Proceedings of parallel computer architectures: theory, hardware, software, applications. Lecturer notes computer science, vol . Springer, London, pp – . Hoshino T, Kamimura T, Kageyama T, Takenouchi K, Abe H () Highly parallel processor array “PAX” for wide scientific applications. In: Proceedings of the international conference on parallel processing, IEEE Computer Society, Columbus, pp – . Linster CU, Stukenbrock W, Thiel T, Turowski S () Das Multiprozessorsystem MEMSY. In: Wedekind (Hrsg.): verteilte systeme. Wissenschaftsverlag, Mannheim, Leipzig Wien Zürich, pp – . Momoi S, Shamada S, Koboyashi M, Tshikawa T () Hierarchical array processor system (HAP). In: Proceedings of the conference on algorithms and hardware for parallel processing on COMPAR , pp – . Rathke M () Paralleliseren ordnungserhaltender Programmsysteme für hierarchische Multiprozessorsysteme. PhD thesis. Arbeitsberichte des IMMD, Universität Erlangen ()
E
. Rusnock K, Raynoha P () Adapting the Unix operating system to run on a tightly coupled multiprocessor system. VMEbus Syst ():– . Vajtersic M () Parallel Poisson and biharmonic solvers implemented on the EGPA multiprocessor. In: ICPP
ES Earth Simulator
Ethernet Miguel Sanchez Universidad Politécnica de Valencia, Valencia, Spain
Synonyms IEEE .; Interconnection network; Network architecture; Thick ethernet; Thin ethernet
Definition Ethernet is the name of the first commercially successful Local Area Network technology invented by Robert Metcalfe and David Boggs while working at Xerox’s Palo Alto Research Center (PARC) in . While the prototype developed worked at . Mbps, the commercial successor was called Ethernet and worked at Mbps. Several faster versions have been developed and marketed since then. Ethernet networks are based on the idea of a shared serial bus and the use of Carrier Sense Multiple Access with Collision Detection (CSMA/CD) technique for network peers to communicate. Ethernet media access is fully distributed. Ethernet evolution brings the use of switching instead of CSMA/CD and full-duplex links. At the same time, coaxial cable is first replaced by twisted pair and then by fiber optics.
Discussion Introduction The world of Computing has always been a scenario of fast changes where state of the practice is challenged on a daily basis. Most of the advances in our field have come from this attitude of challenging the existing wisdom.
E
E
Ethernet
Thanks to that, we moved from the mainframe to the mini, and from it to the Personal Computer and now to embedded systems. When we lived in a world of a few computers, networking seemed not to be in big demand, but as the number of computers in enterprise grew, a new need started to develop to make efficient use of companies’ computing resources. It was the dawn of the Local Area Network (LAN). In the early days of computing, most of the interconnection focused on adding dumb terminals to the existing mainframe computers. For that purpose, different variations of serial connections sufficed. However, as first minicomputers and then personal computers started to spread on the enterprise there was a need of another type of interconnection. In this new environment, the computing power started to move from a central location to be distributed to each desk. Together with that, the need of keeping systems interconnected became apparent, especially at those locations that have a significant number of mini or personal computers. Xerox Corporation played an important role in the development of both the personal computer and LANs. More specifically, Xerox founded Palo Alto Research Center (PARC) in to conduct independent research. Many current technologies can be traced back to PARC, from the personal computer to the Graphical User Interface to Ethernet LANs. Xerox Alto was the first personal computer. It was developed in (Fig. ). The advent of Ethernet as a successful wired networking technology is based on a wireless precursor
Ethernet. Fig. Original Ethernet drawing by Robert Metcalfe
developed at the University of Hawaii and first deployed in by Normal Abramson and others. They used amateur radios to create a computer network so teletypes at different locations could access a time-sharing system. It was called the ALOHA network. All terminals use a shared frequency to transmit to the host. Simultaneous transmissions destroy each other but if a transmission was successful then it was acknowledged from the host. A network technology was needed to link together Alto computers with printers and other devices. No existing technology was found suitable so a team at PARC was set to develop a new technology. That was the embryo of Ethernet.
Collision Sense Multiple Access with Collision Detection The basic principle of the ALOHA network that allowed a node to transmit at any time was simple but not very efficient. Harvard University student Robert Metcalfe sought to improve the basic idea, and so he did his PhD on a better system and joined PARC. The main improvement was that network nodes should check for an ongoing transmission before starting their own. The way to do this was to check whether a carrier signal was present or not. This technique is also called Listen Before Talk (LBT). This gave way to a technique called Carrier Sense Multiple Access (CSMA). CSMA could be improved further by adding another mechanism called Collision Detection (CD). It consisted of stopping an ongoing transmission as soon as a collision was detected. A collision happens when two or more network nodes transmit at the same time. The purpose of this second improvement was to reduce the duration of collisions. This newer version of the access method was called Carrier Sense Multiple Access with Collision Detection or CSMA/CD and it was the base for the Ethernet network []. A third improvement over ALOHA was what to do after an unsuccessful transmission (collision). While ALOHA called for a random (but short) timer, Metcalfe’s proposal was to set the retransmit timer as a random slot number chosen from a special set. Ethernet’s Binary Exponential Backoff algorithm calls for random numbers to be selected from a range that starts as {, } and doubles after each collision. The maximum range is { . . . }. This algorithm ensures that the
Ethernet
probability of a new collision after a collision is halved. While CSMA/CD works on continuous time, Binary Exponential Backoff works on slotted time, each slot being the time required to transmit bits. Robert Metcalfe was helped by Stanford student Dave Boggs and they created a working system by networking together several Alto computers.
Ethernet Topologies Original Ethernet design used a bus topology with a coaxial cable as the shared media for all the transmissions. Data transmission used a baseband signal using Manchester encoding. A bus was found as a suitable way to broadcast information to all the network nodes and no switching was needed. The network bus was terminated with a resistive load on each end to reduce signal reflection. Maximum segment length was up to m. Repeaters could be use to connect two or more segments together. No more than four consecutive repeaters were allowed. Maximum network diameter was , m (Fig. ). Coaxial cable, specially the thick cable, was difficult to use in an office setup due to its size and it was expensive. This topology had the main drawback of a bus being a single point of failure. In order to reduce the cost of the network some revisions of Ethernet called for a thinner cable. This so-called Thin Ethernet was still based on a bus topology but using a thinner cable would make it not only cheaper but also easier to install cable runs. Unfortunately, the new setup with thinner cable was not without its own troubles as the attachment of each station was made using so-called T-connectors that required the coaxial cable to be cut at each joint and required two connectors to be added. Both the number of connectors and the physical characteristic of the cable (RG-) limited the length of Thin Ethernet segments to m.
1
1
0
0
1
1
0
Ethernet. Fig. Manchester encoding as used on Ethernet
0
E
The availability of broadband coaxial cable on some premises for cable TV distribution enabled another variation for Ethernet, but now using broadband signals and these call for a change as transmission is here unidirectional. This broadband version of Ethernet was never very successful but it allowed a maximum distance of , m. In a quest for cheaper alternatives, coaxial cable was replaced by twisted pair and the bus topology gave way to a star topology. While the first attempt at this only worked at a fraction of the coaxial data rate, the same speed was achieved with a later version. The main advantage of using twisted pair was its wide availability in office setups as it was used for voice communications. Twisted pair was also cheaper and easier to install and to handle. But the use of star topology called for a central multi-port repeater or hub. Hub device was, in essence, a bus-in-a-box. However, the star topology meant that if a cable was severed only one network node was affected. In a bus topology, if the coaxial cable is severed the whole network stops working. The star topology proved to be much more robust than the bus counterpart but each cable run was limited to m giving a maximum distance of m for each segment. Support for fiber optic media came from the need for both longer distance links and electrically isolated links. The first use of fiber optic media in Ethernet networks was for the interconnection of repeaters. It later developed as an alternative media to coaxial or twisted pair. While more expensive than copper, fiber optic media allowed longer distances of up to , m. Ethernet fiber optic links were point to point and that meant a star topology. A passive-star over fiber was proposed but it was never implemented.
The First Ethernet, and the First Commercial Ethernet Success The first version of Ethernet was deployed in PARC in and it ran at . Mbps and used -bit values for source and destination addresses. The first bits of the data field were reserved for a type identifier, so different protocols could be used over the same network each one using a different value for the type identifier. Data frames were protected against errors by means of a Cyclic Redundancy Check of bits placed at the end of the frame.
E
E
Ethernet
The original name for this network was the Aloha Alto Network, as it was developed for interconnecting Alto computers together and with laser printers. One of the original prototype boards can be found now at the Smithsonian. However, it was soon recognized that such a name suggested it would only work with Alto computers. The Ethernet name was a vendor-neutral name that was based on an old concept of wireless transmissions. During nineteenth century, it was believed that wireless signals propagate using an invisible media called ether, although physicists Michelson and Morley rejected that misconception in .
IEEE . also developed several choices for physical media and used a naming convention that prepended the data rate to a word describing the modulation scheme (baseband or broadband) and next the physical media used. BASE described a Mbps version of Ethernet using Manchester encoding over RG-X coaxial cable. BASE-T described a similar network but over twisted-pair cable. Eventually twisted pair became the cabling of choice, mostly using two of the four pairs of Category and Category cable. While twisted-pair setups required the use of expensive active hubs, the savings for using cheaper cable and the improved reliability of the network compensated for the additional costs.
DEC, Intel and Xerox In , Xerox, DEC and Intel joined to develop a common specification for Ethernet products: The first commercially successful Ethernet network was born. The specification was published as the “blue book” []. The new Ethernet specification, also known as DIX (acronym based on the three companies names), called for Mbps data rate, -bit addresses, -bit type field, and -bit Cyclic Redundancy Check (CRC) field (Fig. ). The data field was of variable size of up to , bytes. It was a new and improved version of the first Ethernet and now it was set to become a viable commercial product. At this same time, the market for personal computers blossomed with many different models. While this first commercially available version of Ethernet sold well, there were some concerns about the potential lock-in by the three companies. The Institute of Electrical and Electronic Engineers (IEEE) came into play. Ethernet would be standardized within the networking group , who met for the first time on February . Ethernet became the IEEE . standard in []. The availability of the . standard eased the doubts about the maturity level of Ethernet networks. At the same time, many different companies started selling compatible products many of them intended for the — by then — new IBM PC computer.
Ethernet. Fig. Ethernet frame format
CSMA/CD Ethernet Modeling and Problems One of the early criticisms of Ethernet was the fact that media access is probabilistic (nondeterministic). That means transmission latency cannot be known beforehand. The deterministic behavior of other media access algorithms, like IEEE . (Token Ring) was considered superior, but average transmission latency is much lower on Ethernet than on a token-passing network (except under very heavy network loads). Ethernet (and CSMA/CD by extension) was found to exhibit what was called “capture effect,” that happens when two stations contending for the medium have a collision and both have more frames to transmit. The winner will transmit successfully and it will keep an edge on future transmissions against the station that lost the transmission slot after the collision. This effect causes a transient channel-access unfairness that in turn can interfere with higher layer protocols, like TCP, that can amplify the effect. A second cause of concern was the behavior of heavy loaded Ethernet networks [, ]. As traffic grows beyond a certain limit, the chances of a transmission attempt to end up in a collision grow too. Having an increased chance of a collision when traffic load is high introduces a positive feedback that can bring the network to a congestion state. But as Ethernet data rate goes up, so
Ethernet
does the overload limit. Again, the use of full-duplex Ethernet avoids this problem.
Next Stop: Mbps Ethernet Once the Mbps Ethernet IEEE standard was approved, the group was tasked to develop a Mbps version. This faster version was known as Fast Ethernet. Different standards allowed this faster speed over different types of physical media using both copper and fiber optic media. Twisted-pair cable is not very well suited for high speed communication so different modulation schemes were used over different types of twisted-pair cables to give way to several physical layer specifications: BASE-TX use BB MLT- coded signals over Category twisted-pair cable while BASE-T uses BT OAM- coded signals over four twisted pairs of Category . Similarly, BASE-FX is Fast Ethernet media access for multimode fiber. Depending on whether half-duplex or full-duplex links are used, m or km is the maximum link distance respectively for fiber media.
Switched Ethernet Ethernet evolved not only becoming faster but also adding new components to the original architecture. A shared bus proved to be not enough for large networks. The maximum distance constraints together with traffic considerations opened the way to bridges. One way to increase the capacity of a network segment was to split it into two smaller segments, each one now having fewer computers competing for the available capacity. But these two segments need now to be interconnected so distributed applications can still work as before. The original bus topology gave way to a star topology using twisted-pair cable. Hubs provided the interconnection between multiple stations but there was a problem: All the stations had to work at the same data rate on the same collision domain. Bridges could be used to split one collision domain into several interconnected network segments where only intersegment traffic was shared. Therefore, both hubs and repeaters enabled the interconnection of two or more network segments. The term “hub” is used only for the interconnection of point-to-point links while repeaters could be used for interconnecting two coaxial-cable Ethernet segments.
E
From the logical point of view, both exhibit the same behavior working at the physical layer. A repeater retransmits a digital signal so it can reach a longer distance. A repeater does not create a separation of the collision domain. A bridge is an interconnection device that works at the Link Layer. Bridges connect two or more cable runs called network segments. Bridges understand MAC addresses and create several independent collision domains. Only traffic intended for other network segment is actually forwarded by a bridge. The aggregate bandwidth of a network interconnected by a bridge is higher than if the interconnection uses a repeater, but so is latency. The term “bridge” was usually associated with the interconnection of large and usually coax-based segments while the term “switch” referred mostly to interconnection devices using point-to-point links, where each port would have a workstation or server. From the logical point of view, bridges and switches are similar. Little by little, bridges were replaced by new devices called switches that offered similar functionality but many ports per unit. Switches started to replace network hubs. There was a performance increase but the cost was high too. The different topologies of Ethernet had to be loopfree. But given that bridges or switches were used to interconnect network segments there was a risk of creating network loops. On the other hand, having network loops is a way to introduce redundancy to the topology of a network, a feature that could be very useful if it did not interfere with Ethernet media access. The problem was that a new component was needed for Ethernet not to have problems in the presence of loops in the network topology or frames would be forwarded forever within the loops. The brilliant solution to the problem was the Spanning-Tree protocol [], invented by Radia Perlman while working at Digital Equipment Corporation and ratified as the IEEE .d standard. This protocol running on switches and bridges creates a spanning tree out of the existing network topology. Frames are only forwarded on network segments that belong to the spanning tree and, being a tree a loop-free topology, the problem of forwarding loops is solved. Whenever one of the links in use stops working, a new spanning
E
E
Ethernet
tree is calculated making use of the available redundant links. Bridges and switches exchange control frames periodically to maintain the spanning tree. A switch, like a bridge, works on Link Layer and creates an independent collision domain on each port. Switches are store-and-forward devices that can easily handle different data rates on different ports. Switches can handle simultaneous frame forwarding on different ports, an event that would have caused a collision should a hub were used. Different proprietary flow-control mechanisms were developed for half-duplex switches based on back pressure congestion management: A switch may transmit a carrier signal on those ports it wants to alleviate congestion. Ethernet switches may use different approaches to switching. The simpler way was to use the store-andforward approach where a frame is stored in a buffer to be retransmitted once it is received entirely by the switch. Cut-through switches only store the very first bytes of the frame and once the destination address is known the forwarding of the frame can start. This second approach leads to a significant reduction of the switching latency experienced by frames. An Ethernet network that uses switches instead of hubs enables a new improvement: full-duplex links. Provided that the physical media has two different circuits for simultaneous transmission and reception (e.g., two different copper pairs on BASE-TX), it is possible for a station to transmit and receive at the same time. While full-duplex Ethernet does work at the same data rate as half-duplex, higher throughput can be achieved with full-duplex. Besides, collisions do not occur in a full-duplex point-to-point link. A collision-free link is going to be more effective as the time devoted to collisions can now be put to use to transmit more data. However, network adapters still need to keep the CSMA/CD functionality for backward compatibility. Together with full-duplex operation came the .x flow-control mechanism. Sometimes, a receiver on a full-duplex link may not be able to handle frames fast enough. Ethernet flow-control is based on the receiver transmitting a special pause frame to a multicast MAC address. Such a frame reports the amount of time (using bit-time units) the sender on that full-duplex link must wait before the next transmission. If a receiver is
able to accept data sooner than expected, a new pause frame with zero wait-time can be transmitted.
Ethernet Gets Faster In May , the Gigabit Ethernet Alliance was founded by eleven companies interested in the development of Gigabit Ethernet (i.e., , Mbps Ethernet). The original aim of this new Ethernet version, faster than Fast Ethernet, was to make server and backbone connections even faster. Gigabit Ethernet was covered in IEEE .z and .ab standards and it was approved in . Gigabit Ethernet can use either multi-mode fiber, single-mode fiber, or twisted-pair copper cable. Gigabit Ethernet over cooper (known as BASE-T) poses a special challenge for twisted-pair connections that could use Category cable only if it meets some restrictions. This is the reason for Category E or Category copper cable. Different techniques were used for making possible to get gigabit speed over copper (e.g., QAM-modulation to more efficiently utilize the channel bandwidth). Many changes were needed beyond speed. Halfduplex operation was still allowed and so a technique called carrier extension was added to allow Ethernet to keep CSMA/CD and the ordinary m maximum network distance. An extension field is appended as needed to ensure a minimum frame length of bytes. Alternatively, another optional improvement was frame bursting: It allows the transmission of consecutive data frames after one MAC arbitration on half-duplex channels. In June , Gigabit Ethernet was ratified. With this new version of Ethernet the performance level is reaching or surpassing other Metropolitan Area Network (MAN) and Wide Area Network (WAN) technologies such as OC- and OC-, SONET or SDH technologies. This is the first version of Ethernet only supporting fiber optic media (GBASE-CX barely supports m of copper cable, just for attaching a switch to a router). A Gbps link range is between m and km. Both multi-mode and single-mode fibers are supported.
Ethernet and Beyond New versions of Ethernet are coming as Gbps Ethernet and Gpbs Ethernet. The IEEE .ba standard
Exaop Computing
has been ratified in June . A bit unusual was the fact that an interim Gbps version was developed largely as a step towards Gpbs Ethernet. The .ba calls for full-duplex fiber optic links. Data Center Ethernet (DCE) extends Ethernet architecture to improve networking and management in data centers. Several features like Priority-Based Flow-Control, Enhanced Transmission Selection (standardized as .Qaz), Multipathing or Congestion Management (.Qau) focus on improving network features and manageability. The vision of DEC is a unified fabric where LAN, storage area network (SAN), and inter-process communication traffic (IPC) converge. Technologies like Fiber Channel over Ethernet (FCoE) enable data centers to consolidate interconnection and internetworking technologies into a single network, which may simplify connectivity and reduce data center power consumption. Whether this is the last stop of Ethernet’s network technology or not, it is unclear. However, if past is an indication of what may be ahead, an even faster version of Ethernet should be expected in a few years time.
. Metcalfe RM, Boggs DR () Ethernet: distributed packet switching for local computer networks. Commun ACM ():– . The Ethernet () A local area network: data link layer and physical layer specifications (version .). Digital Equipment Corporation, Intel, Xerox . ANSI/IEEE Standard .– Carrier sense multiple access with collision detection. IEEE, October . Boggs DR, Mogul JC, Kent CA () Measured capacity of an Ethernet: myths and reality. In: Proceedings of ACM sigcomm ’ symposium of communications architecture and protocols, Stanford, CA, (Also as Computer Commun Rev (), August ) . Takagi H, Kleinrock L () Throughput analysis for persistent CSMA systems. IEEE Trans Commun COM-():– . Perlman R () Interconnections: bridges, routers, switches, and internetworking protocols (a ed. edición). Addison-Wesley, Reading, MA . Spurgeon CE () Ethernet—the definitive guide. O’Reilly & Associates, Inc, Sebastopol
Event Stream Processing Stream Programming Languages
Buses and Crossbars Distributed-Memory Multiprocessor Network Interfaces
Eventual Values Futures
Network of Workstations
Bibliographic Notes and Further Reading Access to many standards has been fee-based. IEEE standards started a move a few years ago by opening their documents to the public, for free. The documents of published standards are available for free download months after they are published. That means all details about any approved Ethernet standard version is easily available visiting http://standards.ieee.org/ getieee/..html Unfortunately, standards are usually written so all details are covered but many times they are not readerfriendly. For getting a good understanding of Ethernet technology, the Charles Spurgeon’s [] guide is a very appropriate text.
Bibliography
Related Entries Clusters
E
Exact Dependence Dependence Abstractions
Exaflop Computing Exascale Computing
Exaop Computing Exascale Computing
E
E
Exascale Computing
Exascale Computing Ravi Nair IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA
Synonyms Exaflop computing; Exaop computing
Definition Exascale computing refers to computing with systems that deliver performance in the range of (exa) operations per second.
Discussion Introduction In computing literature, it is customary to measure progress in factors of . The term exa, standing for or , is derived from the Greek word `´ε ξ, meaning six. Thus exascale computing refers to systems that execute between and operations per second. This performance level represents a -fold increase in capability over petascale computing, which itself represents a -fold increase over terascale computing. There is often confusion over how to denote the performance of a system. The most common measure is the number of double-precision floating-point operations that are completed every second (flops) when the system is running the Linpack dense linear algebra benchmark. By this measure, an exascale system is one which can execute Linpack at the rate of exaflops or higher. For a given system, this is usually less than the peak double-precision floating-point execution rate, typically by –%, because of practical limitations in exploiting the full floating-point capability in a system. In many systems that implement only single-precision floating point in hardware, the peak single-precision floating-point capability may be more than double the flops as measured for Linpack. Petascale computing was ushered in with the introduction of the IBM Roadrunner system in May , when it ran Linpack at . petaflops. If trends continue as in the past, it is reasonable to expect an exascale system to be operational around .
A list of the Top supercomputers in the world has been maintained since June and is a useful indicator of trends in the capabilities of the largest scientific machines. The list shows that there has indeed been a trend of doubling of Linpack performance every year, but that the manufacturer of the highest-performing system and the architecture of the system have been changing over time. While shared-memory computers and even uniprocessors made the list in earlier days, today the list is dominated by cluster supercomputers and massively parallel processors, the latter being distinguished from the former by their highly customized interconnection between nodes. With growing interest in the use of large-scale machines for applications that are considered commercial (or, more appropriately, nonscientific), the term exascale is also being used to refer to systems which can carry out operations per second ( exaops). There continues to be ambiguity in what constitutes an operation in this terminology. An operation is variously defined as an instruction, an arithmetic or logical operation, an operation intrinsic to the execution of an algorithm, and so on. This has led to one abstract definition of an exascale system to be one that has a performance capability roughly times more than a system solving the same problem in . In any case, the needs of the two worlds, scientific and commercial, appear to be converging especially from the points of view of compute capability, system footprint, energy requirements, and cost. It remains to be seen whether, in comparison to systems of , the first exascale system will execute only Linpack times faster, or whether it will solve a variety of problems, both scientific and commercial, faster by a factor of .
The Importance of Exascale Large supercomputers are already being used to solve important problems in a variety of areas. The International Exascale Software Project (IESP) has categorized the present areas of application of supercomputers as follows: ● ● ● ●
Material Science Energy Science Chemistry Earth Systems
Exascale Computing
● ● ● ● ●
Astrophysics and Astronomy Biology/Life Systems Health Sciences Nuclear and High Energy Physics Fluid Dynamics
Each of these areas has specific problems and a set of algorithms customized to solve those problems. However, there is an overlap in the use of algorithms across these domains, because of similarity in the basic nature of some of the problems across domains. Many of these problems could benefit from even larger supercomputers, but the expectation is that exascale systems will also bring a qualitative change to problem solving, allowing them to be used in new types of problems and in new areas. There are two ways in which larger supercomputers can help in the solution of existing problems. The first is called strong scaling, where the problem remains the same, but the solution takes less time. Thus, a weather calculation that forecasts the weather a week ahead is more useful if it takes h to solve rather than days. The other form of scaling is called weak scaling, where the size of the problem that can be solved is vastly increased with the help of exascale computers with its significantly larger resources. Thus the quality of a weather forecast can be improved if it is based on global data rather than local data, or if the forecast is made based on data from a larger number of more closely located sensors, or if the model is increased in complexity, for example, through the use of more types of meteorological information. Beyond just scaling of existing applications, largescale computing enables new types of computation. It becomes possible to model in a single application a physical phenomenon at different scales in time, space, or complexity simultaneously, for example, at the atomic, molecular, granular, component, or structural scale, or at the molecular, cellular, tissue, organ, or organism scale. This capability is already becoming evident at the petascale level and will be further exploited at the exascale level. Another new and interesting class of applications that will be further exploited in exascale computing is the quantification of uncertainty in mathematical models. These applications involve the use of massive parallel computing to understand the behavior
E
of physical systems in the presence of both systematic and random variations. The list of areas that could benefit from the use of supercomputers has been increasing steadily. Applications for supercomputing are also emerging in areas that were not traditionally considered scientific areas. Examples of these are visualization, finance, and optimization. Supercomputers have been used in the past by large movie production companies to render animated movies frame-by-frame over long periods of time. However, for typical scientific applications, the results produced by supercomputers were often packaged and shipped to a significantly smaller computer which the scientist used to analyze and visualize the results. The point where it will be too expensive to ship the vast amount of data produced by supercomputers is fast approaching. Increasingly, supercomputers will be used to process the results of their computation and produce visualization data to be shipped over to the scientist. The area of business analytics, traditionally considered a commercial application, is fast embracing the use of massive computers. The potential use of massive computation in such areas suggests that exascale computers are likely to represent the point at which supercomputers shift from being the largely simulationoriented scientific machines that they have traditionally been to more versatile machines solving fundamentally different types of problems for a larger section of humanity.
The Challenges of Exascale The principal engine for the advances made by supercomputers over the past decades has been Dennard scaling through which the electronics industry has enjoyed the perfect combination of increased density, better performance, and reduced energy per operation with successive improvements in silicon technology. When all dimensions of an electronic circuit on a chip, for example, the transistor gate width, the diffusion depth, and the oxide thickness, are reduced by a factor α, and when the voltage at the circuit is also reduced by a factor α, it is possible to increase the density of packing of devices on the chip by a factor of α and increase the frequency of operation of the device by a factor of α, with a reduction in energy per operation by a factor of α . With process technologists providing lithography improvements that
E
E
Exascale Computing
allowed a doubling of transistor density roughly every years, with microarchitects discovering new ways to expose parallelism at the instruction level, and with system architects engineering sophisticated techniques to combine multiple processors into ever large parallel systems, it was not difficult to maintain a steady doubling of supercomputer system performance every year at almost constant cost. Going forward from the current nm technology node, it will become increasingly difficult to keep up with Dennard scaling. With oxide thickness down to a few atoms, voltages cannot be scaled proportionally without serious compromise to the reliability and predictability of the operation of devices. It is no longer reasonable to expect the same frequency improvement for constant power density that was enjoyed in the past. At the same time, the nature of typical application programs makes the detection and exploitation of additional parallelism within a single thread of execution difficult. Thus, scaling the total number of threads in a system remains the main option to increase the performance of future large-scale systems. With current petascale systems already sporting hundreds of thousands of cores, the expected number of cores in an exascale system is likely to be in hundreds of millions. Due to various restrictions that will be enumerated below, most of this increase will have to come from using whatever lithography advances are possible to increase the density of cores on a chip. The task of achieving the goal of building an exascale system by is daunting. There are challenges in implementing the hardware, in designing the software environment, in programming the system, and in keeping the machine running reliably for long periods at a stretch. Listed below are brief descriptions of some of these challenges.
Power A quick glance at the Top list shows that the power ratings for the top two petascale systems are, respectively, MW/Petaflops for the Cray system and . MW/Petaflops for the IBM Roadrunner system. At this rate, an exascale system could consume as much as – MW of power, a quantity that is larger than the amount of power consumed by a medium-sized
US city. The cost of acquiring the largest supercomputer has remained largely constant over the years at around $– million. This is the cost of about – MW-years of energy. Thus it is unlikely that exascale installations will be willing to provide more than about – MW of power, a third of which can be expected to be lost in cooling and power distribution. There is thus a gap of x–x between the power efficiency of today’s machines and the desired efficiency of exascale machines. Traditional CMOS scaling would have bridged that gap in about four technology generations, but with the slowing down of CMOS scaling and with longer intervals between introductions of technology generations, it is imperative to look for other avenues to achieve the desired power efficiency. This gap must be bridged for all components of the system – the processor, the memory system, and the interconnection network. At the processor level, it is likely that exascale systems will include domain-specific accelerators because it is easier to build hardware that is power efficient in a specific domain rather than across a general range of applications. Vector units, which perform the same computation on multiple data simultaneously, could play an even bigger role in exascale systems precisely for this reason. Unlike what is seen in desktop and server processors today, processor designers will be forced to take a minimalist view in every part of the processor, paring overhead down to a minimum without significant impact on performance. Power-efficient optics technology will need to be deployed in interconnection networks if the total system bandwidth has to be scaled proportionally from what exists today. Memory system designs, both at the main memory level and at the cache level, will have to be reinvented with the above stringent power-efficiency requirements in mind.
Memory One of the trends seen in the top supercomputers is that the fraction of the total cost of the system and the fraction of the total system power going into memory has been steadily rising, even though the ratio of memory capacity to the computational performance (bytes/flops) has been decreasing. In the Top machines, this ratio has decreased from byte/flop to as low as . byte/flop in some cases. The result of this is that there is a greater pressure on applications
Exascale Computing
to adapt to the reduced amount of memory by carefully managing the locality of applications, and a pressure on processor designers to provide ever-growing cache capacities and deeper cache hierarchies, both of which add to the challenge of streamlining the processor design to achieve a low power-performance ratio for exascale systems. Another problem is the bandwidth at the memory interface. DRAM designs have been catering mainly to the commodity market and hence, while their capacities have been keeping up with Moore’s Law, their bandwidths have not kept up with increases in processor performance. Cost considerations have prevented supercomputer vendors from designing custom DRAM solutions. However, the rampant data explosion in all types of information processing is forcing even commodity memory vendors to rethink the designs of memory interfaces and modules. It is conceivable that such new designs along with new storage-class memory technologies and new packaging technologies will do a better job in meeting the needs of exascale systems.
Communication An important aspect of all large-scale scientific applications is the amount of time spent communicating between computation nodes. Applications impose a certain communication pattern which often reflects the locality of interactions of objects in the physical world. Thus the topologies of interconnection networks in large systems tend to be optimized for such communication patterns. Increasingly, however, the nature of algorithms and the types of problems being solved are imposing greater randomness in the pattern of communications and hence greater pressure on the available bandwidth. One measure used to characterize the behavior of systems for nonlocal communication patterns is the bisection bandwidth. This is the least bandwidth available between any two partitions of the system each having half the total number of computing nodes. Petascale systems for the High Productivity Computing Systems (HPCS) program are required to have a bisection bandwidth of at least . PB/s. But few organizations will either need or will be able to afford such a high bandwidth per flops ratio.
E
At the exascale level, costs will force the bandwidth per flops ratio to be even lower. The ExaScale Computing Study sponsored by DARPA estimated the bisection bandwidth requirements at the exascale level to be in a range between PB/s and EB/s. Such a high bandwidth will force a predominant use of optics technology across the system, not just between racks, but also at the board level and perhaps even at the chip level. Exascale computing would benefit from lower costs through commoditization of optics components and from further advances in optics technology, like silicon photonics.
Packaging As mentioned earlier, the power goal of – MW for an exascale system will impose challenging requirements in all aspects of the system. One such aspect is packaging. The number of racks in an exascale system cannot be allowed to increase dramatically from today because of limitations in the size and cost of installations. Thus each rack will have to accommodate more boards and each board will have to accommodate more chips than at present. Increased packaging density will increase power density at various levels of the system and will make testing, debugging, and servicing of the system more challenging. Two technologies that are maturing and that could help in this respect are D stacking of silicon dies and the use of silicon carriers. D stacking not only helps in providing a compact footprint but also reduces the latency of communication between circuits on different layers. With judicious use of through-silicon vias (TSVs), D technology could also provide greater bandwidth between sections of the processor design without having to resort to large, expensive dies. In comparison to the mm pitch of printed-circuit board technology, silicon carrier technology allows mounting of dies closer than um, wiring pitch of – um, and an I/O pitch of – um. The higher signaling rates, the bandwidth, and the higher I/O density promised by silicon carrier technology at lower power and in a compact footprint will go a long way toward meeting exascale goals. Supercomputers have always been at the forefront in packaging; exascale promises to be no different.
E
E
Exascale Computing
Reliability The number of components in a system has a direct bearing on the reliability of the system. Today’s petascale systems, even with their redundancy and errorcorrecting techniques at the cache and memory levels, have a mean time between failures (MTBF) of less than a month. A direct extrapolation for a -fold increase in the number of components brings this number down to a few minutes. The conventional method of handling failures is to take a checkpoint of the relevant state of an application at an interval well under the MTBF of the system. When a failure occurs, the system is interrupted, the state reverted back to a previously checkpointed state, and execution resumed. The maintenance of checkpoints requires the system to be periodically interrupted and typically involves copying of processor and memory state to some other part of memory or to storage. The overhead of this checkpointing process could impede forward progress of the application if it has to be performed too often. New techniques for detection of faults and for recovery from faults are needed for future exascale systems. While coding and redundancy techniques could be used to improve the MTBF of the system, they come at a cost – in area, in power, and often in performance. With the exascale goal already a significant challenge on all these fronts, there is a growing belief that reliability techniques can no longer be application agnostic, and that applications will need to guide when and how checkpointing needs to be done in future large systems. Usability The eventual success of exascale computing will be measured not by whether it can execute the Linpack benchmark at an exaflops rate but by how successfully it is embraced by the community and how fundamentally it transforms both science and commerce. The preceding discussion about the challenges of implementing an exascale system suggests that the role of software will be important in the effective exploitation of such a system. On the one hand, life for application developers will be made easier if the system supports today’s programming models without change, but on the other hand, software will have to work around the necessary restrictions in hardware due to reasons mentioned above. Perhaps the most effective way in which this can be accomplished is through widespread adoption of a
new exascale programming model. Neither MPI, the standard message-passing paradigm, nor OpenMP, the standard shared-memory paradigm, alone can exploit a system almost certainly likely to include a cluster of shared-memory SIMD processors. There is growing interest in a hybrid model, for example, one that uses OpenMP at the local chip level and MPI at higher levels. New programming languages classified as Partitioned Global Address Space (PGAS) languages like X, UPC, Co-Array Fortran, and Titanium provide the programmer the view of one large shared address space for the entire system, but still allow the programmer to specify partitions of the address space local to each thread. Various parts of the software stack will also be expected to incorporate features to tolerate the expected high rate of failures as discussed in the previous section. While this could be handled directly through the programming language, ease-of-use considerations are likely to force reliability to be taken care of either at the compiler level or in the runtime system, as in the Map-Reduce programming model. The role of compilers and runtime systems also becomes important if the system is heterogeneous like some of today’s petascale systems or if domainspecific accelerators and field-programmable gatearrays (FPGAs) become part of the system hardware. While a small number of proficient users, the so-called top-gun programmers, would enjoy programming with such a diverse array of components, the vast majority of users would prefer a compiler or the system to decide when and how to employ these components to deliver optimal performance for their programs. A new organization, the International Exascale Software Project (IESP) has been formed with the recognition that unless a coordinated global effort is begun now, the programming community will not be ready to utilize exascale systems effectively when they are deployed. The IESP has taken upon itself the task of outlining a roadmap for the development of systems software (including operating systems, runtime systems, I/O systems, systems management software), of development environments (including programming models, frameworks, compilers, libraries, and debugging tools), of algorithm support (including data management, analysis, and visualization), and also of algorithms for common applications.
Exascale Computing
Conclusion Exascale systems promise to usher in a new era of computing where massive supercomputers will be used not only for traditional simulations of large scientific problems but also to process the vast amount of data that is being produced especially from new types of ubiquitous devices. The success of such large systems, however, will be determined by how effectively they are utilized. Exascale systems can be expected to blaze new trails in power-efficient hardware design, in a system-wide approach to ensuring availability despite expected failures, and in new software approaches to maximize the utilization of the available computation power.
Related Entries Chapel (Cray Inc. HPCS Language) Checkpointing Clusters
E
The Top Web site [] gives a historical trend of the performance and capabilities of the highestperforming scientific computers. The limits of CMOS scaling and its effect on power consumption in future technologies are described well in []. An excellent overview of the architecture, power consumption, and economics of large datacenters may be found in []. There are several new technologies that promise to supplant or complement nonvolatile storage. A good description of storage-class memories may be found in []. There is recent interest also in the use of such memories as extensions to traditional DRAM main memory []. A comprehensive description of challenges in packaging and potential solutions can be found in []. The promise of D technology is discussed in []. An introduction to recent advances in silicon photonics appears in [].
Coarray Fortran Distributed-Memory Multiprocessor Fault Tolerance Hybrid Programming With SIMPLE LINPACK Benchmark Massive-Scale Analytics Metrics Networks, Direct Networks, Fault-Tolerant Networks, Multistage PGAS (Partitioned Global Address Space) Languages Power Wall Reconfigurable Computers TOP UPC
Bibliographic Notes and Further Reading The most comprehensive description of the challenges of exascale systems can be found in the report of a study [] under the sponsorship of the DARPA Information Processing Techniques Office. The software challenges are detailed in a similar study described in []. The International Exascale Software Project [] is also actively pursuing an agenda to ensure adequate software exploitation of future exascale systems. DARPA also sponsored a study on reliability aspects of exascale systems and the resulting whitepaper is available at [].
Bibliography . Barroso LA, Hölzle U () The datacenter as a computer: an introduction to the design of warehouse-scale machines. In: Hill M (ed) Synthesis lectures on computer architecture. Morgan and Claypool Publishers, San Rafael, CA. Available: http://www.morganclaypool.com/doi/abs/./ SEDVYCAC. Accessed Feb . Elnozahy EN (ed) () System resilience at extreme scale. Available: http://www.darpa.mil/ipto/personnel/docs/ExaScale_ Study_Resiliency.pdf. Accessed Feb . Frank D () Power-constrained CMOS scaling limits. IBM J Res Dev (/):– . Freitas RF, Wilcke WW () Storage-class memory: the next storage system technology. IBM J Res Dev (/):– . International Exascale Software Project () Main page. Available: http://www.exascale.org/iesp/Main_Page. Accessed Feb . Knickerbocker JU et al () Development of next-generation system-on-package (SOP) technology based on silicon carriers with fine-pitch chip interconnection. IBM J Res Dev (/): – . Kogge PM (ed) () ExaScale computing study: technology challenges in achieving exascale systems. Available: http://www. darpa.mil/ipto/personnel/docs/ExaScale_Study_Initial.pdf . Accessed Feb . Qureshi MK, Srinivasan V, Rivers JA () Scalable high performance main memory system using phase-change memory technology. In: Proceedings of ISCA-. Austin, TX, USA . Sarkar V (ed) () Exascale software study: software challenges in extreme scale systems. Available: http://www.darpa.mil/ ipto/personnel/docs/ExaScale_Study_Software.pdf. Accessed Feb
E
E
Execution Ordering
. Top Supercomputer Sites () Home page. Available: http:// top.org/. Accessed Feb . Topol AW et al () Three-dimensional integrated circuits. IBM J Res Dev (/):– . Tsybeskov L, Lockwood DJ, Ichikawa M () Silicon photonics: CMOS going optical. Proc IEEE ():–
Execution Ordering Scheduling Algorithms
Experimental Parallel Algorithmics Algorithm Engineering
Extensional Equivalences Behavioral Equivalences
F Fast Fourier Transform (FFT) FFT (Fast Fourier Transform)
Fast Multipole Method (FMM) N-Body Computational Methods
Fast Poisson Solvers Rapid Elliptic Solvers
Fat Tree Networks, Multistage
Fault Tolerance Hans P. Zima, Allen Nikora California Institute of Technology, Pasadena, CA, USA
Definition Fault tolerance denotes the capability of a system to adhere to its specification and deliver correct service in the presence of faults.
Discussion Introduction Modern society relies increasingly on highly complex computing systems across a wide spectrum of applications, ranging from commercial transactions to the control of transportation and communication systems,
the electrical power grid, nuclear reactors, and space missions. In many cases, human lives and huge material values depend on the well-functioning of these systems even under adverse conditions. Despite the successes in verification and validation technology over past decades, theoretical as well as practical studies convincingly demonstrate that large systems typically do contain design faults, even when subject to the strictest development disciplines. Moreover, even a perfectly designed system may be subject to external faults, such as radiation effects and operator errors. As a consequence, it is essential to provide methods that avoid system failure and maintain the functionality of a system, possibly with degraded performance, even in the case of faults. This is called fault tolerance. A fault is defined as a defect in a system that may cause an error during its operation. If an error affects the service to be provided by a system, a failure occurs. Fault-tolerant systems were built long before the advent of the digital computer, based on the use of replication, diversified design, and federation of equipment. In an article on Babbage’s difference engine published in , Dionysius Lardner wrote []:“The most certain and effectual check upon errors which arise in the process of computation is to cause the same computations to be made by separate and independent computers; and this check is rendered still more decisive if they make their computation by different methods.” Early fault-tolerant computers include NASA’s Self-Testing-and-Repairing (STAR) computer, developed for a -year mission to the outer planets in the s, and the computers onboard the Jet Propulsion Laboratory’s Voyager systems. Today, highly sophisticated fault-tolerant computing systems control the new generation of fly-by-wire aircraft, such as the Airbus and Boeing airliners, protecting against design as well as hardware faults. Perhaps the most widespread use of fault-tolerant computing has been in the area of commercial transactions systems, such as automatic teller machines and airline reservation systems.
David Padua (ed.), Encyclopedia of Parallel Computing, DOI ./----, © Springer Science+Business Media, LLC
F
Fault Tolerance
The focus of this article is on software methods for dealing with faults that may arise in hardware or software systems. Before entering into a discussion on fault tolerance, the broader concept of dependability is explored below.
system’s capability to prevent the unauthorized disclosure of information. Maintainability characterizes the ability of a system to undergo modifications, including repairs: one specific aspect is the time it takes to restore a failed system to normal operation.
Dependability
Threats A system is associated with a specification, which describes the service the system is intended to provide to its user (a human or another system). It consists of one or more interacting components. A threat is any fact or event that negatively affects the dependability of a system. Threats can be classified as faults, errors, or failures; their relationship is illustrated by the fault-error-failure chain [] shown in Fig. . A fault is a defect in a system. Faults can be dormant – e.g., incorrect program code that is not executed – and have no effect. When activated during system operation, a fault leads to an error, which is an illegal system state. A fault inside a component is called internal; an external fault is caused by a failure propagated from another component, or from outside the system. Errors may be propagated through a system, generating other errors. For example, a faulty assignment to a variable may result in an error characterized by an illegal value for that variable; the use of the variable for the control of a for-loop can lead to ill-defined iterations and other errors, such as illegal accesses to data sets and buffer overflows. A failure occurs if an error reaches the service interface of a system, resulting in system behavior that is inconsistent with its specification. The execution of a system can be modeled by a sequence of states, with state transitions being caused as the result of atomic actions. In a first approximation, the set of all system states can be partitioned into correct states and error states. By separating those error states that allow a recovery from those that represent system failure the set of all error states is further partitioned into tolerated error states and failure states (Fig. ). The transitions between these state categories can be described by classifying unambiguously each action as a correct, fault, or recovery action []:
Fault tolerance is one important aspect of a system’s dependability, a property that has been defined by the IFIP . Working Group on Dependable Computing and Fault Tolerance as the “trustworthiness of a computing system which allows reliance to be justifiably placed on the service it delivers.” Whereas fault tolerance deals with the problem of ensuring correct system service in the presence of faults, the notion of dependability includes also methods for removing faults during system design, for example, via program verification, or to prevent faults a priori, e.g., by imposing coding rules that outlaw common design faults.
Basic Attributes of Dependability The attributes of dependability specify a set of properties that can be used to assess how a system satisfies its overall requirements. A system’s dependability specification needs to include the requirements for each of the individual attributes in terms of the acceptable frequency and severity of service failures for specified classes of faults and the system’s operating environment. Short definitions for a typical list of attributes are provided below; a detailed discussion can be found in [, , ]. The reliability, R(t), of a system at a time, t, is specified as the probability that the system will provide correct service up to time t. For example, the reliability required for aircraft control systems has been specified as R(t) > − − for t = h (this is often informally characterized as “ nines”). The mean time to failure (MTTF) is the expected time that a system will operate correctly before failure. System availability at a time, t, is defined as the probability that the system can provide correct service at time t. The safety of a system is characterized by the absence of catastrophic consequences on its user and the environment it is operating in. More formally, it can be defined as the probability to fail in a controlled, or safe manner (see section “Controlled Failure”). Confidentiality is a
●
A correct action, executed in a correct state, results in a correct state
Fault Tolerance
F
Service interface
Propagation to service interface
Activation Fault
Error−1
Defect
Invalid state
Propagation
....
Error−n
Failure
F
Invalid state Violation of system SPEC
External fault (caused by external failure)
Service interface
Fault Tolerance. Fig. Threats: the fault-error-failure chain
● Failure states
Tolerated error states
Correct states Recovery action
Correct / fault action (unrecoverable)
Fault action Fault action (unrecoverable)
Correction of controlled failure
Fault Tolerance. Fig. State-space partitioning
●
A fault action, executed in a correct state, results in an error state (tolerated or failure)
A correct or fault action, executed in a tolerated error state, results in an error state (tolerated or failure) ● A recovery action, executed in a tolerated error state, results in a correct state. ● For a class of systems that are referred to as fail-controlled recovery from failure is possible using special protocols (see section “Controlled Failure”). A system is fault tolerant if it never enters a failure state: this means that errors may occur, but they never reach the service boundary of the system and always allow recovery to take place. The implementation of fault tolerance in general implies three steps: error detection, error analysis, and recovery. The main categories of faults are physical, design, and interaction faults. A physical fault manifests itself by malfunctioning of hardware, which is not the result of a design fault. Examples include a permanent fault caused by the breakdown of a processor core due to overheating, a transient fault caused by radiation and resulting in a bitflip in a register or memory, or a fault caused by a human operator.
F
Fault Tolerance
Design faults may originate in either hardware or software. An illegal condition in a while loop leading to infinite iteration or an assignment involving incompatible variable types are examples for software design errors. Interaction faults occur during system operation and are caused by the environment in which the system operates. They include illegal input data or operator errors as well as natural faults caused by radiation hitting the system. In addition to these main categories, faults can be characterized along a set of additional properties, often across the above classes. This includes the domain of a fault – hardware or software, its intent – accidental or malicious (e.g., faults caused by viruses, worms, or intrusion), or its persistence – hard or soft. A hard fault (such as a device breakdown) makes a component permanently unusable. In contrast, a soft fault is a temporary fault that can be transient or intermittent. A component affected by such a fault can be reused after the resulting error has been processed. For example, a bitflip in a register caused by a proton hitting a circuit is a transient fault referred to as a Single Event Upset (SEU). Reflecting the diversity of faults, the failures caused by them can be characterized by a broad range of nonorthogonal properties. These include their domain – value or timing, the capability for control, signalling – signalled or unsignalled failures, and their consequences, ranging from minor to catastrophic.
Fault Prevention Fault prevention addresses methods that prevent faults to being incorporated into a system. In the software domain, such methods include restrictive coding standards that avoid common programming faults, the use of object-oriented techniques such as modularization and information hiding, and the provision of high-level programming languages and abstractions. Firewalls are mechanisms for preventing malicious faults. An example for hardware fault prevention includes shielding against radiation-caused faults []. Fault Removal Despite the existence of fault tolerance methods, a software system’s dependability will tend to increase with
the number of faults identified and removed during its development. Fault removal refers to the set of techniques that eliminate faults during the design and development process. Verification and Validation (V&V) are important in this context: Verification refers to methods for the review, inspection, and test of systems, with the goal of establishing that they conform to their specification. Validation checks the specification in order to determine if it correctly expresses the needs of the system’s users. Verification methods can be classified as either dynamic or static, depending on whether or not they involve execution of the underlying program. Both techniques require that a system’s functional and behavioral specification be transformed into a mathematically based specification language, such as first-order logic []. System developers also need to specify a set of correctness properties, which must be satisfied. For example, a correctness property for a camera mounted on a robotic planetary exploration spacecraft would be that the camera’s lens cover must be closed if the camera is pointed within a given angle of the Sun. Static verification of a program implies a static proof – performed before execution of the program – that for all legal inputs the program conforms to its specification. Early techniques include Hoare’s logic and Dijkstra’s predicate transformers [, ]. Theorem provers use mathematical proof techniques to provide a rigorous argument that the correctness properties are satisfied. Some theorem provers, such as the early versions of the Boyer-Moore theorem prover [], are designed to operate in a completely automated fashion. Others, such as the Prototype Verification System (PVS) [], are interactive, requiring the user to “steer” the proof. Model checkers determine whether a system’s correctness properties can be violated by exploring its computational state space. If a violation is detected, the model checker returns a counterexample describing how the violation can occur. This also means that model checkers can be used to generate test cases for the implemented system []. Model checkers are able to identify types of faults associated with multithreaded systems such as deadlocks, lack of progress cycles, and race conditions. The more widely used model checkers include the SPIN model checker [], UPPAAL [], and SMV (Symbolic Model Verifier) [].
Fault Tolerance
Although static verification techniques can identify faults early in a system’s development, these techniques face a number of theoretical and practical challenges and limitations. Specifically, many decision problems are undecidable, that is, there is no general method that provides a complete solution []. Other problems, for which solution algorithms exists – including many graph problems, are NP-complete, that is, computationally intractable due to exponential complexity. Finally, methods such as model checking often need to deal with the scalability challenge, that is, an exploding state space. Furthermore, substantial effort can be required to create the formal specifications to which these tools will be applied. Learning the specification language can require substantial effort as well; even more important is learning the skill of abstracting from nonessential details of the system’s functionality and behavior. For these reasons, these techniques are usually applied only to the critical elements of a system (e.g., the fault detection, identification, and repair component of onboard spacecraft control software, or fault response systems for nuclear power plants). Dynamic verification technologies are based on the actual execution of the program. Most important are test methods. The test of a program for a specific input either demonstrates correctness of the program for this specific input, or results in an error. Thus, a specific test execution can either prove that the program operates correctly for the selected input, or it identifies an error. Thus, as already noticed by Edsger Dijkstra in , tests can prove or disprove the existence of a fault but never the absence of all faults [].
Controlled Failure Non-fault tolerant systems may be fail-controlled, meaning that they fail only in specific, predefined modes, and only to a manageable extent, avoiding complete disruption. Rather than providing the capability of resuming normal operation, a failure in such a system puts it into a state from which recovery is possible after the failure has been detected and identified. As an example, the onboard software controlling robotic planetary exploration spacecraft for those portions of a mission during which there is no critical
F
activity (such as detumbling the spacecraft after launch or descent to a planetary surface) can be organized as a fail-controlled system. When a fault is detected during operation, all active command sequences are terminated, components inessential for survival are powered off, and the spacecraft is positioned into a stable, sunpointed attitude. Critical information regarding its state is transmitted to ground controllers via an emergency link; rebooting and restoring the health of the spacecraft is then delegated to controllers on Earth. Systems that preserve continuity of service can be significantly more difficult to design and implement than fail-controlled systems. Not only is it necessary to determine that a fault has occurred, the software must be able to determine the effects of the fault on the system’s state, remove the effects of the fault, and then place the system into a state from which processing can proceed.
Fault Tolerance General Methods for Software Fault Tolerance Ensuring fault tolerance, that is, continuity of service in the presence of faults, requires replication of functionality. Many of the techniques used to achieve replication of functionality can be categorized as either design diversity or as self-checking software. Using design diversity to tolerate faults in the design of a software system requires that there be at least two variants of a system (i.e., different designs and/or implementations based on a common specification), a decider to provide an error-free result from the variants’ execution, and an explicit specification of decision points in common specification (i.e., specification of when decisions have to be performed, and specification of the data upon which decisions have to be performed). Two well-known types of design diversity are recovery blocks and N-version programming. Recovery Blocks
Recovery blocks [] make use of sequentially applied design diversity. In this case, the decider is an acceptance test that checks the computation results against context-specific criteria to determine whether the result can be accepted as correct. If the acceptance test decides
F
F
Fault Tolerance
that the result is correct, the system continues execution and does not attempt to apply any corrective action. On the other hand, if the acceptance test indicates that the result is incorrect, the result is discarded and a variant (termed an “alternate” for recovery blocks) is applied to the inputs to retry the computation. If all of the alternates have been attempted and no results have met the acceptance criteria, additional steps (such as exception handling, see section “Exception Handling”) are required to recover from the fault. A Petri-net model of a recovery block is shown in Fig. . N-Version Programming
For N-version programming [], design diversity is applied by executing all of the variants (termed versions) in parallel. The decider then votes on all of the results: if a majority of the results meet the decider’s criteria for correctness, the system is considered to be operating normally and continues execution. Otherwise, further action is required to recover from the fault. A Petri-net model for N-version programming is shown in Fig. . A variant of N-version programming, known as backto-back testing [], keeps only one of the N versions in the fielded system, but uses all of the versions during testing. If a defect is found in one of the N versions during test, the other versions are analyzed to determine
whether they also have that defect. The goal of back-toback testing is to improve the reliability of a fielded system without incurring the memory space or execution time penalties that might be associated with N-version programming. However, fielding only a single version removes the fault-tolerance capability associated with N-version programming. N-Self-Checking Programming
N-self-checking programming [] combines the idea of an acceptance test from recovery blocks with the parallel execution of N ≥ variants. In this case, each of the variants has an acceptance test associated with it. The acceptance tests associated to each variant or the comparison algorithms associated to a pair of variants can be the same, or specifically derived for each of them from a common specification. As the variants are executed, the acceptance test associated with each variant determines whether the result meets the criteria for correctness. From the set of correct results, one is selected as the collective output for all the variants and execution proceeds normally. If the set of correct results is empty, the system is considered to have failed and further action is required to restore it to nominal operation. A Petri-net model for N-self-checking software is shown in Fig. .
Start software execution
Primary alternate execution
End primary alternate execution
Alternate #i selected
No further alternates available
End alternate #i execution Acceptance test not passed
Execute acceptance test Alternate selection Acceptance test passed End software execution
Fault Tolerance. Fig. Petri-net model for recovery block
Execution alternate #i
Failed software
Fault Tolerance
F
Start software execution
Version 1 execution
Version 2 execution
Version N execution
End execution version 2
End execution version 1
End execution version N
Start decision algorithm execution N
Gather version results
F Acceptable result not provided
Acceptable result provided Failed software Execute decision algorithm
Fault Tolerance. Fig. Petri-net model for N-version programming Start software execution
Self-checking component 1 execution
Acceptable result provided
Self-checking component N execution
Self-checking component 2 execution
Gather, select acceptable result Result selected
No result selected
Acceptable result not provided
N Gather unacceptable results Failed software
Fault Tolerance. Fig. Petri-net model for self-checking programming
For self-checking software components that are based on the association of two variants, one variant only may be written for the purpose of fulfilling the functions expected from the component, while the other variant can be written as an extended acceptance
test. For example, it could perform the required computations with a different accuracy, compute the inverse function from the primary variant’s results (if possible), or exploit application-specific intermediate results of the primary variant. Algorithm-based fault tolerance
F
Fault Tolerance
(ABFT) belongs to this category []. Many of the faulttolerant systems encountered in real life are based on self-checking software. For example, the software for the Airbus A and A flight control systems and the Swedish railways’ interlocking system are based on parallel execution of two variants – in the latter case, the variants’ results are compared, and operation is stopped if the results of the variants disagree. It is important to note that the actual effectiveness of fault-tolerance techniques based on multiple versions may not be as high as the theoretical limits, because even independent programming teams may make similar errors []. However, recent work indicates that in spite of this limitation, N-version techniques can still be a viable way of tolerating faults [].
Introspection-Based Adaptive Fault Tolerance
Different components of a system may have different requirements for fault tolerance. For example, a space mission may, at one end of the spectrum, contain program components that perform low-level image processing with minimal fault-tolerance requirements (it does no harm if one in a few million pixels is lost), while at the other end the software for spacecraft control and communication needs maximum protection. Such
systems require support for adaptive fault tolerance, based on a fault model and a characterization of applications and their components with respect to their overall importance, exposure to faults, and requirements for fault tolerance. An introspection-based approach can support such an environment by providing a capability for execution-time monitoring and self-checking, analysis, and feedback-oriented recovery []. An introspection module, as illustrated in Fig. , is the core element of such a system, connected to the application via a set of software sensors for receiving input from, and a set of actuators for generating feedback to an application module. An inference engine connected to a knowledge base controls the operation of the introspection module. Introspection modules are organized into an introspection graph that reflects the organization and fault-tolerance requirements of the application. The application-specific error detection, analysis, and recovery actions of an introspection module can be controlled via assertions inserted in a program, directions for generation of specific checks, and corresponding recovery actions. This information can be provided by a user or automatically generated from the source program or a pre-existing knowledge basis characterizing properties of the application [, ]. A detailed analysis of the range of fault-tolerance requirements for a complex real-time environment is provided in [].
Sensors
. . .
Inference engine
Control links To / from introspection modules
Monitoring
Application segment(s)
Analysis
. . .
Feedback / recovery
Prognostics
Actuators
Fault Tolerance. Fig. Introspection module
. . .
Knowledge base
. . .
Fault Tolerance
Fault Tolerance in Concurrent Systems Safety and Liveness Properties
A concurrent system is introduced as an interconnected collection of a finite number of autonomous processes. Processes can either interact via a global shared memory, or communicate via message-passing over a network. No assumptions are made about individual and relative speeds of processes and delays involved in access to shared memory or message-passing communication, except for the requirement that the speed of active processes is finite and positive, and message delays are finite. (Thus, this model does not cover real-time constraints.) An execution of such a system can be modeled by a sequence of system states, with a transition between successive states taking place as a result of an atomic action in a process or a communication step. Lamport [] introduced two key types of correctness properties for parallel systems: safety properties and liveness properties. A safety property can be described by a predicate over the state that must hold for all executions of the system. It rules out that “bad things happen.” An example for a safety property is the requirement of an airline reservation system to provide mutual exclusion for accesses to any record representing a specific flight on a given day. A more mundane example is the requirement for a system controlling the traffic lights of a street intersection that the lights for two crossing streets may never be green at the same time. A liveness property requires that each process, which in principle is able to work, will be able to do so after a finite time. This property can be expressed via a predicate that must be eventually satisfied, guaranteeing that “a good thing will finally happen.” For instance, if a set of processes compete for a resource, each of the processes should be able to acquire it after a finite amount of time. Examples for violation of liveness include the prevention of a process to reach regular termination or to provide a specified service. Another example is a deadlock involving two ore more processes, which cyclically block each other indefinitely in an attempt to access common resources. Alpern and Schneider have shown that every possible property of the set of executions can be expressed as a conjunction of safety and liveness properties []. The manifestation of faults in a concurrent system, and related methods for their toleration depend largely
F
on detailed system properties and the programming model under consideration. In the next subsection, a number of typical problems in concurrent systems are shortly discussed; a comprehensive treatment of such issues can be found in []. In addition to the applicability of the general methods discussed in section “General Methods for Software Fault Tolerance”, fault tolerance can be often achieved using specialized techniques. Example Problems
Mutual exclusion constrains access to a shared resource – for example, a variable, record, file, or output device – such that at any time at most one process may access the resource (a safety property). Any correct solution must also guarantee that each process requesting the resource is granted access after a finite waiting period (a liveness property). A simple method for implementing mutual exclusion in a shared memory system is via a binary semaphore [] that controls access to the resource. The corruption of the semaphore value (by any process or as a result of a hardware fault) destroys the safety property of the algorithm. An error in the semaphore implementation may destroy the liveness property. The mutual exclusion problem is discussed in detail in []. State-based synchronization makes the progress of a process dependent on a condition in its environment. For example, in a simple version of the producerconsumer problem, the producer and consumer are autonomous processes operating asynchronously, with a cyclic buffer between them acting as a temporary storage for data items: in each cycle of the producer, it generates one data item and places it into the buffer; conversely, the consumer removes in each cycle one item from the buffer and then processes it. Wrong synchronization (as a result of a design error or a corruption of variables used for the coordination) can lead to a number of errors, including the corruption of a buffer element, an attempt of the producer to write into a full buffer, or an attempt of the consumer to read from an empty buffer. The result is a violation of the safety property, and possibly of liveness. A deadlock is a system state where two or more processes block each other indefinitely in an attempt to access a set of resources. In a typical example, process p holds resource r and requests resource r , while process p holds r and requests r . This is a violation of the liveness requirement. It can be addressed in a number
F
F
Fault Tolerance
of ways, including deadlock prevention –, by prioritizing resources and allowing access requests only in rising priority order, or by periodically checking for a deadlock and taking a recovery action if it occurs. A different example for a liveness violation is represented by the famous dining philosophers problem, where two processes conspire to block a third process indefinitely from accessing a common resource. Problems of this kind can be avoided by defining an upper limit for the number of unsuccessful resource requests, and prioritizing the blocked process once this limit is reached. In a system where processes communicate over a network, a network fault can propagate into a number of different faults in the process system. These include () transmission of wrong message content, () sending a message to the wrong destination, and, () total loss of the message, resulting in a possibly indefinite delay of the intended receiver. A way to deal with the last of these faults is the use of a watchdog timer that regulary checks the progress of processes. Byzantine faults are arbitrary value or timing faults that can occur as a result of radiation hitting a circuit, or of a malicious intruder. Under certain conditions, such faults can be tolerated using a sophisticated form of replication [].
Checkpointing and Rollback Recovery in Message-Passing Systems This section deals with fault tolerance for long-running applications in message-passing systems implemented on top of a reliable communication layer (which has to be provided by the underlying hardware and software layers of the network). The state of such systems consists of the set of local states of the participating processes, extended by the state of the communication system, which records the messages that are in-flight at any given time. Consistency of the global state implies that the dependences created between processes as a result of message transmissions must be observable in the state: more specifically, at the time a process records receipt of a message the state of the sender must reflect the previous sending of that message. A recovery approach based on checkpointing and rollback requires a stable storage device that is not affected by faults in the system. At a checkpoint, a process saves recovery information to this storage device during fault-free operation; after occurrence of an error
(e.g., as a result of a failing process), the information stored at the checkpoint can be used to restart the process. Recovery can be managed by the application; here a system-supported approach is discussed. There are two categories of recovery-based protocols. The first relies only on the use of checkpoints, whereas the second approach in addition performs logging of all nondeterministic events, which allows the deterministic regeneration of an execution after the occurrence of a fault (which is not possible if only checkpoints are used). This is important in applications that heavily communicate with the outside world (which cannot be rolled back!). However, from the viewpoint of today’s large-scale scientific applications, the first approach, which will just be denoted as checkpointing is more relevant, and will be the sole focus in the rest of this section. There are two main methods for performing checkpointing, which are characterized as uncoordinated or coordinated. In uncoordinated checkpointing, each process essentially decides autonomously when to take checkpoints. During recovery, a consistent global checkpoint needs to be found taking into account the dependences between the individual process checkpoints. Uncoordinated checkpointing has the advantage of allowing a process to take into account local knowledge, for example, for minimizing the amount of data that needs to be saved. However, there are serious disadvantages, including the domino effect that can result in rollback propagation all the way to the beginning of the computation and the need to record multiple checkpoints in each process, with the necessity of periodical cleanups via garbage collection. As a result of these shortcomings, the dominating method used in today’s systems, and particularly in supercomputers, is coordinated checkpointing. In this approach, a global synchronization step involving all processes is used to form a consistent global state. Only one checkpoint is needed in each process, and no garbage collection is required. A straightforward centralized two-stage protocol for the creation of a checkpoint requires blocking of all communication while the protocol executes. In a first step, a coordinator process broadcasts a quiesce message to all processes requesting them to stop. After receiving an acknowledgment for this request from all processes, the coordinator broadcasts a checkpoint message that requests processes to take their checkpoint and acknowledge completion. Finally,
Fault Tolerance
after receipt of this acknowledgment from all processes, the coordinator broadcasts a proceed message to all processes, allowing them to continue their work. On a failure in a process, the whole application rolls back to the last checkpoint and resumes execution from there. Coordinated checkpointing provides a relatively simple approach to recovery. Drawbacks include the overhead for communication in the checkpointing and recovery protocols, and the time required for storing the relevant data sets to a stable storage device, with the obvious consequences for scalability. In fact, a recent study on exascale computing [] shows that checkpointing along the above protocol seems no longer to be feasible, due to the existence of billions of threads in such a system. A potential solution would require new storage technology for saving the recovery information. A comprehensive treatment of rollback-recovery protocols, including log-based approaches, can be found in []. Coordinated checkpointing for largescale supercomputers is discussed in detail in [], including a study of scalability and an approach to deal with failures during the checkpointing and recovery phases.
Fault Injection Testing Fault injection testing refers to a technique for revealing defects in a system’s ability to tolerate faults []. Fault injection inserts faults into a system or mimicks their effects, combined with a strategy that specifies the type and number of faults to be injected into system components, the frequency at which faults are to be injected, and dependencies among fault types. A number of fault injectors have been developed [, , ], which allow faults to be inserted into a computing system’s memory or processor registers while the system is running, in a manner that is transparent to the executing application and system software. This allows developers of fault-tolerant systems to observe the system operating in an environment closely related to the expected environment during fielded use. As faults are injected into the system, system engineers are able to observe the immediate effects of each fault as well as the way in which it propagates through the system. When conducting fault injection testing, it is necessary to design test strategies that will cause the system under test to execute its fault-handling capabilities under a wide variety of conditions. One particular strategy, implemented by the Ballista automated testing
F
methodology [], is to change the parameters passed to software modules and observe the result. In Ballista, a test passes if the system does not hang or crash, ignoring the validity of output values produced. The JPL Implementation of a Fault Injector (JIFI) [] toolset is typical of modern software-controlled fault injectors. It allows automated campaigns of random uniformly distributed bitflip faults into application registers and/or memory space and registers. In order to characterize the fault injection results, it is necessary to verify the final result and classify its effect on the system with respect to its liveness properties and the functional correctness of the output. The main goal of this methodology is to characterize an application’s sensitivity to transient faults. For example, it is possible to determine whether the application is more likely to crash or produce incorrect results, or which subroutines of the program are most vulnerable.
Exception Handling An exception is a special event that signals the occurrence of one among a number of predetermined faults in a program. Exception handling can be considered as an additional fault-tolerance technique in that it provides a structured response to such events. There are two types of exception handling, resumption and termination []. In resumption, if an exception is encountered at a given command in a program, the program is temporarily halted and control is transferred to a handler associated with that program. The handler performs the computations required to mitigate the effects of the exception and then transfers control back to the program. The handler may return to the command immediately following the one at which the exception was encountered, or to a different location that may depend on the details of the program’s state when the exception was signalled. By contrast, termination simply leads to the (controlled) halting of the program in which the exception was observed. Although resumption has a potential advantage in that the interrupted program may be resumed upon successful execution of the handler, there are also advantages associated with choosing termination []. For example, with termination a programmer is encouraged to recover a consistent state of a module in which an exception is detected before signalling it, so that further calls to the module’s procedures find the module state consistent. With resumption, a programmer
F
F
Fault Tolerance
does not know if control will come back after signalling an exception. In addition, recovering a consistent state before signalling (e.g., for example by undoing all changes made since the procedure start) defeats the purpose of resumption, which is to save the work done so far between the procedure entry and the detection of the exception. If a consistent state is not recovered, the handler may never resume execution of the module after the signalling command, and the module will remain in the intermediate, most likely inconsistent, state that existed when the exception was detected, meaning that further module calls can lead to additional exceptions and failures.
Future Directions As computing systems become increasingly embedded in and critical to society, more effective capabilities to deal with faults induced by exceptional environmental conditions, specification and design defects, or both, will be required. As an example, consider implantable medical devices such as insulin pumps or defibrillators. These systems already execute safety-critical software; future versions of these devices intended to respond to a wider set of medical conditions will be substantially more complex than those in operation today. New challenges for fault tolerance will also come from the increased degree of automation in transportation systems (railways, airplanes, automobiles) as well as extreme-scale systems for large-scale simulation with billion-way parallelism []. Last but not least, future robotic space missions in our solar system and beyond will demand an unprecedented degree of autonomy when operating in environments in which direct communication with Earth will no longer be possible within practical time intervals. The additional complexity of these systems and the uncertainties of the environments in which they operate will introduce more opportunities for specification and design defects [], for which effective fault-tolerant techniques must be developed and deployed.
Related Entries Checkpointing Compilers Deadlocks Distributed-Memory Multiprocessor
Exascale Computing Race Detection Techniques Synchronization
Bibliographic Notes and Further Reading The basic concepts of dependability are discussed in detail in the seminal work of Avizienis, Laprie, Randell, and Landwehr [, ]. A comprehensive discussion of software fault tolerance can be found in the collection of articles edited by Lyu []. In [] dependability and fault tolerance are discussed with a focus on hardware issues. Faults and fault tolerance for algorithms in distributed systems, with a detailed treatment of provably solvable and unsolvable issues can be found in []. The premier conference on dependability is the annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), with DSN- held in Chicago in . This conference was established in by combining the IEEE-sponsored International Symposium on Fault-Tolerant Computing (FTCS), which had been held since , with the Working Conference on Dependable Computing for Critical Applications (DCCA) sponsored by IFIP WG .. Other conferences related to dependability include the International Symposium on Software Reliability Engineering (ISSRE) organized by the IEEE Computer Society and the Reliability, Availability, and Maintainability Symposium (RAMS) organized by the IEEE Reliability Society. In addition to many journals discussing fault tolerance in the context of more general or related areas such as software engineering, programming languages and systems, and system security, the IEEE Transactions on Dependable and Secure Computing (TDSC) have a strong focus on dependability and fault tolerance. The bibliography in [] provides a good overview of publications in the field until .
Bibliography . Alpern B, Schneider FB () Defining liveness. Inf Process Lett ():– . Amnell A, Behrmann G, Bengtsson J, D’Argenio PR, David A, Fehnker A, Hune T, Jeannet B, Larsen KG, MÄoller MO, Pettreson P, Weise C, Wang Yi () UPPAAL–now, next, and future. In: Proceedings of modelling and verification of parallel processes (MOVEP’k), June . LNCS tutorial , pp –
Fault Tolerance
. Avizienis A, Laprie JC, Randell B () Fundamental concepts of dependability, UCLA CSD Report No. , University of California, Los Angeles . Avizienis A, Laprie JC, Randell B, Landwehr C () Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing ():– . Avizienis AA () The methodology of N-version programming. In: Lyu MR (ed) Software fault tolerance. Wiley, Chichester . Boyer RS, Kaufmann M, Moore JS () The Boyer-Moore theorem prover and its interactive enhancement. Comput Math Appl ():– . Cai X, Lyu MR, Vouk MA () An experimental evaluation of reliability features of N-version programming. In: Proceedings of the IEEE international symposium on software reliability engineering, pp – . Cristian F () Exception handling and tolerance of software faults. In: Lyu MR (ed) Software fault tolerance. Wiley, New York, pp – . Crow J, Owre S, Rushby J, Shankar N, Srivas M () A tutorial introduction to PVS. http://www.csl.sri.com/papers/ wift-tutorial/ . Dijkstra EW () Co-operating sequential processes. In: Genuys F (ed) Programming languages. Academic, New York, pp – . Dijkstra EW () Notes on structured programming. In: Dahl OJ, Dijkstra EW, Hoare CAR (eds) Structured programming. Academic, London, pp – . Dijkstra EW () A discipline of programming, nd edn. Prentice Hall, Englewood Cliffs . Elnozahy ENM, Alvisi L, Wang Yi-Min, Johnson DB () A survey of rollback recovery protocols in message-passing systems. ACM Comput Surv ():– . Enderton HB () A mathematical introduction to logic, nd edn. Academic, San Diego . Kogge P et al () ExaScale computing study: technology challenges in achieving ExaScale systems. Technical report, DARPA information processing techniques O±ce (IPTO). http:// www.notur.no/news/inthenews/files/exascale_final_report., September . Gargantini A, Heitmeyer CL () Using model checking to generate tests from requirements specifications. In: Proceedings of the joint th european software engineering conference and the th ACM SIGSOFT symposium on the foundations of software engineering, Toulouse, France, September . Gärtner FC () Fundamentals of fault-tolerant distributed computing in asynchronous environments. ACM Comput Surv ():– . Goldberg D, Li M, Tao W, Tamir Y () The design and implementation of a fault-tolerant cluster manager. Technical report CSD-, Computer Science Department, University of California, Los Angeles, California, USA, October . Gunnels JA, Katz DS, Quintana-Ort ES, van de Geijn RA () Fault-tolerant high-performance matrix multiplication: theory and practice. In: Proceedings of international conference
. . .
.
.
.
. . .
.
. .
.
. .
. .
. .
.
F
on dependable systems and networks, (DSN-), Goeteborg, Sweden, June/July Hoare CAR () An axiomatic basis for computer programming. Commun ACM ():– Holzmann GJ () The SPIN model checker. Primer and reference manual. Addison-Wesley, New York Hopcroft JE, Motwani R, Ullman JD () Introduction to automata theory, languages, and computation, rd edn. Addison Wesley Higher Education, New York James M, Shapiro A, Springer P, Zima H () Adaptive fault tolerance for scalable cluster computing in space. IJHPCA (): – Kalbarczyk ZT, Iyer RK, Bagchi S, Whisnant K () Chameleon: a software infrastructure for adaptive fault tolerance. IEEE Trans Parallel Distrib Syst ():– Knight JC, Leveson NG () An experimental evaluation of the assumption of independence in multiversion programming. IEEE Trans Softw Eng ():– Koopman P () Ballista design and methodology, http://www. ece.cmu.edu/koopman/ballista/reports/desmthd.pdf Lamport L () Proving the correctness of multiprocess programs. IEEE Trans Softw Eng ():– Laprie JC, Arlat J, Beounes C, Kanoun K () Definition and analysis of hardware- and software-fault-tolerant architectures. Computer ():– Lardner D () Babbages’s calculating engine. Edinburgh Review, July . Reprinted. In: Morrison P, Morrison E (eds) Charles Babbage and his calculating engines. Dover, New York Lyu MR (ed) () Software fault tolerance, vol . Wiley, Chichester McCluskey EJ, Mitra S () Fault tolerance. In: Tucker AB (ed) Computer science handbook, nd edn. Chapman and Hall/CRC Press, Chapter , London, UK McMillan KL () Symbolic model checking – an approach to the state explosion problem. Ph.D. Dissertation, Carnegie Mellon University Nikora AP, Munson JC () Building high-quality software predictors. Softw Pract Exper ():– Randell B, Xu J () The evolution of the recovery block concept. In: Lyu MR (ed) Software fault tolerance. Wiley, Hoboken, New Jersey, USA, pp – Schneider FB () On concurrent programming. Springer, New York Shirvani PP () Fault-tolerant computing for radiation environments. Technical report -. Ph.D. Thesis, Center for Reliable Computing, Stanford University, Stanford, California , June Critical Software () csXCEPTION. http://www.criticalsoftware.com/products services/csxception/ Some RR, Agrawal A, Kim WS, Callum L, Khanoyan G, Shamilian A () Fault injection experiment results in space-borne parallel application programs. In: Proceedings IEEE aerospace conference, Big Sky, MT, USA, March Some RR, Kim WS, Khanoyan G, Callum L, Agrawal A, Beahan J () A software-implemented fault injection methodology for
F
F .
. .
.
.
Fences
design and validation of system fault tolerance. In: Proceedings of the international conference on dependable systems and networks (DSN), Goteborg, Sweden, pp –, July Stott DT, Floering B, Kalbarczyk Z, Iyer RK () A framework for assessing dependability in distributed systems with lightweight fault injectors. In: Proceedings of the th international computer performance and dependability symposium (IPDS’), IEEE Computer Society, Washington, DC, USA, pp – Gerard Tel () Introduction to distributed algorithms, nd edn. Cambridge University Press, Cambridge, UK Vouk MA () On back-to-back testing. In: Proceedings of the third annual conference on computer assurance (COMPASS’), pp – Wang L, Pattabiraman K, Kalbarczyk Z, Iyer RK () Modeling coordinated checkpointing for large-scale supercomputers. In: Proceedings of the international conference on dependable systems and networks (DSN’) Zima HP, Chapman BM () Supercompilers for parallel and vector computers. ACM Press Frontier Series, New York
Discussion Introduction The discrete Fourier transform (DFT) is a ubiquitous tool in science and engineering including in digital signal processing, communication, and highperformance computing. Applications include spectral analysis, image compression, interpolation, solving partial differential equations, and many other tasks. Given n real or complex inputs x , . . . , xn− , the DFT is defined as yk = ∑ ω kℓ n xℓ ,
Memory Models Synchronization
FFT (Fast Fourier Transform) Franz Franchetti , Markus Püschel Carnegie Mellon University, Pittsburgh, PA, USA ETH Zurich, Zurich, Switzerland
Synonyms Fast algorithm for the discrete Fourier transform (DFT)
Definition A fast Fourier transform (FFT) is an efficient algorithm to compute the discrete Fourier transform (DFT) of an input vector. Efficient means that the FFT computes the DFT of an n-element vector in O(n log n) operations in contrast to the O(n ) operations required for computing the DFT by definition. FFTs exist for any vector length n and for real and higher-dimensional data. Parallel FFTs have been developed since the advent of parallel computing.
()
√ − . Stacking the with ω n = exp(−πi/n), i = xℓ and yk into vectors x = (x , . . . , xn− )T and y = (y , . . . , yn− )T yields the equivalent form of a matrix– vector product: y = DFTn x,
Fences
≤ k < n,
≤ℓ
DFTn = [ω kℓ n ]≤k,ℓ
()
Computing the DFT by its definition () requires Θ(n ) many operations. The first fast Fourier transform algorithm (FFT) by Cooley and Tukey in reduced the runtime to O(n log(n)) for two-powers n and marked the advent of digital signal processing. (It was later discovered that this FFT had already been derived and used by Gauss in the nineteenth century but was largely forgotten since then [].) Since then, FFTs have been the topic of many publications and a wealth of different algorithms exist. This includes O(n log(n)) algorithms for any input size n, as well as numerous variants optimized for various computing platform and computation requirements. The by far most commonly used DFT is for two-power input sizes n, partly because these sizes permit the most efficient algorithms. The first FFT explicitly optimized for parallelism was the Pease FFT published in . Since then specialized FFT variants were developed with every new type of parallel computer. This includes FFTs for data flow machines, vector computers, shared and distributed memory multiprocessors, streaming and SIMD vector architectures, digital signal processing (DSP) processors, field-programmable gate arrays (FPGAs), and graphics processing units (GPUs). Just like Pease’s FFT, these parallel FFTs are mainly for two-powers n and are adaptations of the same fundamental algorithm to structurally match the target platform.
FFT (Fast Fourier Transform)
On contemporary sequential and parallel machines it has become very hard to obtain high-performance DFT implementations. Beyond the choice of a suitable FFT, many other implementation issues have to be addressed. Up to the s, there were many public FFT implementations and proprietary FFT libraries available. Due to the code complexity inherent to fast implementations and the fast advances in processor design, today only a few competitive open source and vendor FFT libraries are available in the parallel computing space.
FFTs: Representation Corresponding to the two different ways () and () of representing the DFT, FFTs are represented either as sequences of summations or as factorizations of the transform matrix DFTn . The latter representation is adopted in Van Loan’s seminal book [] on FFTs and used in the following. To explain this representation, assume as example that DFTn in () can be factored into four matrices DFTn = M M M M .
()
Then () can be computed in four steps as t = M x, u = M t, v = M u, y = M v. If the matrices Mi are sufficiently sparse (have many zero entries) the operations count compared to a direct computation is decreased and () is called an FFT. For example, DFT can be factorized as ⎡ ⎤ ⎤⎡ ⎤⎡ ⎤⎡ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎢ ⎥ ⎥⎢ ⎥⎢ ⎥⎢ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎥ ⎢ ⎥ ⎢ − ⎥ ⎢ ⎥, () DFT = ⎢ ⎢ ⎥ ⎥⎢ ⎥⎢ ⎥⎢ ⎢ − ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎥⎢ ⎥⎢ ⎥⎢ ⎢ ⎢ ⎢ ⎢ −⎥ i⎥ −⎥ ⎥ ⎣ ⎦ ⎦⎣ ⎦⎣ ⎦⎣ where omitted values are zero. This example also demonstrates why the matrix-vector multiplications in () are not performed using a generic sparse linear algebra library, but, since the Mi are known and fixed, by a specialized program. Conversely, every FFT can be written as in () (with varying numbers of factors). The matrices Mi in FFTs are not only sparse but also structured, as a glimpse on () illustrates. This structure can be efficiently expressed using a formalism based on matrix algebra and also clearly expresses the parallelism inherent to an FFT.
F
Matrix formalism and parallelism. The n × n identity matrix is denoted with In , and the butterfly matrix is a DFT of size : DFT = [
]. −
()
The Kronecker product of matrices A and B is defined as A ⊗ B = [ak,ℓ B],
for A = [ak,ℓ ].
It replaces every entry ak,ℓ of A by the matrix ak,ℓ B. Most important for FFTs are the cases where A or B is the identity matrix. As examples consider ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ − ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − ⎢ ⎥ ⎥, I ⊗ DFT = ⎢ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − ⎣ ⎦ ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ DFT ⊗I = ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣
− −
−
⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ −⎥ ⎦
with the corresponding dataflows shown in Fig. . Note that the dataflows are from right to left to match the order of computation in (). I ⊗ DFT clearly expresses block parallelism: four butterflies computing on contiguous subvectors; whereas DFT ⊗I expresses vector parallelism: four butterflies operating on interleaved subvectors which is the same as one vector butterfly operating on vectors of length four as emphasized in Fig. (b). More precisely, consider the code for DFT (i.e., y = DFT x): y[0] = x[0] + x[1]; y[1] = x[0] - x[1];
F
F
FFT (Fast Fourier Transform)
Then code for DFT ⊗I is obtained by replacing every scalar operation by a four-way vector operation: y[0:3] = x[0:3] + x[4:7]; y[4:7] = x[0:3] - x[4:7];
Here, x[a:b] denotes (Matlab or FORTRAN style) the subvector of x starting at a and ending at b. These examples illustrate how the tensor product captures parallelism. To summarize: block parallelism (n blocks): In ⊗ A,
()
vector parallelism (n-way): A ⊗ In ,
()
where A is any matrix. The stride permutation matrix Lmn m permutes the elements of the input vector as in + j ↦ jm + i, ≤ i < m, ≤ j < n. If the vector x is viewed as an n × m
y
a
x
y = (I4 ⊗ DFT2)x
y
b
x
y = (DFT2 ⊗ I4)x
FFT (Fast Fourier Transform). Fig. Dataflow (right to left) of a block parallel and its “dual” vector parallel construct (figure from [])
matrix, stored in row-major order, then Lmn m performs a transposition of this matrix. Further, if P is a permutation (matrix), then AP = P− AP is the conjugation of A with P. Cooley–Tukey FFT. The fundamental algorithm at the core of the most important parallel FFTs derived in the literature is the general-radix decimation-in-time Cooley–Tukey type FFT expressed as n DFTn = (DFTk ⊗Im )Tm (Ik ⊗ DFTm )Lnk ,
n is a diagonal matrix Here, k is called the radix and Tm containing the twiddle factors. The algorithm factors the DFT into four factors as in (), which shows the special case n = = × . Two of the four factors in () contain smaller DFTs; hence the algorithm is divideand-conquer and has to be applied recursively. At each step of the recursion the radix is a degree of freedom. For two-power sizes n = ℓ , () is sufficient to recurse up to n = , which is computed by definition (). Figure shows the special case = × as matrix factorization and as corresponding data-flow graph (again to be read from right to left). The smaller DFTs are represented as blocks with different shades of gray. A straightforward implementation of () suggests four steps corresponding to the four factors, where two steps call smaller DFTs. However, to improve locality, the initial permutation Lnk is usually not performed but interpreted as data access for the subsequent compun is fused with the tation, and the twiddle diagonal Tm
y
DFT4 ⊗ I4
T 16 4
I4 ⊗ DFT4
n = km. ()
x
L16 4
DFT16 =
a
Matrix factorization
b
Data-flow graph
FFT (Fast Fourier Transform). Fig. Cooley–Tukey FFT () for = × as matrix factorization and as (complex) data-flow graph (from right to left). Some lines are bold to emphasize the strided access (figure from [])
FFT (Fast Fourier Transform)
subsequent DFTs. This strategy is chosen, for example, in the library FFTW .x and the code can be sketched as follows: void dft(int n, complex *y, complex *x) { int k = choose_factor(n); // t1 = (I_k tensor DFT_m)L(n,k)*x for(int i=0; i < k; ++i) dft_iostride(m, k, 1, t1 + m*i, x + i); // y = (DFT_k tensor I_m)diag(d(j))*t1 for(int i=0; i < m; ++i) dft_scaled(k, m, precomp_d[i], y + i, t1 + i); } // DFT variants needed void dft_iostride(int n, int istride, int ostride, complex *y, complex *x); void dft_scaled(int n, int stride, complex *d, complex *y, complex *x);
The DFT variants needed for the smaller DFTs are implemented similarly based on (). There are many additional issues in implementing () to run fast on a nonparallel platform. The focus here is on mapping () to parallel platforms for two-power sizes n.
Parallel FFTs: Basic Idea The occurrence of tensor products in () shows that the algorithm has inherent block and vector parallelism as explained in () and (). However, depending on the platform and for efficient mapping, the algorithm should exhibit one or both forms of parallelism throughout the computation to the extent possible. To achieve this, () can be formally manipulated using well-known matrix identities shown in Table . The table makes clear that there is a virtually unlimited set of possible variants of (), which also explains the large set of publications on FFTs. These variants hardly differ in operations count but in structure, which is crucial for parallelization. The remainder of this entry introduces the most important parallel FFTs derived in the literature. All these FFTs can be derived from () using Table . The presentation is divided into iterative and recursive FFTs. Each FFT is visualized for size n = in a form similar to () (and again from right to left) to emphasize block and vector parallelism. In these visualizations, the twiddle factors are dropped since they do not affect the dataflow and hence pose no structural problem for parallelization.
F
FFT (Fast Fourier Transform). Table Formula identities to manipulate FFTs. A is n × n, and B and C are m × m. A⊺ is the transpose of A (BC)⊺
=
C ⊺ B⊺
(A ⊗ B)⊺
=
A⊺ ⊗ B⊺
Imn
=
Im ⊗ In
A⊗B
=
(A ⊗ Im )(In ⊗ B)
In ⊗ (BC)
=
(In ⊗ B)(In ⊗ C)
(BC) ⊗ In
=
(B ⊗ In )(C ⊗ In )
A⊗B
=
mn Lmn n (B ⊗ A)Lm
=
Lmn n
=
mn (Lkn n ⊗ Im )(Ik ⊗ Ln )
=
kn (Ik ⊗ Lmn m )(Lk ⊗ Im )
=
kmn Lkmn km Lkn
− (Lmn m ) Lkmn n Lkmn km Lkmn k
F
Iterative FFTs The historically first FFTs that were developed and adapted to parallel platforms are iterative FFTs. These algorithms implement the DFT as a sequence of nested loops (usually three). The simplest are radix-r forms (usually r = , , ), which require an FFT size of n = r ℓ ; more complicated mixed-radix radix variants always exist. They all factor DFTn into a product of ℓ matrices, each of which consists of a tensor product and twiddle factors. Iterative algorithms are obtained from () by recursive expansion, flattening the nested parentheses, and other identities in Table . The most important iterative FFTs are discussed next, starting with the standard version, which is not optimized for parallelism but included for completeness. Note that the exact form of the twiddle factors differs in these FFTs, even though they are denoted with the same symbol. Cooley–Tukey iterative FFT. The radix-r iterative decimation-in-time FFT ℓ−
ℓ
ℓ
DFTr ℓ = (∏(Iri ⊗ DFTr ⊗Ir ℓ−i− ) Dri ) Rrr ,
()
i=
is the prototypical FFT algorithm and shown in Fig. . ℓ Rrr is the radix-r digit reversal permutation and the ℓ diagonal Dri contains the twiddle factors in the ith stage. The radix- version is implemented by Numerical Recipes using a triple loop corresponding to the two tensor products (inner two loops) and the product (outer loop).
F
FFT (Fast Fourier Transform)
Stage 3
Stage 2
Stage1
Stage 0
Bit reversal
16 16 16 16 ((I1 ⊗ DFT2 ⊗ I8) D16 0 ) ((I2 ⊗ DFT2 ⊗ I4) D1 ) ((I4 ⊗ DFT2 ⊗ I2) D2 ) ((I8 ⊗ DFT2 ⊗ I1) D3 ) R2
FFT (Fast Fourier Transform). Fig. Iterative FFT () for n = and r =
Formal transposition of () yields the iterative decimation-in-frequency FFT: ℓ
ℓ−
ℓ
DFTrℓ = Rrr ∏ Dri (Ir ℓ−i− ⊗ DFTr ⊗Iri ) .
()
i=
Both () and () contain the bit reversal permutation ℓ Rrr . The parallel and vector structure of the occurring butterflies depends on the stage. Thus, even though every stage is data parallel, the algorithm is neither well suited for machines that require block parallelism nor vector parallelism. For this reason, very few parallel triple-loop implementations exist and compiler parallelization and vectorization tend not to succeed in producing any speedup when targeting the triple-loop algorithm. Pease FFT. A variant of () is the Pease FFT ℓ−
ℓ
ℓ
ℓ
DFTr ℓ = (∏ Lrr (Ir ℓ− ⊗ DFTr ) Dri ) Rrr ,
()
i=
shown in Fig. for r = . It has constant geometry, that is, the control flow is the same in each stage, and maximizes block parallelism by reducing the block sizes to r on which single butterflies are computed. However,
the Pease FFT also requires the digit reversal permutation. Each stage of the Pease algorithm consists of the twiddle diagonal and a parallel butterfly block, followed by the same data exchange across parallel blocks specified through a stride permutation. The Pease FFT was originally developed for parallel computers, and its regular structure makes it a good choice for fieldprogrammable gate arrays (FPGAs) or ASICs. Formal transposition of () yields a variant with the bit-reversal in the end. Korn–Lambiotte FFT. The Korn–Lambiotte FFT is given by ℓ
ℓ−
ℓ
ℓ
DFTrℓ = Rrr (∏ Lrr ℓ− Dri (DFTr ⊗Ir ℓ− )) ,
()
i=
and is the algorithm that is dual to the Pease FFT in the sense used in Fig. . Namely, it has also constant geometry, but maximizes vector parallelism as shown in Fig. for r = . Each stage contains one vector butterfly operating on vectors of length n/r, and a twiddle diagonal. As last step it performs the digit reversal permutation. The Korn–Lambiotte FFT was developed for early vector computers. It is derived from the Pease
FFT (Fast Fourier Transform)
Stage 3 Comm
Stage 2
Parallel
Comm
Stage 1
Parallel
Comm
Stage 0
Parallel
Comm
Parallel
F
Bit reversal communication
F
16 16 16 16 16 16 16 16 (L16 2 (I8 ⊗ DFT2) D0 ) (L2 (I8 ⊗ DFT2) D1 ) (L2 (I8 ⊗ DFT2) D2 ) (L2 (I8 ⊗ DFT2) D3 ) R2
FFT (Fast Fourier Transform). Fig. Pease FFT in () for n = and r =
Bit reversal Communication
Stage 3 Comm
Vbutterfly
Stage 2 Comm
Vbutterfly
Stage 1 Comm
Vbutterfly
Stage 0 Comm
Vbutterfly
16 16 16 16 16 16 16 16 R16 2 (L8 D0 (DFT2 ⊗ I8)) (L8 D1 (DFT2 ⊗ I8)) (L8 D2 (DFT2 ⊗ I8)) (L8 D3 (DFT2 ⊗ I8))
FFT (Fast Fourier Transform). Fig. Korn–Lambiotte FFT in () for n = and r = . Vbutterfly = vector butterfly
F
FFT (Fast Fourier Transform)
algorithm through formal transposition followed by the translation of the tensor product from a parallel into a vector form. Stockham FFT. The Stockham FFT DFTr ℓ =
ℓ−
rℓ r ℓ−i ∏(DFTr ⊗Ir ℓ− ) Di (Lr i=
⊗ Iri ) ,
()
is self-sorting, that is, it does not have a digit reversal permutation. It is shown in Fig. for r = . Like the Korn–Lambiotte FFT, it exhibits maximal vector parallelism but the permutations change across stages. Each of these permutations is a vector permutation, but the vector length increases by a factor of r in each stage (starting with ). Thus, for most stages a sufficiently long vector length is achieved. The Stockham FFT was originally developed for vector computers. Its structure is also suitable for graphics processors (GPUs), and indeed most current GPU FFT libraries are based on the Stage 3 Vbutterfly
VShuffle
Stage 2 Vbutterfly
VShuffle
Stockham FFT. The formal transposition of () is also called Stockham FFT.
Recursive FFT Algorithms The second class of Cooley–Tukey-based FFTs are recursive algorithms, which reduce a DFT of size n = km into k DFTs of size m and m DFTs of size k. The advantage of recursive FFTs is better locality and hence better performance on computers with deep memory hierarchies. They also can be used as kernels for iterative algorithms. For parallelism, recursive algorithms are derived, for example, to maximize the block size for multicore platforms, or to obtain vector parallelism for a fixed vector length for platforms with SIMD vector extensions. The most important recursive algorithms are discussed next. Recursive Cooley–Tukey FFT. The recursive, general-radix decimation-in-time Cooley–Tukey FFT was shown before in (). Typically, k is chosen to be Stage 1 Vbutterfly
VShuffle
Stage 0 Vbutterfly VShuffle
2 16 4 16 8 16 16 ((DFT2 ⊗ I8) D16 0 (L2 ⊗ I8)) ((DFT2 ⊗ I8) D1 (L2 ⊗ I4)) ((DFT2 ⊗ I8) D2 (L2 ⊗ I2)) ((DFT2 ⊗ I8) D3 (L2 ⊗ I1))
FFT (Fast Fourier Transform). Fig. Stockham FFT in () for n = and r = . Vbutterfly = vector butterfly, VShuffle = vector shuffle
FFT (Fast Fourier Transform)
Recursive FFT
F
Accumulated shuffles
F
8 4 4 8 16 (DFT2 ⊗ I8) T 16 8 (I2 ⊗ ((DFT2 ⊗ I4) T 4 (I2 ⊗ ((DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2) L2)) L2)) L2
FFT (Fast Fourier Transform). Fig. Recursive radix- decimation-in-time FFT for n =
small, with values up to . If () is applied to n = r ℓ recursively with k = r the algorithm is called radix-r decimation-in time FFT. As explained before, the initial permutation is usually not performed but propagated as data access into the smaller DFTs. For radix- the algorithm is shown in Fig. . Note that the dataflow is equal to Fig. , but the order of computation is different as emphasized by the shading. Formal transposition of () yields the recursive decimation-in-frequency FFT n (DFTk ⊗Im ), DFTn = Lnm (Ik ⊗ DFTm )Tm
n = km. ()
Recursive application of () and () eventually leads to prime sizes k and m, which are handled by a special prime-size FFT. For two-powers n the butterfly matrix DFT terminates the recursion. The implementation of () and () is more involved than the implementation of iterative algorithms, in particular in the mixed-radix case. The divide-andconquer nature of () and () makes them good choices for machines with memory hierarchies, as at some
recursion level the working set will be small enough to fit into a certain cache level, a property sometimes called cache oblivious. Both () and () contain both vector and parallel blocks and stride permutations. Thus, despite their inherent data parallelism, they are not ideal for either parallel or vector implementations. The following variants address this problem. Four-step FFT. The four-step FFT is given by n n DFTn = (DFTk ⊗Im )Tm Lk (DFTm ⊗Ik ),
n = km, ()
and shown in Fig. . It is built from two stages of vector FFTs, the twiddle diagonal and a transposition. Typi√ cally, k, m ≈ n is chosen (also called “square root decomposition”). Then, () results in the longest possible vector operations except for the stride permutation in the middle. The four-step FFT was originally developed for vector computers and the stride permutation (or transposition) was originally implemented explicitly while the smaller FFTs were expanded with some other FFT—typically iterative. The transposition can
F
FFT (Fast Fourier Transform)
Vector FFT
Shuffe
Vector FFT
16 4 4 (((DFT2 ⊗ I2) T 42 (I2 ⊗ DFT2) L42) ⊗ I4) L16 4 T 4 (((DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2) L2) ⊗ I4)
FFT (Fast Fourier Transform). Fig. Four-step FFT for n = and k = m =
be implemented efficiently using blocking techniques. Equation () can be a good choice on parallel machines that execute operations on long vectors well and on which the overhead of a transposition is not too high. Examples includes vector computers and machines with streaming memory access like GPUs. Six-step FFT. The six-step FFT is given by DFTn = Lnk (Im
n ⊗ DFTk )Lnm Tm (Ik
⊗ DFTm )Lnk ,
n = km, ()
and shown in Fig. . It is built from two stages of parallel butterfly blocks, the twiddle diagonal, and three global transpositions (all-to-all data exchanges). () was originally developed for distributed memory machines and out-of-core computation. Typically, √ n is chosen to maximize parallelism. The k, m ≈ transposition was originally implemented explicitly as all-to-all communication while the smaller FFTs were expanded with some other FFT algorithm—typically iterative. As in (), the required matrix transposition can be blocked for more efficient data movement. () can be a good choice on parallel machines that have
√ n =
multiple memory spaces and require explicit data movement, like message passing, off-loading to accelerators (GPUs and FPGAs), and out-of-core computation. Multicore FFT. The multicore FFT for a platform with p cores and cache block size μ is given by kp
((L p ⊗Im/pμ )⊗I μ )
DFTn = (Ip ⊗ (DFTk ⊗Im/p ))
n Tm
n/p
× (Ip ⊗ (Ik/p ⊗ DFTm )Lk/p ) pm
× ((Lp ⊗ Ik/pμ ) ⊗ I μ ) ,
n = km,
()
and is a version of () that is optimized for homogeneous multicore CPUs with memory hierarchies. An example is shown in Fig. . () follows the recursive FFT () closely but ensures that all data exchanges between cores and all memory accesses are performed with cache block granularity. For a multicore with cache block size μ and p cores, () is built solely from permutations that permute entire cache lines and p-way parallel compute blocks. This property allows for parallelization of small problem sizes across a moderate
FFT (Fast Fourier Transform) Communication
Parallel DFTs
Communication
Parallel DFTs
F
Communication
F
4 4 16 16 4 4 16 L16 4 (I4 ⊗ ((DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2) L2)) L4 T 4 (I4 ⊗ ((DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2) L2)) L4
FFT (Fast Fourier Transform). Fig. Six-step FFT for n = and k = m =
Block exchange
Parallel DFTs
Block exchange
√ n =
Parallel DFTs
Block exchange
4 8 8 (L84 ⊗ I2) (I2 ⊗ ((DFT2 ⊗ I2) T 42 (I2 ⊗ DFT2) L42) ⊗ I2) (L82 ⊗ I2) T 16 4 (I2 ⊗ (I2 ⊗ (DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2)) R 2) (L2 ⊗ I2)
FFT (Fast Fourier Transform). Fig. Multicore FFT for n = , k = m = , p = cores, and cache block size μ =
F
FFT (Fast Fourier Transform)
Vector FFTs
In-Register Shuffes
4 (((DFT2 ⊗ I2) T 42 (I2 ⊗ DFT2) L42) ⊗ I2) T 16 4 (I2 ⊗ (L 2
Vector FFTs
Vector Shuffe
⊗ I2) (I2 ⊗ L 42) (((DFT2 ⊗ I2) T 42 (I2 ⊗ DFT2) L42) ⊗ I2) (L 82 ⊗ I2)
FFT (Fast Fourier Transform). Fig. Short vector FFT in () for n = , k = m = , and (complex) vector length ν =
number of cores. Implementation of () on a cachebased system relies on the cache coherency protocol to transmit cache lines of length μ between cores and requires a global barrier. Implementation on a scratchpad-based system requires explicit sending and receiving of the data packets, and depending on the communication interface additional synchronization may be required. The smaller DFTs in () can be expanded, for example, with the short vector FFT (discussed next) to optimize for vector extensions. SIMD short vector FFT. For CPUs with SIMD ν-way vector extensions like SSE and AltiVec and a memory hierarchy, the short vector FFT is defined as n (Ik/ν DFTn = ((DFTk ⊗Im/ν ) ⊗ Iν ) Tm
×
(Im/ν ⊗ Lνν
n/ν )(DFTm ⊗Iν )) (Lk/ν
⊗ (Lm ν
ν-way vectorized is the stride permutations Lνν , which can be implemented efficiently using in-register shuffle instructions. () requires the support or implementation of complex vector arithmetic and packs ν complex elements into a machine vector register of width ν. A variant that vectorizes the real rather than the complex dataflow exists. Vector recursion. The vector recursion performs a locality optimization for deep memory hierarchies for the first stage (I k ⊗ DFTm )Lnk of (). Namely, in this stage DFTm is further expanded using again () with m = m m and the resulting expression is manipulated to yield m ) (Ik ⊗ DFTm )Lnk = (Ik ⊗ (DFTm ⊗Im )Tm
⊗ Iν )
⊗ Iν ) , n = km, ()
and can be implemented using solely vector arithmetic, aligned vector memory access, and a small number of vector shuffle operations. An example is shown in Fig. . All compute operations in () have complex ν-way vector parallelism. The only operation that is not
× (Lkm ⊗ Im ) (Im ⊗ (Ik ⊗ DFTm ) Lkm k k )
× (Lm m ⊗ Ik ) .
()
While the recursive FFT () ensures that the working set will eventually fit into any level of cache, large two-power FFTs induce large -power strides. For caches with lower associativity these strides result in a high number of conflict misses, which may impose
FFT (Fast Fourier Transform) Vector stage
Vector stage
Vector shuffle
Recursive FFTs
F
Vector shuffle
F
8 4 4 8 8 (DFT2 ⊗ I8) T 16 8 (I2 ⊗ (DFT2 ⊗ I4) T 4) (L2 ⊗ I4) (I2 ⊗ ((DFT2 ⊗ I2) T 2 (I2 ⊗ DFT2))) R 2) (L2 ⊗ I2)
FFT (Fast Fourier Transform). Fig. Vector recursive FFT for n = . The vector recursion is applied once and yields vector shuffles, two recursive FFTs, and two iterative vector stages
a severe performance penalty. For large enough twopower sizes, in the first stage of () every single load will result in a cache miss. The vector recursion alleviates this problem by replacing the stride permutation in () by stride permutations of vectors, at the expense of an extra pass through the working set. Since () matches (Ik ⊗ DFTn ) Lkn k , it is recursively applicable and will eventually produce child problems that fit into any cache level. The vector recursion produces algorithms that are a mix of iterative and recursive as shown in Fig. .
Other FFT Topics So far the discussion has focused on one-dimensional complex two-power size FFTs. Some extensions are mentioned next. General size recursive FFT algorithms. DFT algorithms fundamentally different from () include primefactor (n is a product of coprime factors), Rader (n is
prime), and Bluestein or Winograd (any n). In practice, these are mostly used for small sizes < , which then serve as building blocks for large composite sizes via (). The exception is Bluestein’s algorithm that is often used to compute large sizes with large prime factors or large prime numbers. DFT variants and other FFTs. In practice, several variants of the DFT in () are needed including forward/inverse, interleaved/split complex format, for complex/real input data, in-place/out-of-place, and others. Fortunately, most of these variants are close to the standard DFT in (), so fast code for the latter can be adapted. An exception is the DFT for real input data, which has its own class of FFTs. Multidimensional FFT algorithms. The Kronecker product naturally arises in D and D DFTs, which respectively can be written as DFTm×n = DFTm ⊗ DFTn , DFTk×m×n = DFTk ⊗ DFTm ⊗ DFTn .
() ()
F
FFT (Fast Fourier Transform)
For a D DFT, applying identities from Table to () yields the row-column algorithm DFTm×n = (DFTm ⊗In )(Im ⊗ DFTn ).
()
The D vector-radix algorithm can also be derived with identities from Table from (): I ⊗L rn r ⊗Is
DFTmn×rs = (DFTm×r ⊗Ins ) m
I ⊗L rn r ⊗Is
× (Imr ⊗ DFTn×s ) m
(Tnmn ⊗ Tsrs )
rs (Lmn m ⊗ Lr ) .
()
Higher-dimensional versions are derived similarly, and the associativity of ⊗ gives rise to more variants.
Related Entries ATLAS (Automatically Tuned Linear Algebra Soft-
ware) FFTW Spiral
Bibliographic Notes and Further Reading The original Cooley–Tukey FFT algorithm can be found in []. The Pease FFT in [] is the first FFT derived and represented using the Kronecker product formalism. The other parallel FFTs were derived in [] (Korn– Lambiotte FFT), [] (Stockham FFT), [] (four-step FFT), [] (six-step FFT). The vector-radix FFT algorithm can be found in [], the vector recursion in [], the short vector FFT in [], and the multicore FFT in []. A good overview on FFTs including the classical parallel variants is given in Van Loan’s book [] and the book by Tolimieri, An and Lu []; both are based on the formalism used here. Also excellent is Nussbaumer FFT book []. An overview on real FFTs can be found in []. At the point of writing the most important fast DFT libraries are FFTW by Frigo and Johnson [, ], Intel’s MKL and IPP, and IBM’s ESSL and PESSL. FFTE is currently used in the HPC Challenge as Global FFT benchmark reference implementation. Most CPU, GPU, and FPGA vendors maintain DFT libraries. Some historic DFT libraries like FFTPACK are still widely used. Numerical Recipes [] provides C code for an iterative radix- FFT implementation. BenchFFT provides up-to-date FFT benchmarks of about singlenode DFT libraries. The Spiral system is capable of generating parallel DFT libraries directly from the tensor product-based algorithm description [, ].
Bibliography . Bailey DH () FFTs in external or hierarchical memory. J. Supercomput :– . Cooley JW, Tukey JW () An algorithm for the machine calculation of complex Fourier series. Math Comput :– . Franchetti F, Püschel M () Short vector code generation for the discrete Fourier transform. In: Proceedings of the th International Symposium on Parallel and Distributed Processing, pp –, Washington, DC, USA . Franchetti F, Voronenko Y, Püschel M () FFT program generation for shared memory: SMP and multicore. In: Proceedings of the ACM/IEEE conference on Supercomputing, New York, NY, USA . Franchetti F, Püschel M, Voronenko Y, Chellappa S, Moura JMF () Discrete Fourier transform on multicore. IEEE Signal Proc Mag, special issue on “Signal Processing on Platforms with Multiple Cores” ():– . Frigo M, Johnson SG () FFTW: An adaptive software architecture for the FFT. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, vol , pp –, Seattle, WA . Frigo M, Johnson SG () The design and implementation of FFTW. Proc IEEE special issue on “Program Generation, Optimization, and Adaptation” ():– . Harris DB, McClellan JH, Chan DSK, Schuessler HW () Vector radix fast Fourier transform. In: Proc Inter Conf Acoustics, Speech, and Signal Processing. Conference Proceedings (ICASSP ’), pp –, Los Alamitos, IEEE Comput. Soc. Press . Heidemann MT, Johnson DH, Burrus CS () Gauss and the history of the fast fourier transform. Arch Hist Exact Sci :– . Korn DG, Lambiotte JJ, Jr. () Computing the fast Fourier transform on a vector computer. Math Comput (): – . Norton A, Silberger AJ () Parallelization and performance analysis of the Cooley-Tukey FFT algorithm for shared-memory architectures. IEEE Trans Comput ():– . Nussbaumer HJ () Fast Fourier transformation and convolution algorithms, nd ed. Springer, New York . Pease MC () An adaptation of the fast Fourier transform for parallel processing. J ACM ():– . Press WH, Flannery BP, Teukolsky SA, Vetterling WT () Numerical recipes in C: the art of scientific computing, nd ed. Cambridge University Press, Cambridge . Püschel M, Moura JMF, Johnson J, Padua D, Veloso M, Singer BW, Xiong J, Franchetti F, Gaˇci´c A, Voronenko Y, Chen K, Johnson RW, Rizzolo N () SPIRAL: Code generation for DSP transforms. Proc IEEE special issue on “Program Generation, Optimization, and Adaptation” ():– . Schwarztrauber PN () Multiprocessor FFTs. Parallel Comput :– . Tolimieri R, An M, Lu C () Algorithms for discrete Fourier transforms and convolution, nd ed. Springer, New York . Van Loan C () Computational framework of the fast Fourier transform. SIAM, Philadelphia, PA, USA
File Systems
. Voronenko Y, de Mesmay F, Püschel M () Computer generation of general size linear transform libraries. In: Proceedings of the th annual IEEE/ACM International Symposium on Code Generation and Optimization, pp –, Washington, DC, USA . Voronenko Y, Püschel M () Algebraic signal processing theory: Cooley-Tukey type algorithms for real DFTs. IEEE Trans Signal Proc ():–
F
File Systems Robert B. Ross Argonne National Laboratory, Argonne, IL, USA
Synonyms Cluster file systems
Fast Algorithm for the Discrete Fourier Transform (DFT) FFT (Fast Fourier Transform)
FFTW FFTW is a C library for computing the Discrete Fourier Transform (DFT). It was originally developed at MIT by Matteo Frigo and Steven G. Johnson. FFTW is an example of autotuning in two ways. First, the library is adaptive by allowing for different recursion strategies. The best one is chosen at the time of use by a feedback-driven search. Second, the recursion is terminated by small optimized kernels (called codelets) that are generated by a special purpose compiler. The library continues to be maintained by the developers and is widely used due to its excellent performance. The name FFTW stands for the somewhat whimsical “Fastest Fourier Transform in the West.”
Related Entries Autotuning FFT (Fast Fourier Transform) Spiral
Bibliography . Frigo M, Johnson SG () FFTW: An adaptive software architecture for the FFT. In: Proceedings IEEE international conference acoustics, speech, and signal processing (ICASSP), , pp – . Frigo M () A fast Fourier transform compiler. In Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI ‘). ACM, New York, pp – . Frigo M, Johnson SG () The design and implementation of FFTW. Proceedings of the IEEE ():–
F Definition A parallel file system (PFS) is a system software component that organizes many disks, servers, and network links to provide a file system name space that is accessible from many clients; distributes data across many devices to enable high aggregate bandwidth; and coordinates changes to file system data so that clients’ views of the data are kept coherent.
Discussion Introduction File systems are the traditional mechanism for reading and writing data on persistent storage. The current standard interface for file systems, part of the Portable Operating System Interface (POSIX) standard, was first defined in []. As networking of computers became common, sharing storage between multiple computers became desirable. In [], the Network File System (NFS) standard was created, allowing a file system stored locally on one computer to be accessed by other computers. With the advent of NFS, shared file name spaces were possible – many computers could see and access the same files at the same time. Figures a and b show a high-level view of a local and network file system, respectively. In the local file system case, a single software component manages local storage resources (typically one or more disks) and allows processes running on that computer to store and retrieve data organized as files and directories. In the network file system case, the software allowing access to storage is split from the software managing the storage. The file system client presents the traditional view of files and directories to processes on a computer, while the file system server software manages the storage and coordinates access from multiple clients.
F
File Systems
Application
Application
Application
File system client
File system client
File system software Interconnection network
a
File system server
Local file system
b
Networked file system
Parallel application
Parallel application
Parallel application
Parallel FS client
Parallel FS client
Parallel FS client
Interconnection network
Parallel FS server
c
Parallel FS server
Parallel file system
File Systems. Fig. Local, network, and parallel file systems
As distributed-memory parallel computers became popular, the utility of a shared file name space became obvious as a way for parallel programs to store and retrieve data without concern for the actual location of files and file data. However, accessing data stored on a single server quickly became a bottleneck, driving the implementation of parallel file systems – file systems that opened up many concurrent paths between computers and persistent storage. Figure c depicts a parallel file system. With multiple network links, servers, and local storage units, high degrees of concurrent access are possible. This enables very high aggregate I/O rates for the parallel file system as a whole when it is accessed by many processes on different nodes in the system, especially when those accesses are relatively evenly distributed across the storage servers. A wide variety of network configurations
and technologies are used in PFS deployments, from storage area networks and InfiniBand to Gigabit Ethernet and TCP/IP. This discussion focuses on parallel file system deployments where processes accessing the file system are on compute nodes distinct from the nodes that manage storage of data (storage nodes). This model is the most common in high-performance computing (HPC) deployments, because (a) there are typically many more compute nodes than needed to provide storage services, (b) placing disks in all compute nodes has an impact on their reliability and power consumption, (c) servicing I/O operations for some other process on a node executing an HPC application can perturb performance of that application, and (d) keeping storage nodes distinct enables higher-reliability components to be used and helps colocate storage resources in the machine room. That said, Beowulf-style commodity clusters and clusters used for Data Intensive Scalable Computing (DISC) applications are typically configured with local drives, and organizing those drives with a parallel file system can be an effective way to provide a low-cost, high-performance, cluster-wide storage resource.
Data Distribution A number of approaches are used for managing data distribution in parallel file systems. Figures a and b show a simple example file system name space and a distribution of that name space onto multiple storage nodes, respectively. In the name space, two HPC applications have generated three files. A fusion code has written a large checkpoint, while a DISC application is analyzing a pair of relatively small images from observations of the night time sky. The checkpoint has been split into a set of blocks distributed across multiple storage nodes, while the data for each of the smaller images resides on a single server. In addition to distributing the file data across storage nodes, in this example the file system metadata has also been distributed, as depicted by directory data residing on different storage nodes. File system metadata is the data that represents the directory tree as well as information such as the owners and permissions of files. File data is typically divided into fixed-size units called stripe units and distributed in a round-robin manner over a fixed set of servers. This approach is similar
File Systems
/disc
ckpoint43.h5
sky4325.img
B232
sky8792.img
B089
B756
B443 B781
a
Parallel file system name space
Application
Application
Application
Parallel FS client
Parallel FS client
Parallel FS client
Interconnection network
Parallel FS server
/disc B089
b
Parallel FS server
/pfs B756
Access Patterns
/pfs
/fusion
F
Parallel FS server
B443 /fusion
Parallel FS server
B781 B232
Distribution of data across parallel file system servers
File Systems. Fig. Data distribution in a parallel file system (PFS)
to the way in which data is distributed over disks in a disk array. How PFSes keep track of where data is stored will be covered in the section on Metadata Storage and Access. In a configuration such as the one shown here, the file system deployment is said to be symmetric, meaning that metadata and data are both distributed and stored on the same storage nodes. Not all parallel file systems operate this way, and those that keep metadata on separate servers are considered asymmetric. There are advantages to both approaches. The symmetric approach provides a high degree of concurrency for metadata operations, which can be important for applications that generate and process many files. On the other hand, storing metadata on a smaller set of nodes leads to greater locality of access for metadata and simplifies fault tolerance for the name space. Both approaches are commonly used, and some parallel file systems, such as GPFS [] and PVFS [], can be deployed with either configuration depending on workload expectations.
The designs of parallel file systems have been heavily influenced by the characteristics and access patterns of HPC applications. These applications execute at very large scale, with current systems allowing over , application processes to execute at once. Potentially all of those processes might initiate an I/O operation at any time, including all at once. Additionally, HPC applications are primarily scientific simulations CROSSREF. These codes operate on complex data models, such as regular or irregular grids, and these data models are reflected in the data that they store. The end result is that HPC access patterns are often highly concurrent and complex, including noncontiguity of access []. This often arises from access to subarrays of large global arrays. A primary use case for parallel file systems in HPC environments is for checkpointing. Checkpointing is the writing of application state to persistent storage for the purpose of enabling an application to restart at some point in the event of a system failure. This state is typically referred to as a “checkpoint.” If a node on which the application is running crashes, then the application can be restarted using the checkpoint to avoid having to recalculate early results. Typically an application will compute for some amount of time, and then all the processes will pause and write their checkpoint so that the checkpoint represents the same point in time across all the processes. Users are often surprised that their serial workloads are not faster when run on a parallel file system. The cost of coordination in the PFS in conjunction with different caching policies and the latency of communicating with file servers can result in performance no different from, or even worse than, that of local storage. Serial workloads are simply not the primary focus of PFS designs.
Consistency In the context of PFSes, consistency is about the guarantees that the file system makes with respect to concurrent access by processes, particularly processes on different compute nodes. The stronger these guarantees, the more coordination is generally necessary, inside the PFS, to ensure the guarantees.
F
F
File Systems
For example, most parallel file systems ensure that data written by one process is visible to all other processes immediately following the completion of the write operation. If no other copies of file data are kept in the system, then this is easy to enforce: the server owning the data simply always returns data from the most recent completed write. Some parallel file systems make a stronger guarantee, that of atomicity of writes. This means that other processes accessing the PFS will always see either all, or none, of a write operation. This guarantee requires coordination, because if a write spans multiple servers, then the servers need to agree on which data to return in reads if reading is allowed while a write is in progress. Alternatively, the PFS could force readers to wait when a write is ongoing. Many parallel file systems do guarantee atomicity of writes, and this approach of delaying reading when writes are ongoing is commonly used to provide this guarantee. These PFSes include a distributed locking component. Distributed locking systems in parallel file systems are typically single-writer, multireader, meaning that they allow many client nodes to simultaneously read a resource but permit only a single client node access when writing is in progress. Scalable parallel file systems perform locking on ranges of bytes in files, usually rounded to some block size. This allows concurrent writes to different ranges from different compute nodes. The use of distributed locking in parallel file systems has two significant impacts on designs that rely on them. First, the granularity of locking places a limit on concurrency and leads to false sharing during writes. False sharing occurs when two processes on different compute nodes attempt to write to different bytes of a file that happen to map to a single lock. While the two processes are not really performing shared access to the same bytes, the locking system behaves as if they were, serializing their writes. Given the nature of HPC access patterns, false sharing is common. Second, availability is impacted. If a compute node fails while holding a lock, access to the locked resource is halted until the lock can be reclaimed. Determining the difference between a busy compute node and a failed one can be difficult at large scale, especially in the presence of lightweight, non-preemptive kernels. For these reasons, some PFS designs eschew the use of distributed
locking, instead slightly relaxing consistency guarantees to a level that may be supported without locking infrastructure [].
Parallel File System Design Five major mechanisms shape a PFS design: ● ● ● ● ●
Underlying storage abstraction Metadata storage and access Caching Tolerance of failures Enforcement of consistency semantics
Consistency semantics have been discussed. The other four topics are covered next.
Underlying Storage Abstraction Some early parallel file systems relied on storage area networks (SANs) to provide shared access to physical drives. In that approach, data was accessed directly by block locations on drives or logical volumes. Eventually designers realized that interposing storage servers had some advantages, but this block access model was retained. IBM’s General Parallel File System (GPFS) uses this model, with software on the server side providing access to storage over an interconnection network such as Gigabit Ethernet. Figure a shows the division of functionality in such a parallel file system. On the client side, PFS software translates application I/O operations directly into block accesses on specific file servers and obtains permission to access those blocks. File servers receive block access requests and perform them on locally accessible storage. An important development that shaped more recent PFS designs was the concept of object storage devices. Object storage devices (OSDs) are storage devices (e.g., disks, disk arrays) that allow users to store data in logical containers called objects. By abstracting away block locations, file systems built using this abstraction are simpler to build, and blocks can be managed locally, at the device itself. Recently developed parallel file systems, such as Lustre [], PVFS [], and PanFS [], all use object-based access. Figure b shows how functionality is divided in such a design. Application accesses are translated into accesses to objects stored on file
File Systems
PFS client translates application I/O requests into accesses to data blocks stored on remote servers. Blocks are allocated from a global pool when needed.
File system server provides access to locally attached storage devices, using a block protocol similar to SCSI.
a
Parallel application
Parallel application
Block translation
Object translation
Block I/O interface
Object I/O interface
Interconnect
Interconnect
Object I/O manager
Block I/O manager
Block I/O manager
Block-based data access
b
F
PFS client translates application I/O requests into accesses to objects stored on remote servers.
F File system server provides access to locally attached storage devices through an object interface. Objects are mapped to local storage blocks on the
Object-based data access
File Systems. Fig. The two most common approaches to data storage in parallel file systems are block-based (left) and object-based (right)
servers. File servers translate these accesses into locations on locally accessible storage and manage allocation of space when needed. A less obvious advantage of the object-based approach is the possibility of algorithmic distribution of file data. This relates to file system metadata.
Metadata Storage and Access There are three major components of PFS metadata: directory contents, file and directory attributes, and file data locations. Directory contents are simply the listing of files and directories residing in a single directory. Typically the contents for a single directory are stored on a single server. This can become a bottleneck for workloads that generate a lot of files in a single directory, so some parallel file systems will split the contents of a single directory across multiple servers (e.g., GPFS []). File and directory attributes are pieces of information such as the owner, group, modification time, and permissions on a file or directory. Some of this information changes infrequently (e.g., owner), while
other information can change rapidly (e.g., modification time). Some parallel file systems keep all these attributes on a single server, while others distribute the attributes for different files to different servers to allow concurrent access. File data locations are the metadata necessary to map a logical portion of a file to where it is stored in the file system. For a block-based PFS, this information generally consists of a list of blocks or (server, block) pairs, similar to the traditional inode structure in a local file system. For an object-based PFS, a list of objects or (server, object) pairs is kept along with some parameters that define how data is mapped from the logical file into the objects. In a typical round-robin distribution, a stripe unit parameter defines how much data is placed in each object for a single stripe, and this pattern is repeated as necessary over the set of objects. This algorithmic approach to data distribution has the advantages that file data location metadata does not grow over time and does not need to be modified as the file grows.
F
File Systems
Updates to metadata are usually handled in one of two ways. First, if the PFS already implements a distributed locking system, that system might be used to synchronize access to metadata by clients. Clients can then read, modify, and write metadata themselves in order to make updates. This approach has the advantage of allowing caching of metadata as well. Second, the PFS might implement a set of atomic operations that allow clients to update metadata without locks. For example, a special “insert directory entry” operation can be used by a client to add a new file. The server receiving this message is then responsible for ensuring atomicity of the operation, blocking read operations, or returning previous state until it completes its modifications. This approach reduces network traffic for modifications, but it does not lend itself to caching of modifiable data on the compute node side.
Caching Caching of file system data contributes greatly to the observed performance of local file systems, in part because access patterns on local file systems (e.g., editing files, compiling code, surfing the Internet) often focus on a relatively small subset of data with multiple accesses to the same data in a short period of time. Implementing caching of file system data in the local file system context is also fairly straightforward, because the file system code sees all accesses and can ensure that the cache is kept consistent with what is on storage without any communication. Caching in the context of a parallel file system is more complex when consistency semantics are taken into account. Keeping multiple copies of file system data and metadata consistent across multiple compute nodes requires the same sort of infrastructure as needed to enforce serialization of changes to file data and to ensure notification of changes. Typically parallel file systems that allow caching of data on compute nodes rely on a distributed locking system to ensure that all cached copies are the same, or coherent. Thus the caches are organized much the same as a software distributed memory system. The benefits of caching in parallel file systems are highly dependent on the applications using the system. Checkpointing operations do not typically benefit from caching, because the amount of data written usually exceeds data cache sizes, and the data is not accessed
again soon after. Likewise, large analysis applications that read datasets in parallel often exhibit access patterns that confuse typical cache read-ahead algorithms. Nevertheless, if the parallel file system is also used for home directory storage, or if applications access many small files, then caching can be beneficial.
Fault Tolerance Leading HPC systems use parallel file systems with hundreds of servers and thousands of drives. With that many components, failures are inevitable. In order to ensure that data is not lost, and that the file system remains available in the event of component failures, parallel file systems rely on a collection of fault tolerance techniques. Nearly all parallel file systems rely on redundant array of independent disk (RAID) techniques to cope with drive failures, usually employed only on drives directly accessible by servers (i.e., not between multiple servers). These techniques use parity or erasure codes to allow data to be reconstructed in the event of a failure using the data that remains. Typically, RAID is performed on blocks of storage, but some systems perform RAID on an object-by-object basis []. Server failures are often handled through the use of failover techniques. Figures a and b depict a set of PFS servers configured with shared storage to allow for failover. In Fig. a, no failures have occurred. The system is running in an active–passive configuration, meaning that an additional server is sitting idle in case a failure occurs. When a server does fail (Fig. b), the idle server is brought into active service to take over where the previous server failed. The active–passive configuration allows for a certain number of failures to occur without degrading service. Alternatively, in an active–active configuration all servers are active all the time, with some server “doubling up” and handling the workload of a failed server when a failure occurs. This can lead to a significant degradation of performance. While not depicted in the figures, this configuration can also tolerate storage controller failures, because all drives are accessible through either of the storage controllers. If a storage controller were to fail, the servers on the remaining functional storage controller would take over the responsibilities of the servers that could
File Systems
F
Interconnection network
Active PFS server
Active PFS server
Storage controller
Active PFS server
Idle PFS server
Storage controller
Internal storage network
a
F
Fault tolerant storage configuration with active-passive failover, no failures. Interconnection network
Active PFS server
Failed PFS server
Storage controller
Active PFS server
Active PFS server
Storage controller
Internal storage network
b
Fault tolerant storage configuration after failure; previously idle server has taken over.
File Systems. Fig. Typical PFS configuration using shared backend storage to enable server failover, before (a), and after (b) a server failure. Compute nodes have been omitted from the figure
no longer see storage, maintaining accessibility but with obvious performance penalties. Designs exist that do not share access to storage on the back-end [, ]. As opposed to the approach described above, these designs apply RAID techniques across multiple servers. This approach eliminates the need for expensive enterprise storage on the back-end, but it requires additional coordination between clients and servers in order to correctly update erasure code blocks. In the presence of concurrent writers to the same file region (e.g., writing to different portions of the same file stripe), performance can be significantly reduced.
Parallel File System Interfaces Most applications access parallel file systems through the same interface provided for local file systems, and the industry standard for file access is the POSIX standard. As mentioned previously, the POSIX standard was developed primarily with local file systems and singleuser applications in mind, and as a result it is not an ideal interface for parallel I/O. One reason is its use of the open model for translating a file name into a reference for use in I/O. This model forces name translation to take place on every process, which can be expensive at scale.
F
File Systems
Additionally, the POSIX read and write I/O calls allow for description only of contiguous I/O regions. Thus, complex I/O patterns cannot be described in a single call, leading instead to a long sequence of small I/O operations. Moreover, the POSIX consistency semantics require atomicity of writes and immediate visibility of changes on all nodes. These semantics are stronger than are needed for many HPC applications, and require communication to enforce. An effort is underway to define a set of extensions to the POSIX standard for HPC I/O. This effort includes a set of new function calls that address the three issues outlined here. Implementations of these calls have not yet begun to appear in operating systems, but a few custom implementations do exist in specific parallel file systems, and early results are encouraging. The MPI-IO interface standard is the only current alternative to POSIX for low-level file system access.
Related Entries Checkpointing Clusters Cray XT and Cray XT Series of Supercomputers Cray XT and Seastar -D Torus Interconnect Exascale Computing Fault Tolerance HDF IBM Blue Gene Supercomputer MPI-IO NetCDF I/O Library, Parallel Roadrunner Project, Los Alamos Synchronization Software Distributed Shared Memory
Bibliographic Notes and Further Reading Early Parallel File Systems The Vesta parallel file system [, ] was a file system built at the user level (i.e., not in the kernel) and tested on IBM SP systems. AIX Journaled File System volumes were used for storage by file system servers. Vesta is notable for using an algorithmic distribution of data, exposing a two-dimensional view of files, and providing numerous collective I/O modes and control over consistency semantics.
The Frangipani [] parallel file system and Petal [] virtual block device were originally developed at Digital Equipment Corporation and are well-documented examples of one way of organizing a block-based parallel file system. The Petal system combines a collection of block devices into a single, large, replicated pool that the Frangipani system uses as its underlying storage.
Current Parallel File Systems Currently the largest HPC systems tend to use one of four parallel file systems: IBM’s General Parallel File System (GPFS); Sun’s Lustre; Panasas’s PanFS; and the Parallel Virtual File System (PVFS), a communitybuilt parallel file system whose development is led by Argonne National Laboratory and Clemson University. GPFS [] grew out of the Tiger Shark multimedia file system. It uses a block-based approach, and can operate either with clients physically attached to a storage area network or, more typically, with clients accessing storage attached to file system servers via a shared disk model. A lock-based approach is used for coordinating access to blocks of data, and through its locking system GPFS is able to cache data on the client side and provide full POSIX semantics. Lustre [] is an asymettric parallel file system that uses software-based object storage servers (OSSes) to provide data storage. It is the file system chosen by Cray for use on the Cray XT// systems. File data is striped across objects stored on multiple OSSes for performance. A locking subsystem is used to coordinate file access by clients and allows for coherent, client-side caching. Failure tolerance is provided by using shared storage and traditional server failover. PanFS [] is an asymmetric parallel file system that uses a clustered metadata system in conjunction with lightweight data storage blades that provide data access via the ANSI T- object storage device (OSD) protocol [, ]. It is the file system used on the IBM Roadrunner system at Los Alamos National Laboratory. PanFS is notable in that it does not rely on shared storage on the back-end; instead clients compute and update parity information directly on data storage blades. Parity is calculated on a perfile basis, and objects in which file data is stored are spread across all storage blades to enable declustering of rebuilds in the event of a failure.
Fixed-Size Speedup
The PVFS [] design team chose to specifically target HPC workloads with their design. It is one of the two parallel file systems deployed on the IBM Blue Gene/P system at Argonne National Laboratory; GPFS is the other. Coordinated caching and locking were not used, eliminating false sharing and locking overheads, and API extensions are available that allow complex noncontiguous accesses to be described with single I/O operations. Data is distributed algorithmically across a user-specifiable number of servers, and an object-based model is used that allows servers to locally allocate storage space. Failure tolerance is provided via shared storage between file system servers.
Acknowledgments This work was supported by the Office of Advanced Scientific Computing Research, Office of Science, U.S. Department of Energy, under Contract DE-ACCH. The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC-CH. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government.
. Welch B, Unangst M, Abbasi Z, Gibson G, Mueller B, Small J, Zelenka J, Zhou B () Scalable performance of the Panasas parallel file system. In: Proceedings of the th USENIX Conference on File and Storage Technologies (FAST). San Jose, . Weil SA, Brandt SA, Miller EL, Long DDE. Maltzahn C () Ceph: a scalable, high-performance distributed file system. In: Proceedings of the th USENIX Symposium on Operating Systems Design and Implementation. Berkeley, , pp – . Corbett PF, Feitelson DG () The Vesta parallel file system. ACM Trans Comput Syst ():– . Corbett PF, Feitelson DG, Prost J-P, Baylor SJ () Parallel access to files in the Vesta file system. In: Proceedings of Supercomputing ’. IEEE Computer Society Press, Portland, , pp – . Thekkath C, Mann T, Lee E () Frangipani: a scalable distributed file system. In: Proceedings of the Sixteenth ACM Symposium on Operating System Principles (SOSP). Saint-Malo, October . Lee EK, Thekkath CA () Petal: DISTRIBUTED virtual disks. In: Proceedings of the Seventh International Conference on Architectural Support for Programming Languages and Operating Systems. Cambridge, October , pp – . ANSI INCITS T Committee, Project t/-d working draft: Information technology – SCSI object-based storage device commands (OSD). July . Weber RO () SCSI object-based storage device Commands- (OSD-), revision a. INCITS Technical Committee T/-D working draft, work in progress, Tech. Rep., January . Posix, IEEE () (ISO/IEC) [IEEE/ANSI Std ., Edition] Information Technology — Operating System Interface (POSIX ) — Part : System Application: Program Interface (API) [C Language]. IEEE, New York, NY, USA . Nowicki, B () NFS: Network File System Protocol Specification, RFC , Sun Microsystems, Inc. March, , http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc.html
Bibliography . Schmuck F, Haskin R () GPFS: a shared-disk file system for large computing clusters. In: First USENIX Conference on File and Storage Technologies (FAST’), Monterey, January , pp – . Lang S, Carns P, Latham R, Ross R, Harms K, Allcock W () I/O performance challenges at leadership scale. In: Proceedings of Supercomputing. Portland, November . Nieuwejaar N, Kotz D, Purakayastha A, Ellis CS, Best M () File-access characteristics of parallel scientific workloads. IEEE T Parall Distr, ():– [Online]. Available: http://www.computer.org/tpds/td/abs.htm . Carns PH, Ligon III WB, Ross RB, Thakur R () PVFS: a parallel file system for linux clusters. In: Proceedings of the th Annual Linux Showcase and Conference. Atlanta: USENIX Association, October , pp –. [Online]. Available: http://www.mcs.anl.gov/thakur/papers/pvfs.ps . Braam PJ () The luster storage architecture. Cluster File Systems, Tech. Rep., [Online]. Available: http://lustre.org/docs/lustre.pdf
F
Fill-Reducing Orderings METIS and ParMETIS
First-Principles Molecular Dynamics Car-Parrinello Method
Fixed-Size Speedup Amdahl’s Law
F
F
FLAME
FLAME libflame
Floating Point Systems FPS-B and Derivatives Chris Hsiung Hewlett Packard, Palo Alto, CA, USA
Synonyms SIMD (single instruction, multiple data) machines
Definition FPS AP-B was the first commercially successful attached array processor product designed and manufactured by Floating Pont Systems Inc., in . It is designed to be attached to a host computer, such as the DEC PDP-, as a number crunching accelerator. The key innovation behind this family of array processors is the horizontal micro-coding of the wide instruction set architecture with pipelined execution units to achieve high parallelism. Floating Point Systems Inc., later introduced several derivative products to -bit arithmetic and floating point calculations, by employing higher speed silicon technology, VLSI, and multiple floating point coprocessors.
Discussion Introduction Floating Point Systems (FPS) Inc. was founded in by former Techtronix engineer C. Norm Winningstad in Beaverton, Oregon, to manufacture low-cost, high performance attached floating point accelerators, mainly, for minicomputers, targeting signal processing applications. FPS AP-B was the first attached processor product marketed under the company’s own brand. It was co-designed by George O’Leary and Alan Charlesworth, first delivered in . Its pipelined architecture gave it a peak performance of MFLOPS (Million FLoating-point Operations Per Second), with -bit floating point calculations. By , roughly , machines had been delivered []. Since FPS-B
was designed for mini-computer host machines, such as the DEC PDP-, a larger memory version, FPS-L, was also introduced for mainframe hosts such as the IBM- series. In order to expand to more general purpose scientific and engineering applications, FPS announced the FPS- system in . The word length was extended from -bit to -bit. The addressing capability was expanded from -bit to -bit, and the memory size grew to . MW (Million -bit Words). Even though the peak speed went down a bit, from MFLOPS to MFLOPS, the size of the problem and the precision of the calculations have both increased. The silicon technology remained the same, TTL. In , FPS announced the first -bit machine at higher speed with enhanced architecture, which is the FPS-/MAX. This was the first VLSI design which incorporated matrix accelerator (MAX) boards. This is a SIMD (Single Instruction stream and Multiple Data stream) design that can incorporate up-to MAX boards, each board having computation power equivalent to that of two FPS- processors. With a total of FPS- CPU’s, the peak computational power reached MFLOPS. In , FPS announced the first ECL version, FPS, with a peak speed of MFLOPS. In the meantime, FPS- also went through a makeover with the new VLSI technology, which can now be attached to both the MicroVAX or the Sun workstations. Another derivative of the FPS-B system is a MIMD (Multiple Instruction and Multiple Data streams) version called FPS-, announced in . It incorporated a shared-memory bus architecture among a control processor, three arithmetic coprocessors and an I/O processor, connected to a host. The control processor is the traditional AP-B, and the arithmetic coprocessors, XP, were a new design, but by employing the same concept as AP-B. Since this MIMD architecture approach digressed from the array concept, we will not go into any length in this discussion. One word of clarification regarding AP. In the high performance computing circle, AP stands for Array Processors, even though, unlike the ICL DAP family, there was not necessarily any “array” of processors presented in the architecture. So, realistically, AP can be viewed as Attached Processors, or, we can interpret “array processor” as the attached processor designed
Floating Point Systems FPS-B and Derivatives
especially for efficient array operations. For this family of systems, the common grounds are the horizontal micro-coding concept (each instruction contains multiple fields of micro-instructions, and are decoded in parallel) and the pipelined architecture.
FPS AP-B Hardware The FPS AP-B was a micro-coded, pipelined, array processor design, attached to host computers such as the DEC PDP-/. Data transfer between host and AP was done via MDA (Direct Memory Access). Its cycle time was ns ( MHz), with a peak speed (multiple-add) of MFLOPS. For a detailed architecture block diagram, please consult with Hockney and Jesshope’s book, Fig. ., p. , in []. The whole idea of micro-coding was based on the concept of the (synchronous) parallel execution of multiple execution units. A single -bit instruction word was divided into arithmetic field (address calculation), floating point fields, and two data pad register fields, all under the control of the decoding and execution unit. The address and logic unit (ALU) was -bit wide. There were of -bit S-registers for the ALU. The floating point units (FMUL and FADD) were -bit wide. There were two data pad registers (DX and DY), each was -bit wide. They were both for receiving data from memory and used as operands in floating point calculations. The FADD was a two-stage adder, and FMUL was a three-stage multiplier. In order to keep the pipelines moving, they need to be explicitly “pushed.” For example, in order to push the result of a FADD operation into its destination register, we need to add another FADD operation in the next cycle, either with a valid floating add of another pair of operands (FADD DX, DY), or with another FADD operation (FADD) without any operands. The same principle is true with FMUL. There were three kinds of memory units, the program memory, the main (data) memory, and the table memory. The program memory was K × -bit. The table memory was K × -bit. It contained approximations for sine/cosine and other special functions, and was used to calculate intrinsic function values. The main memory was interleaved into two banks, odd and even. For fast memory option ( ns access time), each bank was capable of producing one result every other cycle
F
(two cycle latency to data pad register). Hence, in the case of fast memory, sequential accesses could be done at each cycle with no delay. For slow memory, access time ns, each bank can only deliver one result every three cycles. The floating point format of the -bit data was a bit unconventional in its word length. The exponent field had -bits in Radix-, and the mantissa had -bits in ’s complement format. Thus, the exponent had a dynamic range of to the power ±, and the mantissa had -digit of accuracy. Hence, it provided much better range and accuracy compared to the IBM -bit format. Extra guard bits were also kept in the calculation in order to improve precision. Format conversion between host and array was done on the fly during the data move. Data moving into, or out of, AP-B were performed by I/O-port (IOP) or a general programmable I/O port (GPIOP) by DMA operations, by stealing cycles if necessary. There were two data width, -bit and -bit. The bandwidth of moving into AP memory was . MW/s, and that of moving out was . MW/s. Special features to support real-time audio and video processing were also provided.
Software and Performance In order to achieve optimal performance on pipelined and horizontally micro-coded machines, it was best to write compact loops, based on skillful arrangement of the critical path. If done right, it was truly a thing of beauty. In carrying out matrix-vector type of operations, for example, the main loop body sometimes could be written in an one-line (single instruction) loop structure. Within that one line, there could be a floating multiple-add pair (might be unrelated ops, but typically “chained” together), indexing, memory ops, and conditional test and branch. Sometimes, an extra cycle could even be saved by storing/accessing some temp data in the table memory, since table memory had an independent data path. Of course, once the loop body was set, the programmer needed to wrap the initialization part and the winding-down code around the loop in order to complete the whole execution. The whole philosophy was very much like coding for vector machines, except that there were multiple instruction fields in the horizontally micro-coded instruction set.
F
F
Floating Point Systems FPS-B and Derivatives
There was no hardware support for divide in the machine. The loop body for a vector divide was three-cycles, for example. A scalar divide was much longer. Initially, in order to use the array processor, users had to use library procedures from the main program running on the host machine. Users use library procedures to transfer data to-and-from the attached machine, and then library procedures to do the computation. Library procedures mainly were used to execute vector (array) operations including intrinsic functions, basic arithmetic operations, signal processing library, image processing library (including D FFT, convolution, filtering), and advanced math library (inc. sparse matrix solution and eigenvalues, Runge–Kutta integration). The next step was the availability of cross Fortran compiler on the host machine, to compile Fortran IV code into object code for the AP, and executed on the array processor side. With this offering, the AP became much more versatile. But, since AP-B did not provide a real time clock, most timing analyses were done on the host side, through APSIM, a timing simulator. Of course, with the use of a simulator, which was , times slower, it could only be useful for shorter running codes.
FPS- The FPS-, announced in , was a -bit enhancement from the -bit AP-B. It incorporated some VLSI components in its design. The cycle time was at ns, a bit slower than its predecessor. It made for MFLOPS peak speed. Other than the double precision design, there are other improvements as well. First, the instruction memory (program memory) and the data memory were merged into one large main memory. Instruction cache was loaded automatically from the instruction memory as needed. Hence this instruction cache (size: , -bit) replaced the program memory of the AP-B. There was also a stack memory, consisting of -bit entries used to store subroutine return addresses, to expedite procedure calls. The table memory, also called auxiliary memory, has primarily become random access memory, used to store intermediate data from registers (e.g., register overflow area). However,
the first K or memory space was read-only, used to store commonly used constants and table values. The main memory consisted of memory modules. Each module had two boards (hence two banks). The memory size ranged from the initial . MW (with K-bit dynamic NMOS) to . MW later (with Kbit dynamic NMOS). All memory accesses were done in a pipelined fashion, both for main memory and table memory. Memory mapping and protections were also available. The FPS- did provide programmable real-time clock and a CPU timer, which enabled accurate timing of program executions.
FPS-/MAX The FPS-/MAX was introduced in . The “MAX” here stood for matrix algebra accelerator. The machine consisted of one FPS- processor, and up-to MAX boards. Each board was capable of two FPS- CPU. Besides, four vector registers were added, each with , elements, at each CPU. In total, the computing power consisted of -CPUs, and had a peak performance of MFLOPS. The MAX boards could execute just a handful of vector operations (SDOT, vector multiple, vector add, scalar vector multiple add, etc.). The logic was implemented with CMOS VLSI technology. The vector registers were memory mapped with the top MW of the MW main memory. That means, vector operations were executed simply by reading from, or writing to, appropriate memory locations. Of course, in order to take advantage of the extra computing power, the arrays had to be very long. Hence, this type of SIMD architecture was more of a special purpose machine.
FPS- The FPS-, announced in , was an (custom aircooled Fairchild K) ECL implementation of the (Schottky TTL technology) FPS-. It increased the clock cycle time from to ns, a speed up of more than three times. Other improvements in the instruction cache and data memory (static NMOS) boasted a total improvement of x to x over the FPS- generation.
Flow Control
Bibliographic Notes and Further Reading For interested readers, the best textbook that covers the complete family of FPS products can be found in []. For a complete history of all parallel computers, please find it in []. In , FPS brought to market its long anticipated T-Series product, a hypercube, distributed memory, message passing multicomputer, with Occam software. Because of its arcane software environment, and nongeneral-purpose nature, it was not very successful in the marketplace. Since then, it changed direction, and brought FPS- product family to the market. These developments are beyond the scope of this entry. Interested readers can find more details from: “FPS Computing: A History of Firsts,” by Howard Thraikill, former President and CEO of FPS, in [].
Bibliography . Hockney RW, Jesshope CR () Parallel computers : architecture, programming, and algorithms. IOP, Bristol . Wilson GV The history of the development of parallel computing. http://ei.cs.vt.edu/~history/Parallel.html . Ames KR, Brenner A () Frontiers of supercomputing II: a national reassessment. University of California Press, Berkeley
Flow Control Jos´e Flich Technical University of Valencia, Valencia, Spain
Synonyms Backpressure
Definition Flow control is a synchronization protocol for transmitting and receiving units of information. It determines the advance of information between a sender and a receiver, enabling and disabling the transmission of information. Since messages are usually buffered at intermediate switches, flow control also determines how resources in a network are allocated to messages traversing the network.
F
Discussion Flow control is defined, in its broad sense, as a synchronization protocol that dictates the advance of information from a sender to a receiver. Flow control determines how resources in a network are allocated to packets traversing the network. First, basic definitions and types of flow control are introduced. Then, differentiation between flow control and switching is highlighted. Finally, basic flow control mechanisms are described and briefly compared.
Messages, Packets, Flits, and Phits Usually, the sender needs to transmit information to the receiver. To do this, the sender transmits blocks of information, referred to as messages, through a path. A header including the routing information is prepended to the message. Depending on the switching technique used (see below and the “switching” entry), the message may be further divided into packets. A packet is made up of a header (with the proper information to route the packet through the network) and a payload. Every message or packet is then further split into flits. The flit (flow control bit) term is defined as the minimum amount of information that is flow controlled. Flits can then be divided into smaller data units called phits (physical units). A phit is usually the amount of information that is forwarded in a cycle through a network link. Depending on the link width a certain number of phits will be required to transmit a flit. Phits are not flow controlled, they only exist because of the link width. Figure shows the breakdown of a message into packets, flits, and phits. Not all the communication systems use these four types of data units. Indeed, in some networks, packets are not needed and the entire message is transmitted (see Wormhole Switching in Switching Techniques entry). Also, in some networks, the flit size equals the phit size, and thus, there is no need to split a flit into smaller phits. In addition, as will be highlighted later, the flit size can span from few bits to the entire packet size.
Types of Flow Control The sender and receiver can be connected in different ways, for instance they can be directly connected through a link, or they can be connected through a
F
F
Flow Control
Message
Packets
Flits
Phits
Flow Control. Fig. Messages, packets, flits, and phits
network (made of routers, switches, and links) where several routers/switches are visited, and links traversed, prior to reaching the receiver. In both cases, a flow control protocol is defined to set up the proper transmission of information. Therefore, different levels of flow control can be used at the same time: end-to-end flow control and link-level flow control.
the type of network (lossy or lossless) constrains the solutions for packet routing, congestion management, and deadlock issues. Also, network performance will be significantly different. Typically, storage/system-area networks (SAN) used in parallel systems and on-chip networks are lossless and local area networks (LAN) and wide area networks (WAN) are lossy.
Bufferless Flow Control Versus Buffered Flow Control
Flow Control Versus Switching
Usually, flow control is used to prevent the overflow of buffering resources when transmitting information. When the queue at the receiver gets full then the flow control must stop the transmission of information at the sender. Once there is room at the queue the flow control resumes the transmission of information. Therefore, flow control is closely related to buffer management. However, there are cases where buffers (or queues) are not used along the transmission path of messages. In that case a bufferless flow control is defined. This is the simplest form of flow control. In that case information is simply forwarded, and potentially the information gets discarded or misrouted (due to contention along the path from the sender to the receiver). A bufferless flow control is also used in the so-called lossy networks were packets may be dropped (discarded) at receiver when the buffer fills up. In such networks, the sender will need to retransmit the packet (via time-out or a NACK control packet sent back from the receiver from upper communication layers, e.g., transport layer). However, in buffered networks, some flow control mechanism is usually used to prevent packet dropping, and such networks are known as lossless (or flow controlled) networks. This is an important distinction of types of networks with huge implications. Indeed,
Flow control is also closely related to switching. Switching determines when and how the information advances through a switch or router. Nowadays, the most frequently used switching mechanisms are wormhole and virtual cut-through. One of the key differences between wormhole switching and virtual cut-through switching lies in the size of buffers and messages. In wormhole switching buffers at routers are usually smaller than message sizes, having slots for only a few flits (and messages can be tens of flits in size or even much larger). However, in virtual cut-through switching, buffers must be larger than packets so as to store an entire packet in a router. Indeed, in virtual cut-through, whenever a packet blocks in the network the packet needs to be stored completely at the blocking router. On the contrary, in wormhole switching the message is spread along its paths across several routers. This fact greatly affects the flow control mechanism and, indeed, affects the flit size of the flow control protocol. In wormhole switching, the flit size is a small part of the message, whereas in virtual cut-through, the flit size is the size of the packet. Because original messages from the sender could be larger than buffers, packetization (splitting a message into bounded packets) is required in virtual cut-through.
Flow Control
As information may travel across multiple routers, the switching inside a router is also seen as a means of flow control. Indeed, some authors do not distinguish between flow control and switching [], and they describe most used flow control mechanisms as wormhole flow control and virtual cut-through flow control. However, a basic distinction exists depending on where the flow control is applied. If applied inside a router or switch then switching mechanism is used and if applied over a link connecting two nodes then flow control is used. Keep in mind also the end-to-end flow control as a third possibility. Figure shows a pair of connecting devices (source and destination nodes) connected through a series of switches and links. The figure illustrates the different levels of flow control (end-toend and link-level) and where switching is performed (inside the switches). Taking into account this basic differentiation between flow control and switching, the following sections describe the most frequently used link-level flow control mechanisms. They are described linked to the suitable switching mechanisms they usually work with. Another key property that distinguishes flow control from switching is that flow control is used as a means to provide backpressure not to overflow receiving buffers. Instead, switching is related to decide when and how information is switched from input ports of a switch/router to output ports of the switch/router.
Impact of Flow Control and Buffer Size Flow control is a key component in a network. An optimized flow control protocol will keep a fraction
of the buffers used most of the times and will minimize the time flits are seated waiting at queues, thus achieving high performance in the network. However, a suboptimal flow control protocol may keep resources underutilized and flits blocked most of the time waiting at buffers, thus ending up in a network with low performance (low throughput and high latencies). In a flow control protocol, the buffer resources at the receiver end must be efficiently used to keep maximum performance. To do this, the buffer depth must be sized according to the latency of the path between the transmitter and the receiver (the time it takes for a flit to travel from the transmitter to the receiver). As both receiver and transmitter will exchange flow control information, the buffer must be sized, at minimum, to allow for the round-trip time of the link. This value is the path latency multiplied by two (two way transmission) plus the time it takes to generate and process the flow control command at both sides. Notice that this is needed in any type of flow-controlled transmission, either link-level or end-to-end. Flow control is also highly related to congestion management. Congestion management deals with high levels of blocking in the network among different messages. To solve such a situation, a congestion management technique may reduce the injection of traffic from a router or from an end node, thus resembling flow control methods (as they dictate when flows advance). However, the reader must be aware that in a congestion management technique the goal is to reduce the blocking experienced in the network, whereas in a flow control method the goal is not to overrun a buffer.
End-to-end flow control
... Destination node
Source node
...
Switch
Switch
Switch Point-to-point flow control
Flow Control. Fig. Different levels of flow control
F
... Switch
Switching
F
F
Flow Control
Link-Level Flow Control The most commonly used link-level flow control protocols are Xon/Xoff, credit-based, and Ack/Nack. However, there are other solutions that, although basic and probably low in performance, help to better understand the concept. This is the case for a handshake protocol. Next, each of these flow control protocols is described. Handshake protocol. The most straightforward method to implement flow control among two devices is by setting a handshake protocol. As an example, a sender may assert a request signal to the receiver thus notifying it has a flit to send. The receiver (depending on its availability for storing new flits) may assert a grant signal to let the transmitter know that there is slot at the queue. Then, the transmission of the flit may begin. For the next flit the handshake protocol needs to be repeated (the handshake works at the flit granularity). Although such protocol is correct and ensures that no queue will overrun, the problem is the low performance achieved in the transmission. Indeed, the delay incurred in the control signal exchange is not overlapped with the transmission of flits, and therefore, there are bubbles introduced along the transmission path. Another potential problem is the excessive exchange of control information (request, grant signals) between the transmitter and the receiver for every transmitted flit. The following link-level flow control methods reduce such overhead. Xon/Xoff flow control. Xon/Xoff flow control is also known as Stop&Go. It is mostly used in networks with wormhole switching. Basically, the receiver notifies the sender either to stop or to resume sending flits once high and low buffer occupancy levels are reached, respectively. To do this, two queue thresholds are set at the receiving side. When the queue fills and passes the Xoff threshold, then the receiver side sends an off flow control signal to the transmitter to stop transmission (in order to prevent buffer overrun). At the transmitter side, when the off control signal is received the transmission is stopped and the sender enters the off mode. Transmission will be resumed only when notified by the receiver. Indeed, when the queue drains and reaches the Xon threshold, then the receiver sends an on flow control signal to the transmitter. The transmitter enters again the on mode and resumes transmission.
Xon and Xoff thresholds are critical from the performance point of view. Indeed, Xoff threshold must be set not to overrun the receiving queue, and the Xon threshold must be set not to introduce bubbles along the transmission path (once forwarding of flits is resumed). Therefore, the round-trip time must be taken into account when setting the thresholds. In addition, Xoff and Xon thresholds must be away enough so as not to send many off and on control signals unnecessarily. It is common to set the Xoff threshold at / and the Xon threshold at / of buffer capacity. Control signals may be sent back to the sender either by using special control signals (thus not using the link data path) or using special control packets through the link data path. With appropriate thresholds, Xon/Xoff significantly reduces the amount of flow control information exchange. However, it requires large buffer memories to avoid buffer overruns. However, there are other approaches to set up the thresholds that may be beneficial. This is the case for the Prizma switch [] where both thresholds are set to the same value. This is efficient if the link length is short enough (round trip time is short) as is the case for the Prizma switch. The benefit in this case is the use of a single control signal that when set means a Go, and when reset means a Stop. Credit-based flow control. Credit-based flow control is usually linked to networks using virtual cutthrough switching. Indeed, the flit size is the packet size and, therefore, packet size is limited to a maximum (buffer size needs to be larger or equal than packet size). In credit-based flow control the sender has a counter of credits. The counter is initially set to the number of flits/slots available at the receiver side. Typically, a credit means a slot for a flit is granted at the receiver side. There are, however, systems where the credit means a chunk of bytes not necessarily the size of the packet (e.g., in InfiniBand network the credit means a slot for bytes). The sender transmits a flit whenever it has credits available. If the credit counter is different from zero, then a flit can be transmitted, and the counter is decremented by one. If the counter reaches zero then the transmitter cannot send more flits to the receiver. The receiver keeps track of receptions and transmissions. Whenever a flit is consumed the receiver sends a new credit upstream. This is notified by a flow
F
Flow Control
Stop & Go
control signal (again this info can travel decoupled from the transmission link or as a control packet through the transmission link). Alternatively, and to reduce the overhead in transmitting control information, the receiver may collect credits and send them in a batch. However, in that case, performance may suffer since forwarding of flits may be delayed. Credit-based flow control requires less than half the buffer size required by Xon/Xoff flow control, thus being more effective regarding resource requirements. In addition, the link reaction time is shorter in creditbased flow control. When the sender runs out of credits then it stops sending flits. When a single credit is returned from the receiver then the transmission is resumed. In Xon/Xoff the go threshold needs to be reached before sending the go signal, thus incurring higher transmission latencies. Figure shows a comparison between the Stop&Go flow control protocol (Xon/Xoff) and the credit-based flow control. As can be seen, the credit-based flow control protocol is able to resume forwarding flits sooner than Stop&Go. Ack/Nack flow control. Flow control is also linked to fault-tolerance in message transmissions. Indeed, the
Stop
Stop
Stop
Stop
Go
Go
Go
Go
Stop signal Sender Last flit returned by stops reaches receiver receiver transmission buffer
Sender uses last credit
Last flit reaches receiver buffer
Flits in buffer get processed
Flits get processed and credits returned
Credit based
# credits returned to sender
Ack/Nack flow control allows the receiver to notify the sender that the last flit sent was correct (received without error) or erroneous (due to a failure or even because there was no buffer available). There is no state information at the transmitter side. Indeed, the transmitter is optimistic and keeps sending flits to the receiver. In this sense, Ack/Nack protocol is also known as optimistic flow control. This flow control method, although it reduces the latency (the sender just transmits the flit when it is available) has a poor bandwidth utilization (sending flits that are dropped). In addition, the transmitter needs logic to deal with timeouts in order to trigger retransmissions (in the absence of either an ACK or NACK signal). Also, it needs to keep a copy of the flit until it receives an ACK signal. Thus, the protocol incurs in an inefficient buffer utilization. Xon/Xoff and credit-based flow control methods do not support failures. Indeed, these protocols only care about buffer flooding. When using such protocols and in the face of a failure, the system needs to rely on a higher-level flow control protocol, most of the time being an end-to-end flow control protocol.
Flow Control. Fig. Flow control comparison (Xon/Xoff vs credits)
Go signal returned to sender
Sender transmits flits
First flit reaches buffer
Sender resumes transmission
First flit reaches buffer
Time Flow control latency observed by receiver buffer
Time
F
F
Flow Control
Other Specific Flow Control Protocols Depending on the type of network used, other flow control protocols may be used trying to add new functionalities, like higher performance, lower flit latencies, fault coverage, etc. One example is T-ERR [] flow control protocol, proposed for on-chip networks. T-ERR increases the operating frequency of a link with respect to a conventional design. In such situations, timing errors become likely on the link. These faults are handled by a repeater architecture leveraging upon a second clock to resample input data, so as to detect any inconsistency. Therefore, T-ERR obtains higher link throughput at the expense of introducing timing errors. Flow Control and Virtual Channels An important issue related to flow control is how to deal with virtual channels (Switch Architecture). Flow control can be instantiated for each virtual channel so that individual virtual channels do not get overflowed. However, this leads to the assumption that buffer resources are statically assigned to each virtual channel, leading to low memory utilization. The problem with this approach comes from the fact that every virtual channel must be deep enough to cover the round-trip time of the link and avoid introducing bubbles in the transmission line. To reduce such overhead, it is common to design a single (and shared) landing pad at the input port of the switch. The landing pad is flow controlled. Flits arriving to the landing pad are then moved to the appropriate virtual channel. More interesting approaches rely on the dynamic assignment of memory resources to virtual channels. Thus virtual channels may grow in depth depending on the traffic demands. In order to cope with such dynamicity, it is common to combine two flow control mechanisms. One example is the use of the Stop&Go flow control protocol per virtual channel and the credit-based flow control protocol for the entire memory at the input port of the switch. Thus, whenever a switch wants to forward a flit to the next switch, it needs to check whether there are global credits available at the input port and whether the receiving virtual channel is in go state. Notice that with the proper setting of thresholds a particular
virtual channel will not consume the entire memory resources, thus leaving room for other virtual channels.
End-to-End Flow Control Link-level flow control protocols can, in principle, be applied in end-to-end scenarios. Indeed, nothing impedes them to work. This is the case for the Ack/Nack protocol implemented at end nodes, in order to guarantee the reception of messages at destination. However, since the round trip time is longer, large buffers are required at both sides to transmit multiple messages while waiting for ACK signals. Usually this is not a major problem, since end nodes implement much larger buffer resources than routers and switches. There are, however, other protocols designed at that level. Indeed, these flow control protocols are known also as message protocols. Two examples are the eager protocol and the rendezvous protocol. The eager protocol just assumes the receiver will have enough allocated space at its buffers to store the transmitted message. In case the receiver was not expecting the message then it has to allocate more buffering and copy the message. During that time, the network may fill up. Thus, the eager protocol may impact performance. In contrast, the rendezvous protocol does not send a message unless the receiver is aware of it. This is of particular interest when the amount of information to send is large. When using the rendezvous protocol, the transmitter first sends a single short message requesting buffer storage at the reception so as to allocate a large buffer. The receiver, after allocating the buffer, sends an acknowledgment to the sender. Upon reception, the sender injects all the messages, which include a sequence number in their header. The receiver just stores every incoming message at the correct buffer slot, depending on the sequence number the message has. Upon reception of all the messages the data info is at the receiver and ordered. The use of the eager or the rendezvous protocol may also affect the way programs are coded. This is the case for MPI programming where different commands can be implemented with different end-to-end flow control protocols. Also, out of order delivery of
Flynn’s Taxonomy
messages can be avoided with the rendezvous protocol. As there are applications that do not work in networks with unordered delivery of messages (e.g., using adaptive routing), then the choice of the rendezvous protocol may be a good choice for exploiting the benefits of adaptive routing without suffering out-of-order delivery.
Michael Flynn Stanford University, Stanford, CA, USA
Related Entries
SIMD (single instruction, multiple data) machines
F
Flynn’s Taxonomy
Synonyms
Congestion Management Switching Techniques
Bibliographic Notes and Further Reading There are several books that cover the field of interconnection networks. High-performance interconnection networks (used in Massively Parallel Systems and/or clusters of computers) are covered in the books of Duato [] and Dally []. Both offer a good text for flow control and in some cases complementary views. In particular, Dally’s book couples flow control with switching providing a common text for both terms. In Duato’s book the term is clearly differentiated and is more focused on switching (rather than flow control). For more specific and emerging scenarios, like onchip networks, the book by De Micheli and Benini [] provides alternative flow control protocols (link-level flow control in Chap. and end-to-end flow control protocols in Chap. ). For computer networks (LANs and WANs) the reader should focus on books like Tanenbaum [] (Chap. for the transport layer and Chap. for the data link layer).
Bibliography . Dally W, Towles B () Principles and practices of interconnection networks. Morgan Kaufmann, San Francisco, CA . Duato J, Yalamanchili S, Ni N () Interconnection networks: an engineering approach. Morgan Kaufmann, San Francisco, CA . de Micheli G, Benini L () Networks on chips: technology and tools. Morgan Kaufmann, San Francisco, CA . Tanenbaum AS () Computer networks. Prentice-Hall, Upper Saddle River, NJ . Tamhankar R, Murali S, Micheli G () Performance driven reliable link for networks-on-chip. In: Proceedings of the Asian Pacific Conference on Design Automation, Shahghai . Denzel WE, Engbersen APJ, Iliadis I () A flexible sharedbuffer swtich for ATM at Gb/s rates. Comput Netw ISDN Syst ()
Definition Flynn’s taxonomy is a categorization of forms of parallel computer architectures. From the viewpoint of the assembly language programmer, parallel computers are classified by the concurrency in processing sequences (or streams), data, and instructions. This results in four classes SISD (single instruction, single data), SIMD (single instruction, multiple data), MISD (multiple instruction, single data), and MIMD (multiple instruction, multiple data).
Discussion Introduction Developed in [] and slightly expanded in [], this is a methodology to classify general forms of parallel operation available within a processor. It was proposed as an approach to clarify the types of parallelism either supported in the hardware by a processing system or available in an application. The classification is based on the view of either the machine or the application by the machine language programmer. It implicitly assumes that the instruction set accurately represents a machine’s micro-architecture. At the instruction level, parallelism means that multiple operations can be executed concurrently within a program. At the loop level, consecutive loop iterations are ideal candidates for parallel execution provided that there is no data dependency between subsequent loop iterations. Parallelism available at the procedure level depends on the algorithms used in the program. Finally, multiple independent programs can obviously execute in parallel. Different instruction set (or instructions set architectures or simply, computer architectures) have been built to exploit this parallelism. In general, an instruction set architecture consists of instructions each
F
F
Flynn’s Taxonomy
of which specifies one or more interconnected processor elements that operate concurrently, solving a single overall problem.
The Basic Taxonomy The taxonomy uses the stream concept to categorize the instruction set architecture. A stream is simply a sequence of objects or actions. There are both instruction streams and data streams, and there are four simple combinations that describe the most familiar parallel architectures []: . SISD – single instruction, single data stream; this is the traditional uniprocessor (Fig. ) which includes pipelined, superscalar, and VLIW processors. . SIMD – single instruction, multiple data stream, which includes array processors and vector processors (Fig. shows an array processor). . MISD – multiple instruction, single data stream, which includes systolic arrays, GPUs, and dataflow machines (Fig. ). . MIMD – multiple instruction, multiple data stream, which includes traditional multiprocessors (multicore and multi-threaded) as well as the newer work of networks of workstations (Fig. ). As the stream description of instruction set architecture is the programmer’s view of the machine, there are limitations to the stream categorization. Although it serves as useful shorthand, it ignores many subtleties of
I
PE
Data memory
Flynn’s Taxonomy. Fig. SISD – single instruction operating on a single data unit
an architecture or an implementation. As pointed out, the original work in even an SISD processor can be highly parallel in its execution of operations. This parallelism need not be visible to the programmer at the assembly language level, but generally becomes visible at execution time through improved performance and, possibly, programming constraints.
SISD: Single Instruction, Single Data Stream The SISD class of processor architecture includes most commonly available workstation type computers. Although a programmer may not realize the inherent parallelism within these processors, a good deal of concurrency can be present. Obviously, pipelining is a powerful concurrency technique and was recognized in the original work. Indeed, in that work, the idea of simultaneously decoding multiple instructions was mentioned but was regarded as infeasible for the technology of the time. (That paper referred to the issuance of more than one instruction at a time as a bottleneck and later publications sometimes refer to this problem as Flynn’s bottleneck). Of course by now almost all machines use pipelining and many machines use one or another form of multiple instruction issue. These multiple issue techniques aggressively exploit parallelism in executing code whether it is declared statically or determined dynamically from an analysis of the code stream. During execution, a SISD processor executes one or more operations per clock cycle from the instruction stream. Strictly speaking, the instruction stream consists of a sequence of instruction words. An instruction word is a container that represents the smallest execution packet visible to the programmer and executed by the processor. One or more operations are contained within an instruction word. In some cases, the distinction between instruction word and operations is crucial to distinguish between processor behaviors. When there is only one operation in an instruction word, it is simply referred to an instruction. Scalar and superscalar processors executes one or more instructions per cycle where each instruction contains a single operation. VLIW processors, on the other hand, execute a single instruction word per cycle where this instruction word contains multiple operations.
Flynn’s Taxonomy
F
I
PE[n –1]
PE[1]
PE[0]
F Data memory[0]
Data memory[n –1]
Data memory[1]
Inter PE Communication Network
Flynn’s Taxonomy. Fig. SIMD – array processor type using PEs with local memory
Data memory
I[0]
I[1]
PE[0]
PE[1]
I[n –1]
DATA IN[0]
Data in[n –1] PE[n –1]
Data out[n –1] to memory
Data out[1]
Flynn’s Taxonomy. Fig. MISD – a streaming processor example
The amount and type of parallelism achievable in a SISD processor has four primary determinates: . The number of operations that can be executed concurrently on a sustainable basis. . The scheduling of operations for execution. This can be done statically at compile time, dynamically at execution, or possibly both. . The order that operations are issued and retired relative to the original program order – these operations can be in order or out of order. . The manner in which exceptions are handled by the processor – precise, imprecise, or a combination.
On the occurrence of an exception (or interrupt), a precise exception handler will (usually) complete processing of the current instruction and immediately process the exception. An imprecise handler may have already completed instructions that followed an instruction that caused the exception and will simply signal that the exception has occurred and it is left to the operating system to manage the recovery. Most SISD processors implement precise exceptions, although a few high-performance architectures allow imprecise floating-point exceptions.
F
Flynn’s Taxonomy I[0]
I[1]
I[n –1]
PE[0]
PE[1]
PE[n–1]
Data memory[1]
Data memory[n–1]
Data memory[0]
Data communications network
Flynn’s Taxonomy. Fig. MIMD – multiple instruction
Scalar (Including Pipelined) Processors A scalar processor is a simple (usually pipelined) processor that processes a maximum of one instruction per cycle and executes a maximum of one operation per cycle. These processors process instructions sequentially from the instruction stream. The next instruction is not processed until the execution for the current instruction is complete, and its results have been stored. The semantics of the instruction determines that a sequence of actions that must be performed to produce the desired result (instruction fetch, decode, data or register access, operation execution, and result storage). These actions can be overlapped but the result must appear in the specified serial order. This sequential execution behavior describes the sequential execution model that requires each instruction executed to completion in sequence. In the sequential execution model, execution is instruction-precise if the following conditions are met: . All instructions (or operations) preceding the current instruction (or operation) have been executed and all results have been committed. . All instructions (or operations) after the current instruction (or operation) are unexecuted or no results have been committed. . The current instruction (or operation) is in an arbitrary state of execution and may or may not have completed or had its results committed.
For scalar and superscalar processors with only a single operation per instruction, instruction-precise and operation-precise executions are equivalent. The traditional definition of sequential execution requires instruction-precise execution behavior at all times, mimicking the execution of a nonpipelined sequential processor. Most scalar processors directly implement the sequential execution model. The next instruction is not processed until all execution for the current instruction is complete and its results have been committed. Pipelining is based on concurrently performing different phases (instruction fetch, decode, execution, etc.) of processing an instruction, with a maximum of one instruction being decoded each cycle. Ideally, these phases are independent between different operations and can be overlapped; when this is not true, the processor stalls the process to enforce the dependency. Thus, multiple operations can be processed simultaneously with each operation at a different phase of its processing. For a simple pipelined machine, only one operation occurs in each phase at any given time. Thus, one operation is being fetched, one operation is being decoded, one operation is accessing operands, one operation is in execution, and one operation is storing results. The most rigid form of a pipeline, sometimes called the
Flynn’s Taxonomy
static pipeline, requires the processor to go through all stages or phases of the pipeline whether required by a particular instruction or not. Dynamic pipeline allows the bypassing of one or more of the stages of the pipeline depending on the requirements of the instruction. There are at least two basic types of dynamic pipeline processors: . Dynamic pipelines that require instructions to be decoded in sequence and results to be executed and stored in sequence. This type of dynamic pipeline provides a small performance over a static pipeline. In-order execution requires the actual change of state to occur in order specified in the instruction sequence. . Dynamic pipelines that require instructions to be decoded in order, but the execution stage of operation need not be in order. In these organizations, the address generation stage of the load and store instructions must be completed before any subsequent ALU instruction does a write back. The reason is that the address generation may cause a page fault and affect the execution sequence of following instructions. This type of pipeline can result in imprecise exceptions mentioned earlier without the use of special order-preserving hardware.
Superscalar Processors Even the most sophisticated scalar dynamic pipelined processor is limited to decoding a single operation per cycle. Superscalar processors decode multiple instructions in a cycle and use multiple functional units and a dynamic scheduler to process multiple instructions per cycle. These processors can achieve execution rates of several instructions per cycle (usually limited to or , but more is possible in favorable application). A significant advantage of a superscalar processor is that processing multiple instructions per cycle is done transparently to the user and provide binary code compatibility. Compared to a dynamic pipelined processor, a superscalar processor adds a scheduling instruction window that analyzes multiple instructions from the instruction stream each cycle. Although processed in parallel, these instructions are treated in the same manner as in a pipelined processor. Before an instruction is
F
issued for execution, dependencies between the instruction and its prior instructions must be checked by hardware. Advanced superscalar processors usually include order-preserving hardware to insure precise exception handling, simplifying the programmer’s model. Because of the complexity of the dynamic scheduling logic, highperformance superscalar processors are usally limited to processing four to eight instructions per cycle.
F VLIW Processors VLIW processors also decode multiple operations per cycle and use multiple functional units. A VLIW processor executes operations from statically scheduled instruction words that contain multiple independent operations. In contrast to superscalar which uses hardware-supported dynamic analyses of the instruction stream to determine which operations can be executed in parallel, VLIW processors rely on static analyses by the compiler. VLIW processors are thus less complex than superscalar processors, and have, at least, the potential for higher performance. For applications that can be effectively scheduled statically, VLIW implementations offer high performance. Unfortunately, not all applications can be so effectively scheduled as execution does not proceed exactly along the path defined by the compiler code scheduler. Two classes of execution variations can arise and affect the scheduled execution behavior: . Delayed results from operations whose latency differs from the assumed latency scheduled by the compiler. . Exceptions or interrupts, which change the execution path to a completely different and unanticipated code schedule. While stalling the processor can control delayed result, this can result in significant performance penalties. The most common execution delay is a data cache miss. Most VLIW processors avoid all situations that can result in a delay by avoiding data caches and by assuming worst-case latencies for operations. However, when there is insufficient parallelism to hide the worst-case operation latency, the instruction schedule can have many incompletely filled or empty operation slots in the instructions, resulting in degraded performance.
F
Flynn’s Taxonomy
SIMD: Single Instruction, Multiple Data Stream The SIMD class of processor architecture includes both array and vector processors. The SIMD processor is created around the use of certain regular data structures, such as vectors and matrices. From the reference point of an assembly-level programmer, programming SIMD architecture appears to be very similar to programming a simple SISD processor except that some operations perform computations on these aggregate data structures. As these regular structures are widely used in scientific programming, the SIMD processor has been very successful in these environments. Array processor and the vector processor differ both in their implementations and in their data organizations. An array processor consists of interconnected processor elements with each having its own local memory space. A vector processor consists of a single processor that references a single global memory space and has special functional units that operate specifically on vectors.
Array Processors The array processor consists of multiple processor elements connected via one or more networks, possibly including local and global inter element communications and control communications. Processor elements operate in lockstep in response to a single broadcast instruction from a control processor. Each processor element has its own private memory, and data is distributed across the elements in a regular fashion that is dependent on both the actual structure of the data and also on the computations to be performed on the data. Direct access to global memory or another processor element’s local memory is expensive, so intermediate values are propagated through the array through local interprocessor connections, which requires that the data be distributed carefully so that the routing required to propagate these values is simple and regular. It is sometimes easier to duplicate data values and computations than it is to affect a complex or irregular routing of data between processor elements. As instructions are broadcast, there is no means local to a processor element of altering the flow of the instruction stream; however, individual processor elements can conditionally disable instructions based on local status information – these processor elements are
idle when the specified condition occurs. Often implementation consist of an array processor coupled to a SISD general-purpose control processor that executes scalar operations and issues array operations that are broadcast to all processor elements in the array. The control processor performs the scalar sections of the application, interfaces with the outside world, and controls the flow of execution; the array processor performs the array sections of the application as directed by the control processor. The programmer’s reference point for an array processor is typically the high-level language level; the programmer is concerned with describing the relationships between the data and the computations but is not directly concerned with the details of scalar and array instruction scheduling or the details of the inter processor distribution of data within the processor. In fact, in many cases, the programmer is not even concerned with size of the array processor. In general, the programmer specifies the size and any specific distribution information for the data and the compiler maps the implied virtual processor array onto the physical processor elements that are available and generates code to perform the required computations.
Vector Processors A vector processor is a single processor that resembles a traditional SISD processor except that some of the function units (and registers) operate on vectors – sequences of data values that are processed as a single entity. These functional units are deeply pipelined and have a high clock rate; although the vector pipelines have a long or longer latency than a normal scalar function unit, their high clock rate and the rapid delivery of the input vector data elements results in a significant throughput that cannot be matched by scalar function units. Early vector processors processed vectors directly from memory. The primary advantage of this approach was that the vectors could be of arbitrary lengths and were not limited by processor registers or resources; however, the high startup cost, limited memory system bandwidth, and memory system contention were significant limitations. Modern vector processors require that vectors be explicitly loaded into special vector registers and stored back into memory from these registers, the same course that modern scalar processors have taken for similar reasons. The vector register file consists of several sets of
Flynn’s Taxonomy
vector registers (with perhaps registers in each set). Vector processors have several features that enable them to achieve high performance. One feature is the ability to concurrently load and store values between the vector registers and main memory while performing computations on values in the vector register file. This feature is important because the limited length of vector registers requires that long vectors be processed in segments. Not being able to overlap memory accesses and computations would pose a significant performance bottleneck. Just like SISD processors, vector processors support a form of result bypassing – in this case called chaining – that allows a follow-on computation to commence as soon as the first value is available from the preceding computation. Thus, instead of waiting for the entire vector to be processed, the follow-on computation can be largely overlapped with the preceding computation that it is dependent on. Sequential computations can be efficiently compounded and behave as if they were a single operation with a total latency equal to the latency of the first operation with the pipeline and chaining latencies of the remaining operations but none of the start-up overhead that would be incurred without chaining. As with the array processor, the programmer’s reference point for a vector machine is the high-level language. In most cases, the programmer sees a traditional SISD machine; however, as vector machines excel on vectorizable loops, the programmer can often improve the performance of the application by carefully coding the application, in some cases explicitly writing the code to perform register memory overlap and chaining, and by providing hints to the compiler that help to locate the vectorizable sections of the code. As languages are defined (such as Fortran or HighPerformance Fortran) that make vectors a fundamental data type, the programmer is exposed less to the details of the machine and to its SIMD nature.
MISD: Multiple Instruction, Single Data Stream The MISD architecture is a pipelined ensemble where the function of the individual stages are determined by the programmer before execution is begun; then data is streamed through the pipeline, forwarding results from one (stage) function unit to the next. On the micro-architecture level this is exactly what the vector processor does. However, in the vector pipeline
F
the programmer initially organizes the data into suitable data structure, then streams the data elements into stages where the operations are simply fragments of an assembly-level operation, as distinct from being a complete operation. Surprisingly, some of the earliest attempts in the s could be seen as the MISD computers (they were actually programmable calculators; as they were not “stored program machines” in the sense that the term is now used). They used plug boards for programs, where data in the form of a punched card was introduced into the first stage of a multistage processor. A sequential series of actions was taken where the intermediate results were forwarded from stage to stage until, at the final stage, a result would be punched into a new card. While the MISD term is not commonly used, there are interesting uses of the MISD organization. A frequently used synonym for MISD is “streaming processor.” Some of these processors are commonly available. One example is the GPU (graphics processing unit); early and simpler GPUs provided limited programmer flexibility in selecting the function or operation of a particular stage (ray tracing, shading etc.). More modern GPGPUs (general purposed GPU) are more truly MISD with a more complete operational set at the individual stage in the pipeline. Indeed, many modern GPGPU incorporate both MISD and MIMD as these systems offer multiple MISD cores. Other MISD style architectures include dataflow machines. In these machines, the source program is converted into a data flow graph, each node of which is a required operation. Data then is streamed across the implemented graph. Each path through the graph is a MISD implementation. If the graph is relatively static for particular data sequences, then the path is strictly a MISD if multiple paths are invoked during program execution then each path is MISD and there are MIMD instances in the implementation. Such dataflow machines have been realized by FPGA implementations.
MIMD: Multiple Instruction, Multiple Data Stream The MIMD class of parallel architecture consists of multiple processors (of any type) and some form of interconnection. From the programmer’s point of view each processor executes independently but to cooperatively execute to solve a single problem although some
F
F
Flynn’s Taxonomy
form of synchronization is required to pass information and data between processors. Although no requirement exists that all processor elements be identical, most MIMD configurations are homogeneous with all processor elements identical. There have been heterogeneous MIMD configurations that use different kinds of processor elements to perform different kinds of tasks, but these configurations are usually for special-purpose applications. From a hardware perspective, there are two types of homogeneous MIMD: multi-threaded and multi-core processors; frequently implementations use both.
Multi-Threaded Processors In multi-threaded MIMD, a single base processor is extended to include multiple sets of program and data registers. In this configuration, separate threads or programs (instruction streams) occupy each register set. As resources are available, the threads continue execution. Since the threads are independent so is their resource usage. So ideally, the multiple threads make better use of resources such as floating point units and memory ports resulting in a higher number of instructions executed per cycle. Multi-threading is a quite useful complement to large superscalar implementations as these have relative extensive set of expensive floating point and memory resources. Critical threads (tasks) can be given priority to insure short execution while less critical tasks avail of the underused resources. As a practical matter, these designs must insure a minimum service for every task to avoid a type of task lockout. Otherwise, the implementation is straightforward since all tasks can share data caches and memory without any interconnection network.
Multi-Core and Multiple Multi-Core Processor Systems At the other end of the MIMD spectrum are multiple multi-core processors. These implementations must communicate results through an interconnection network and coordinate task control. Their implementation is significantly more complex than the simpler MIMD or even more complex SIMD array processor. The interconnection network in the multiprocessor passes data between processor elements and synchronizes the independent execution streams between processor elements. When the memory of the
processor is distributed across all processors and only the local processor element has access to it, all data sharing is performed explicitly using messages and all synchronization is handled within the message system. When communications between processor elements are performed through a shared memory address space – either global or distributed between processor elements (called distributed shared memory to distinguish it from distributed memory) – there are two significant problems that arise. The first is maintaining memory consistency: the programmer-visible ordering effects on memory references, both within a processor element and between different processor elements. This problem is usually solved through a combination of hardware and software techniques. The second is cache coherency – the programmer invisible mechanism to ensure that all processor elements see the same value for a given memory location. This problem is usually solved exclusively through hardware techniques. The primary characteristic of a large MIMD multiprocessor system is the nature of the memory address space. If each processor element has its own address space (distributed memory), the only means of communication between processor elements is through message passing. If the address space is shared (shared memory), communication is through the memory system. The implementation of a distributed memory machine is easier than the implementation of a shared memory machine, when memory consistency and cache coherency are taken into account, while programming a distributed memory processor can be more difficult. In recent times, large-scale (more than , cores) multiprocessors have used the distributed memory model based on implementation requirements.
Related Entries Cray Vector Computers Data Flow Computer Architecture EPIC Processors Fujitsu Vector Computers IBM Blue Gene Supercomputer Illiac IV Superscalar Processors VLIW Processors
FORGE
Bibliographic Notes and Further Reading Parallel processors have been implemented in one form or another almost as soon as the first uniprocessor implementation. Most early implementations were straightforward MIMD multiprocessors. With the advent of SIMD array processors, such as Illiac IV [], the need for a distinction became clear. Because the taxonomy is simple, numerous extensions have been proposed; some fairly general such as those of Handler [], Kuck [], and El-Rewini []. Other extensions were of more specialized nature such as Göhringer [], Chemij []. Additional comments on the taxonomy and elaborations can be found in the standard texts on parallel architecture such as Hwang [], Hockney [], and El-Rewini []. Xavier [] and Quinn [] are helpful texts which view the taxonomy from the programmer’s view point.
Bibliography . Flynn MJ () Very high speed computers. Proc IEEE :– . Flynn MJ () Some computer organizations and their effectiveness. IEEE Trans Comput ():– . Flynn MJ, Rudd RW () Parallel architectures. ACM Comput Surv ():– . Barnes GH, Brown RM, Kato M, Kuck D, Slotnick DL, Stokes RQ () The ILLIAC IV computer. IEEE Trans Comput C-: – . Handler W () Innovative computer architecture – how to increase parallelism but not complexity. In: Evans DJ (ed) Parallel processing systems, an advanced course. Cambridge University Press, Cambridge, pp –, --- . Kuck D () The Structure of Computers and Computation vol. , J Wiley, New York . El-Rewini H, Abd-El-Barr M () Advanced computer architecture and parallel processing. Wiley, New York . Göhringer D et al () A taxonomy of reconfigurable single-/ multiprocessor systems-on-chip. Int J Reconfigurable Comput : . Chemij W () Parallel computer taxonomy, MPhil, Aberystwyth University . Hwang K, Briggs FA () Computer architecture and parallel processing. McGraw Hill, London, pp –, --- . Hockney RW, Jesshope CR () Parallel computers . Adam Hilger/IOP Publishing, Bristol, --- . Xavier C, Iyengar SS () Introduction to parallel algorithms, Wiley, New York . Quinn MJ () Designing efficient algorithms for parallel computers. McGraw Hill, London, ---
F
Forall Loops Loops, Parallel
FORGE John M. Levesque , Gene Wagenbreth Cray Inc., Knoxville, TN, USA University of Southern Califorina, Topanga, CA, USA
Synonyms Interactive parallelization; Parallelization; Vectorization; Whole program analysis
Definition FORGE is an interactive program analysis package built on top of a database representation of an application. FORGE was the first commercial package that addressed the issue of performing interprocedural analysis for Program Consistency, Shared and Distributed Parallelization, and Vectorization. FORGE was developed by a team of computer scientists employed by Pacific Sierra Research and later Applied Parallel Research.
Discussion Introduction When FORGE was marketed in , whole program parallelization was not quick, easy, or convenient. Vectorization was a mature user and compiler art. Many users could write vectorizeable loops that a compiler could translate to efficient vector instructions. Distributed and shared memory multiprocessors could be treated as vector processors and parallelized in the same manner. This is frequently very inefficient. Parallelization introduces much more overhead than vectorization. To overcome overhead parallelization requires a much higher granularity than vectorization. Parallelization of outer loops containing nested subroutine calls is usually the best approach. Achieving this high granularity was very difficult for a user to do by hand, and practically impossible for a compiler to accomplish automatically. FORGE was created to bridge
F
F
FORGE
this gap, allowing the user to interactively initiate compiler analysis and view the results. The user controlled the parallelization process using high-level program knowledge. FORGE performed the time-consuming, tedious analysis that computers are better at than programmers. At the time, a group at Rice University, under the direction of Ken Kennedy, were developing a “Whole Program Analysis” software called Parascope. At the same time, a few of the developers of the VAST vectorizing pre-compiler began a project to develop FORGE, based on a database representation of the input application. In addition to the source code, runtime statistics could be entered into the database to identify important features derived during the execution of the application. Upon this database representation of the program an interactive, windows-based environment was built that allowed the user to interactively:
. Peruse the database to find inconsistencies in the program a. COMMON block analysis b. Tracing of variable through the call chain showing aliases from subroutine calls, equivalences, and COMMON blocks c. Identification of variables that are used and not set d. Identification of variables that are set and not used e. Queries to identify variables that satisfied certain user-defined templates . Interactively Vectorize Application a. Perform loop restructuring i. Loop splitting ii. Routine inlining – pull routine into a loop iii. Routine outlining – push loop into a routine iv. Loop inversion . Interactive Shared Memory Parallelization a. Array scoping b. Storage conflicts due to COMMON blocks c. Identification of Reduction function variables d. Identification of Ordered regions due to data dependencies . Interactive Distributed Memory Parallelization a. User-directed loop parallelization b. User-directed array distribution
c. Automatic loop parallelization based on array distribution d. Automatic array distribution based on parallel loop access e. Iteration of c. and d. for whole program parallelization f. Generate HPF . Interactive Program Restructuring a. Loop Inversion b. Loop splitting c. Array index reordering d. Routine inlining e. Routine outlining f. TASK COMMON generation . Interactive CACHE View tool a. Given a hardware’s Level and Level Cache sizes and runtime statistics i. Instrument code ii. Identify Cache alignment issues iii. Identify Cache overflows iv. Propose Blocking strategies
General Structure Forge was written in the C programming language. The parser was constructed using the compiler-compiler tools LEX and YACC. The windowing system used was originally a custom interface written for SUN, APOLLO, and DEC workstations. Later a MOTIF based X-Windows system was used.
The Database To perform the aforementioned functions, FORGE needed significant information concerning the input application. At the end of the development of FORGE in , the database could input the application source code (Only Fortran ) and runtime statistics. From the source code, the following data was extracted: . For each routine a. Variable information i. Aliasing from COMMON, EQUIVALENCE, SUBROUTINE argruments ii. Read and Write references iii. Data type and size iv. Array dimensions v. Array indexing information
FORGE
b. Control structures within the routine – IF’s and DO’s . Call chain information a. Static Call chain – all possible calling sequences b. Dynamic Call chain – derived from the runtime statistics . Variable Aliasing through call chain, both static and dynamic
F
. Basic Block (control flow) information for the entire application a. All IF constructs controlling the execution of the basic block b. All reaching definitions for variables used in the basic block c. All uses of variable set in the basic block . Array section analysis a. Information on every looping structure i. Section of array that is accessed by each DO loop . Integer value information (FACTS) for array index and data dependency analysis
Database Design The FORGE database was designed and implemented by the developers. An off-the-shelf database was not used. In design of the database, the generation of the database was performed incrementally with the ability to automatically update the database when the source code was modified. The intent was to spend the time building the database so the interactive analysis within the FORGE environment could be performed quickly. FORGE stored a program as a Package. Each package contained a list of the source files in a program, along with other information such as directories to search, compiler options to use, and target hardware. Brute force whole program analysis using the hardware available at the time was very time consuming for convenient interactive program analysis. For that reason, a database was created containing only data needed for parallelization of a program. Each program unit (subroutine, function, main program) was analyzed and stored separately in the database, on disk. Last modified dates were saved for each file. A checksum was saved for each program unit in a file. When the modification date for a file indicated that the database was out of date, the file was rescanned and the checksums for each routine compared to the checksums saved with the database. The database was regenerated only for modified routines. Building the initial database for a large program could take an hour. After the initial creation, updating the database for source file changes usually took a few seconds. When the user selected a package for analysis, tables were built in memory combining the databases for called and calling routines. This process typically took
F
F
FORGE
a few seconds. The efficiency of the incremental update of database and the handling of called routines in memory are the key features that made interactive whole program analysis feasible. The database was designed to quickly supply information to the Vectorizer and Parallelizer when performing array section and scalar dependency analysis. All of the analysis features used by the parallelization analysis were made available to the user for program browsing. This only required the implementation of menus, since all the analysis was already implemented. The browser ended up as one of the most popular and powerful features of FORGE. Users could develop templates to define routines that were called from the application and whose source was not available. These templates simply supplied the information about variables that were either passed into the routine through the argument list or through COMMON blocks.
Runtime Statistics FORGE had an option to instrument the application to gather information about important routines, DO loops, IF constructs, and Cache utilization. The instrumented program could then be run with an impor-
tant dataset and an ASCII output would be generated that could be fed into the database to assist in performing analysis. There was an option to ignore static information in favor of the dynamic information when performing the aforementioned analysis. The following runtime statistics were gathered. – – – – –
CPU time of each routine, inclusive and exclusive Lengths of DO loops CPU time of each DO loop, inclusive and exclusive Frequency of IF constructs (optional) Array addresses (optional)
Interactive Tools available to the User Query Database A popular tool named Query allowed the user to find all variables in a program that matched criteria defined in templates. Templates specified criteria for: ● ● ● ● ●
Variable name Variable type Variable storage Statement context Statement type
FORGE
F
Many common templates were supplied or the user could create their own templates.
contained I/O or unknown routine, the user would be alerted to that. The user would then say:
● ● ● ●
. Parallelize Automatically . Parallelize Interactively
Show all variables read into the program Show all variables written out Show all variables used within particular DO loop Show all variables set under a condition and used unconditionally
Cache Vu Tool Cache Vu tool, another tools within the FORGE system, was extremely powerful and informative. The runtime statistics to support this feature was very large and had to be collected in a controlled manner. FORGE was able to visually show the user which arrays were overlapping in the cache and being evicted from cache due to cache associativity.
COMMON Block Grid A grid was displayed with routines down the side and COMMON blocks across the top. When a COMMON block was selected, variables within that COMMON block were shown across the top showing how each variable was referenced in each routine.
USE-DEF and DEF-USE This was extremely useful in debugging an application. Plans to attach FORGE to a debugger never materialized. – User could click on a variable in the source and drop down a menu to select, show reaching definition, or show subsequent uses. The uses and/or sets were shown within the control structures controlling the execution of the line.
Tracing Variables Through the Call Chain All occurrences of a variable were shown interspersed within the call chain, including IF constructs. Aliasing was shown where occurs due to EQUIVALENCE, COMMON block, or SUBROUTINE call.
Shared Memory Parallelizer If runtime statistics were available, the window would present the user with the highest level DO loop that is a potential candidate for parallelization. If the DO loop
The automatic mode would complete without interaction and show the user if there were any dependency regions, storage conflicts, or other inhibitors. The Interactive mode would converse with the user, showing each inhibitor as encountered. The users could then instruct the parallelizer to ignore dependencies, if appropriate, and continue. Finally, the user could output the parallelized code in OpenMP, Cray Micro-tasking, or as a parallelized routine to be run with FORGE’s shared memory runtime library. The latter was accomplished by outlining the DO loop into a subroutine call that got called in parallel. Shared variables were passed as arguments and private variables made local to the outlined routine.
Distributed Memory Parallelizer The user could select a set of arrays to decompose, decompose the arrays through a menu – only single index decomposition was supported. FORGE would then identify every DO loop that accessed the decomposed dimension and the user could have FORGE distribute the loop iterations using the owner execute rule automatically. During the loop parallelization, FORGE searched for additional arrays to decompose and loops to parallelize. This cycle of array decomposition and loop distribution continued until no new transformations were found. Parallelization was saved in internal format between sessions. The user could choose to save the parallelization in a new copy of the source program with HPF directives. Executable parallel code could be generated either with the interactive parallelizer or a batch translator named XHPF. The translated code contained calls to a runtime library. Storage for distributed arrays was allocated at runtime as determined by the size of the array and the number of distributed CPUs used. Ghost cells were allocated as needed. Parallel loop bounds were computed at runtime. Internode communication was performed pre and post loop as required. Communication was inserted in serial code as required. One node was used to perform all IO operations.
F
F
FORGE
Program Restructurer The parallelized code was available to the user for all the program transformations executed by FORGE. When the source code was modified, it would be presented to the user and the user could accept the new version and the database would be updated accordingly.
Parallel Program Generation FORGE used the result of analysis to generate parallel programs. It performed source-to-source translation. The input could contain compiler parallelization directives. Syntax defined by most major vendors of the time (IBM, NEC, FUJITSU, DEC, INTEL, NCUBE) were supported, as well as HPF and OMP standards. FORGE could take any of these as input, combine them with interactive or automatic analysis, and generate directives. FORGE could also generate a parallel program containing calls to a FORGE runtime library. Versions of the library existed for shared and distributed memory parallelization for the major vendors. The user would compile the FORGE output with the vendor supplied F compiler, link with the appropriate FORGE library, and run the parallel application. Instrumented versions of the FORGE parallel libraries were provided. When linked with the instrumented libraries and run, timing data was created. The timing data could be viewed interactive with FORGE, supper imposed on a dynamic callchain. Efficiencies and “hotspots” were identifiable and correctable. Batch tools, SPF – shared memory parallelizer, and XHPF (translate HPF) performed the same translation functions as FORGE. A user could modify program directives with a conventional editor and run these tools as precompilers as part of the normal compilation stack. Future Directions There were three major deficiencies in Forge. First, Forge handled only a subset of Fortran . Second, Forge was developed with Unix Xwindows interface. A convenient native MS Windows interface was needed. Last, the display of parallel loop inhibitors due to loop carried dependencies often contained many false
FORGE
F
F
dependencies. The minimization of the false dependencies and a convenient method for the user to view and ignore dependencies were problems for Forge, as well as all other parallelization tools then and now. Work continued in these areas through the year when Applied Parallel Research went out of business.
Bibliography . Balasundaram V, Kennedy K, Kremer U, McKinley K, Subhlok J () The parascope editor: an interactive parallel programming tool. In: Conference on high performance networking and computing proceedings of the ACM/IEEE conference on Supercomputing, ACM, New York . Kushner EJ () Automatic parallelization of grid-based applications for the iPSC/. In: Lecture notes in computer science, vol , pp – . Gernt M () Automatic parallelization of a crystal growth simulation program for distributed-memory system. In: Lecture notes in computer science, vol , pp – . Applied Parallel Research. FORGE Baseline system user’s guide . Applied Parallel Research. FORGE high performance Fortran xHPF user’s guide
. Applied Parallel Research. FORGE Distributed memory parallelizer user’s guide . Song ZW, Roose D, Yu CS, Berlamont J () Parallelization of software for coastal hydraulic simulations for distributed memory parallel computers using FORGE . In: Power H (ed) Applications of high-performance computing in engineering IV, pp – . Frumkin M, Hribar M, Jin H, Waheed A, Yan J () A comparison of automatic parallelization tools/compilers on the SGI Origin. In: Proceedings of the ACM/IEEE SC Conference (SC’), Orlando. IEEE Computer Society, Washington . Bergmark D () Optimization and parallelization of a commodity trade model for the IBM SP/, using parallel programming tools. In: International Conference on Supercomputing, Proceedings of the th international conference on Supercomputing, Barcelona . Kennedy K, Koelbel C, Zima H () The rise and fall of high performance Fortran: an historical object lesson - History of Programming Languages. In: Proceedings of the Third ACM SIGPLAN Conference on History of Programming Languages, San Diego. ACM, New York, pp --- . Saini S () NAS Experiences of Porting CM Fortran Codes to HPF on IBM SP and SGI Power Challenge. NASA Ames Research Center, Moffett Field
F
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
Formal Methods–Based Tools for Race, Deadlock, and Other Errors Peter Müller ETH Zurich, Zurich, Switzerland
Definition Formal methods–based tools for parallel programs use mathematical concepts such as formal semantics, formal specifications, or logics to examine the state space of a program. They aim at proving the absence of common concurrency errors such as races, deadlocks, livelocks, or atomicity violations for all possible executions of a program. Common techniques applied by formal methods–based tools include deductive verification, model checking, and static program analysis.
Discussion Introduction Concurrent programs are notoriously difficult to test. Their behavior depends not only on the input but also on the scheduling of threads or processes. Many concurrency errors occur only for very specific schedulings, which makes them extremely hard to find and reproduce. An alternative to testing is to apply formal methods. Formal methods use mathematical concepts to examine all possible executions of a program. Therefore, they are able to guarantee certain properties for all inputs and all possible schedulings of a program. Most formal methods–based tools for concurrent programs focus on races and deadlocks, the two most prominent concurrency errors. Some of them also detect other errors such as livelocks, atomicity violations, undesired nondeterminism, and violations of specifications provided by the user. This entry covers tools based on three kinds of formal methods. () Tools for deductive verification use mathematical logic to construct a correctness proof for the input program. () Tools for state space exploration (model checkers) check properties of a program by exploring all its executions. () Tools for static program analysis approximate the possible dynamic behaviors of a program and then check properties of these approximations. This entry summarizes the
foundations and applications of these approaches, discusses their strengths and weaknesses, and names a few representative tools for each approach. The evaluation of the three approaches focuses on the following criteria: Soundness. A verification or analysis tool is sound if it produces only results that are valid with respect to the semantics of the programming language. In other words, a sound tool does not miss any errors. Soundness is the key property that distinguishes formal methods from testing. However, some formal methods–based tools sacrifice soundness in favor of reducing false positives or improving efficiency. Completeness. A verification or analysis tool is complete if it does not produce false positives, that is, each error detected by the tool can actually occur according to the semantics of the programming language. Minimizing the number of false positives is crucial for the practical applicability of a tool because each reported error needs to be subjected to further investigation, which usually involves human inspection. Modularity. A verification or analysis tool is modular if it can analyze the modules of a software system independently of each other and deduce the correctness of the whole system from the correctness of its components. Modularity allows a tool to analyze program libraries and improves scalability. Automation. A verification or analysis tool has a high degree of automation if it requires little or no user interaction. Typical forms of user interaction include providing the specification to be checked by the tool, assisting the tool through program annotations, and constructing proofs manually. A high degree of automation is necessary to make the application of formal methods-based tools economically feasible. Efficiency. A verification or analysis tool is efficient if it can analyze large programs in a relatively short amount of time and space. In general, the number of states and executions of a program grows exponentially in the size of the program (for instance, the number of variables, branches, and threads). Therefore, tools must be highly efficient in order to handle practical programs.
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
Deductive Verification Overview Tools for deductive verification construct a formal proof that demonstrates that a given program satisfies its specification. Common ways of constructing the correctness proof are () to use a suitable calculus such as the weakest precondition calculus to compute verification conditions – logical formulas whose validity entails the correctness of the program – and then prove their validity in a theorem prover, or () to use symbolic execution to compute symbolic states for each path through the program and then use a theorem prover to check that the symbolic states satisfy the specification. Both approaches employ a formal semantics of the programming language. The specification to be verified typically includes implicit properties (such as deadlock freedom) as well as properties explicitly specified by the user (for instance through pre- and postconditions). Deductive verifiers for concurrent programs apply various verification methodologies, which are illustrated by the Java example below. Monitor invariants (see entry Monitors, Axiomatic Verification of) allow one to express properties of the monitor variables (here, the fields in class Account) that hold when the monitor is not currently locked by any thread. For instance, one could express that the transaction count is a nonnegative number. Rely-guarantee reasoning [] allows one to reason about the possible interference of other threads. For instance, one could prove that no thread ever decreases the transaction count. Deadlock freedom can be proved by establishing a partial order on locks and proving that each thread acquires locks in ascending order (see entry on Deadlocks). For instance, one could specify that the order on the locks associated with Account objects is given by the less-than order on the accounts’ numbers. The check that locks are acquired in ascending order would fail for the example because the locks of this and to are acquired irregardless of the order. Therefore, the concurrent execution of x.transfer(y, a) and y.transfer(x, b) might deadlock. class int int int
Account { number; balance; transactionCount;
F
void transfer(Account to, int amount) { if(this == to) return; acquire this; acquire to; balance = balance - amount; to.balance = to.balance + amount; transactionCount = transactionCount + 1; to.transactionCount = to.transactionCount + 1; release to; release this; } // constructors and other methods omitted }
The absence of data races can be proved using a permission system that maintains an access permission for each shared variable. The permission gets created when the variable comes into existence; it can be transferred among threads and between threads and monitors, but not duplicated. Each shared variable access gives rise to a proof obligation that the current thread possesses the permission for that variable. Since access permissions cannot be duplicated, this scheme prevents data races. In the example above, one could store the permissions for balance and transactionCount in the monitor of their object. By acquiring the monitors for this and to, the current thread obtains those permissions and, therefore, is allowed to access the fields. When the monitors are released, the permissions are transferred back to the monitors. Such a permission system is for instance applied by concurrent separation logic []. An alternative technique for proving the absence of data races is through object ownership []. Ownership schemes associate zero or one owner with each object. An owner can be another object or a thread. A thread becomes the owner of an object by creating it or by locking its monitor. Each access to an object’s field gives rise to a proof obligation that the object is (directly or transitively) owned by the current thread. Since an object has at most one owner, this scheme prevents data races. Besides preventing data races, both permissions and ownership guarantee that the value of a variable does not change while a thread holds its permission or owns the object it belongs to. Therefore, they also give an atomicity guarantee, which allows verifiers to apply sequential reasoning techniques in order to prove additional program properties.
F
F
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
Discussion Program verifiers are sound unless there is a flaw in the underlying logic or the implementation of the verifier. In principle, program verification is complete if the underlying logic is complete with respect to the semantics of the programming language. However, program verifiers typically enforce some kind of verification methodology as described above. These methodologies simplify the verification of certain programs, but fail for others. For instance, a tool that requires lock ordering for deadlock prevention might not be able to handle programs that prevent deadlocks by other strategies. Another source of incompleteness are the limitations of theorem provers. Since the validity of verification conditions is in general undecidable, program verifiers based on automatic provers such as SMT solvers report spurious errors when the prover cannot confirm the validity of a verification condition. A number of program verifiers support modular reasoning. For instance, a permission system removes the necessity to inspect the whole program in order to detect data races. Nevertheless, modular verification is still challenging for certain program properties such as global invariants and liveness properties. Deductive program verifiers are typically not fully automatic. Users have to provide the specification the program is verified against. Even if the property to be proved is the absence of a simple error such as data races, users typically have to annotate the code with auxiliary assertions such as loop invariants. Moreover, interactive verifiers require users to guide the proof search. The efficiency of automatic verifiers (those verifiers where the only user interaction is by writing annotations) depends on the number and complexity of the generated verification conditions as well as on the performance of the underlying automatic theorem prover. The former aspect is positively influenced by applying a modular verification methodology, the latter benefits from the enormous progress in the area of automatic theorem proving. In summary, since program verifiers are sound and since modular verifiers scale well, they are well suited for software projects with the highest quality requirements, especially safety-critical software. However, the overhead of writing specifications as well as limitations of existing verification methodologies and automatic
theorem provers have so far prevented program verification from being applied routinely in mainstream software development.
Examples Most verifiers for concurrent programs are in the stage of research prototypes. Microsoft’s VCC is an automatic verifier for concurrent, low-level C code. It has been used to verify the virtualization layer Hyper-V. VCC applies a verification methodology based on ownership []. Permission systems are used by verifiers based on separation logic such as VeriFast for C programs [], SmallfootRG [], and Heap-Hop [], as well as by Chalice []. Both Heap-Hop and Chalice support shared variable concurrency and message passing. Lock-free data structures have been verified using the interactive verifier KIV [].
State Space Exploration Overview Software model checkers consider all possible states that may occur during the execution of a program. They check properties of individual states or sequences of states (traces) through an exhaustive search for a counterexample. This search is either performed by enumerating all possible states of a program execution (explicit state model checking) or by representing states through formulas in propositional logic (symbolic model checking). In both approaches, programs are regarded as transition systems whose transitions are described by a formal semantics of the programming language. The specifications to be checked are given as formulas in propositional logic for individual states (assertions) or as formulas in a temporal logic (typically LTL or CTL) for traces. The use of temporal logic allows model checkers in particular to check liveness properties such as “every request will eventually be handled.” Liveness properties are important for the correctness of all parallel programs, but especially for those based on message passing (such as MPI programs) because they often apply sophisticated protocols. Other errors such as deadlocks can be detected without giving an explicit specification. For instance for the account example above, a model checker detects that for some input values and thread interleavings, method transfer does not reach a valid terminal
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
F
state. The inputs and interleavings are then reported as a counterexample that can be inspected and replayed by the user. The main limitation of software model checking is the so-called state space explosion problem. The state space of a program execution grows exponentially in the number of variables and the number of threads. Therefore, even small programs have a very large state space and require elaborate reduction techniques to be efficiently checkable. Partial order reduction reduces the number of relevant interleavings by identifying operations that are independent of actions of other threads (for instance, local variable access). Abstraction is used to simplify the transition system before checking it. It is common that many aspects of a program execution are not relevant for a given property to be checked. For instance, the values of amount and balance in the transfer example are not relevant for the existence of a deadlock. Therefore, they can be abstracted away, which reduces the state space significantly. Yet another approach to reduce the state space is to limit the search to program executions with a small number of data objects, threads, or thread preemptions. Such limits compromise soundness, but practical experience shows that many concurrency errors may be detected with a small number of threads and preemptions. For instance, detecting the deadlock in method transfer requires only two Account objects, two threads, and one preemption.
iteratively refine the abstraction. In contrast to deductive verification, model checking is not limited by the expressiveness of a verification methodology. Model checkers are thus capable of analyzing arbitrary programs, including for instance optimistic concurrency, benevolent data races, and interlocked operations. In practice, software model checking can be inconclusive because the state space is too large, causing the model checker to abort. Although there is work on compositional model checking, most model checkers are not modular because it is difficult to summarize temporal properties at module boundaries. Model checking, including abstraction, is fully automatic. Users need to provide only the property to be checked. When the checking is not feasible with the available resources, users also need to manually simplify the program or provide upper limits on the number of objects, threads, etc. Due to the large state space of programs (in particular of parallel programs and programs that manipulate large data structures) and the non-modularity of the approach, model checking is typically too slow to be applied routinely during software development (in contrast to type checking and similar static analyses). However, for control-intensive programs with rather small data structures such as device drivers or embedded software, model checking is an efficient and very effective verification technique.
Discussion For finite transition systems (i.e., programs with a finite number of states such as programs with an upper bound on the number of threads, the number of dynamically allocated data objects, and the recursion depth, in particular, terminating programs), model checking is sound since all possible states and traces can be checked. However, some model checkers explore only a subset of the possible states of a program execution based on heuristics that are likely to detect errors; this approach trades efficiency for soundness. Model checking for finite transition systems is in principle complete, although abstraction may lead to spurious errors. Counterexample-guided abstraction refinement [] is a technique to recover completeness by using the counterexample traces of spurious errors to
Examples Even though most software model checkers focus on sequential programs, there are a number of tools available that check concurrency errors. JavaPathFinder [] is an explicit state model checker for concurrent Java programs that has been applied in industry and academia. It detects data races and deadlocks as well as violations of user-written assertions and LTL properties. MAGIC [] checks whether a labeled transition system is a safe abstraction of a C procedure. The C procedure may invoke other procedures which are themselves specified in terms of state machines, which enables compositional checking. MAGIC uses counterexampleguided abstraction refinement to reduce the state space of the system.
F
F
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
BLAST [] finds data races in C programs. It combines counterexample-guided abstraction refinement and assume-guarantee reasoning to infer a suitable model for thread contexts, which results in a low rate of false positives. The Zing tool [] checks for data races, deadlocks, API usage rules, and other safety properties. Its distinctive features are the use of procedure summaries to make the model checking compositional and the use of Lipton’s theory of reduction [] to prune irrelevant thread interleavings. CHESS [] is a tool for systematically enumerating thread interleavings in order to find errors such as data races, deadlocks, and livelocks in Win DLLs and managed.NET code. The analysis is complete, that is, every error found by CHESS is possible in an execution of the program. CHESS uses preemption bounding, that is, limits the number of thread preemptions per program execution to reduce the state space. Modulo this preemption bound, CHESS is sound; all remaining errors require more preemptions than the bound. Soundness is achieved by capturing all sources of non-determinism, including memory model effects. Model checkers for message-passing concurrency include for instance MPI-Spin [], which check Promela models of MPI programs, and ISP [], which operates directly on the MPI/C source code. Both tools detect deadlocks and assertion violations; MPI-Spin also checks liveness properties.
Static Program Analysis Overview Static analyzers compute static approximations of the possible behaviors of a program and then check properties of these approximations. Common forms of static analysis include, among others, data flow analysis and type systems. Both techniques are typically described formally as sets of inference rules that allow the analyzer to infer properties of execution states, for instance, the set of possible values of a variable. When annotations are given by the programmer, these rules are also used to check the annotations. Both data flow analysis and type systems have been applied extensively to detect concurrency errors.
Static data flow analyses typically apply the lockset algorithm to detect races. They compute the set of locks each thread holds when accessing any given variable. If for each variable the locksets of all threads are not disjoint then there is mutual exclusion on the variable accesses (see entry on Race Detection Techniques for details). In the example above, the static analysis would render the field accesses safe if it can show that for all accesses to x.balance and x.transactionCount, the executing thread holds at least the lock associated with the object x. Static analyses detect atomicity violations using Lipton’s theory []. Via a classification of operations into right and left movers, an analysis can show that a given code block is atomic, that is, any execution of the block is equivalent to an execution without interference from other threads. For this purpose, the analysis must show that variable accesses are both left and right movers, which amounts to showing the absence of data races. In the example, the conditional statement does not access any shared variable and is, thus, a both-mover. The acquire statements are right movers, the release statements are left movers, and the field accesses are both-movers provided that there are no data races. Consequently, the method is atomic. Deadlocks can be detected by computing a static lock-order graph and checking this graph to be acyclic. A deadlock analysis would complain about the transfer example if it cannot rule out that two threads execute transfer concurrently with reversed arguments. Most data flow analyses infer the information needed to check for concurrency errors. This also involves the computation of a whole range of additional program properties, which are required for these checks. For instance, the analyses require alias information in order to decide whether to expressions refer to the same object at run time. A may-alias analysis is for instance used for sound deadlock checking (to check whether two threads potentially compete for the same lock), whereas a must-alias analysis is required for sound race checking (to check whether two locksets definitely overlap). Another common auxiliary analysis is thread-escape analysis which distinguishes thread-local variables from shared variables; this is for instance useful to suppress warnings when accesses to thread-local variables are not guarded by any lock.
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
Discussion Static analyses are in principle sound. However, many existing tools sacrifice soundness in favor of efficiency. For instance, several data race analyses omit a mustalias analysis, which is necessarily flow-sensitive, and, therefore, cannot check soundly whether two locksets overlap. Static analyses are by nature incomplete since they over-approximate the possible executions of a program. Another source of incompleteness is that static analyses – much like deductive verification – check whether programs comply with a given methodology, such as lock-based synchronization or static lock ordering. Programs that do not comply might lead to false positives. Whether or not a static analysis is modular depends mostly on the existence of program annotations. Most data flow analyses are whole-program analyses, but there exist a number of analyses that compute procedure summaries to enable modular checking. By contrast, type systems typically require annotations, which are then checked modularly. Aside from annotations that may be required from programmers, static analyses are fully automatic. However, they often produce a large number of false positives, which need to be inspected manually or filtered aggressively (which may compromise soundness). Static analyses are extremely efficient. Many static analyzers have been applied to programs consisting of hundreds of thousands or even millions of lines of code. This scale is orders of magnitudes larger than for deductive verification and model checking. In summary, static analysis tools typically favor efficiency over soundness and completeness. This choice makes them ideal for finding errors in large applications, whereas deductive verification and model checking are more appropriate when strong correctness guarantees are needed. Examples Warlock [], RacerX [], and Relay [] use the lockset algorithm to detect data races in C programs. RacerX also computes a static lock-order graph to detect deadlocks. It uses various heuristics to improve the analysis results, for instance, to identify benign races. Relay is a modular analysis that limits the sources of unsoundness to four very specific sources, one of them being the syntactic filtering of warnings to reduce
F
the number of false positives. RacerX and Relay have been applied to millions of lines of code such as the Linux kernel. Chord [] is a context-sensitive, flow-insensitive race checker for Java that supports three common synchronization idioms: lexically scoped lock-based synchronization, fork/join, and wait/notify. Since Chord does not perform a must-alias analysis, it is not sound. Chord has been applied to hundreds of thousands of lines of code. rccjava [] uses a type system to check for data races in Java programs. The annotations required from the programmer include declarations of thread-local variables and of the locks that guard accesses to a shared variable. The type system has been extended to atomicity checking []. rccJava scales to hundreds of thousands of lines of code.
Related Entries Deadlocks Determinacy Intel Parallel Inspector Monitors, Axiomatic Verification of Race Conditions Race Detectors for Cilk and Cilk++ Programs Race Detection Techniques Owicki-Gries Method of Axiomatic Verification
Bibliographic Notes and Further Reading This entry focuses entirely on tools that detect concurrency errors present in programs. Two related areas are () tools that prevent concurrency errors during the construction of the program and () tools that detect concurrency errors in other artifacts. Tools of the first kind support correctness by construction. Programs are typically constructed through a stepwise refinement of some high-level specification to executable code. Since the code is derived systematically from a formal specification, errors including concurrency errors are prevented. Interested readers may start from Abrial’s book []. Tools of the second kind do not operate on programs but on other representations of software and hardware. Races and deadlocks may occur for instance also in workflow graphs; they can be detected by the
F
F
Formal Methods–Based Tools for Race, Deadlock, and Other Errors
approaches described here, but also through Petri-net and graph algorithms []. Similarly, hardware can be verified not to contain concurrency errors such as deadlock [], an area in which model checking has been applied with great success.
Acknowledgments Thanks to Felix Klaedtke and Christoph Wintersteiger for their helpful comments on a draft of this entry.
Bibliography . Abrial J-R () Modeling in Event-B. Cambridge University Press, Cambridge . Andrews T, Qadeer S, Rajamani SK, Xie Y () Zing: exploiting program structure for model checking concurrent software. In: Gardner P, Yoshida N (eds) Concurrency Theory (CONCUR). Lecture Notes in Computer Science, vol . Springer, Berlin, pp – . Bryant R, Kukula J () Formal methods for functional verification. In: Kuehlmann A (ed) The best of ICCAD: years of excellence in computer aided design. Kluwer, Norwell, pp – . Calcagno C, Parkinson MJ, Vafeiadis V () Modular safety checking for fine-grained concurrency. In: Nielson HR, Filé G (eds) Static analysis (SAS). Lecture Notes in Computer Science, vol . Springer, Berlin, pp – . Chaki S, Clarke EM, Groce A, Jha S, Veith H () Modular verification of software components in C. IEEE Trans. Softw Eng ():– . Clarke EM, Grumberg O, Jha S, Lu Y, Veith H () Counterexample-guided abstraction refinement for symbolic model checking. J ACM ():– . Cohen E, Dahlweid M, Hillebrand M, Leinenbach D, Moskal M, Santen T, Schulte W, Tobies S () VCC: a practical system for verifying concurrent C. In: Berghofer S, Nipkow T, Urban C, Wenzel M (eds) Theorem proving in higher order logics (TPHOLs ). Lecture Notes in Computer Science, vol . Springer, Berlin, pp – . Engler DR, Ashcraft K () RacerX: effective, static detection of race conditions and deadlocks. In: Scott ML, Peterson LL (eds) Symposium on operating systems principles (SOSP). ACM, New York, pp – . Flanagan C, Freund SN () Type-based race detection for Java. In: Proceedings of the ACM conference on programming language design and implementation (PLDI). ACM, New York, pp – . Flanagan C, Freund SN, Lifshin M, Qadeer S () Types for atomicity: static checking and inference for Java. ACM Trans Program Lang Syst ():– . Henzinger TA, Jhala R, Majumdar R () Race checking by context inference. In: Pugh W, Chambers C (eds) Programming Language Design and Implementation (PLDI). ACM, New York, pp –
. Jacobs B, Leino KRM, Piessens F, Schulte W, Smans J () A programming model for concurrent object-oriented programs. ACM Trans Program Lang Syst ():– . Jacobs B, Piessens F () The VeriFast program verifier. Technical Report CW-, Department of computer science, Katholieke Universiteit Leuven . Jones CB () Specification and design of (parallel) programs. In: Proceedings of IFIP’. North-Holland, pp – . Leino KRM, Müller P, Smans J () Verification of concurrent programs with Chalice. In: Aldini A, Barthe G, Gorrieri R (eds) Foundations of security analysis and design V. Lecture Notes in Computer Science, vol . Springer, Berlin, pp – . Lipton RJ () Reduction: a method of proving properties of parallel programs. Commun ACM ():– . Musuvathi M, Qadeer S, Ball T, Basler G, Nainar PA, Neamtiu I () Finding and reproducing heisenbugs in concurrent programs. In: Draves R, van Renesse R (eds) Operating systems design and implementation (OSDI). USENIX Association, pp – . Naik M, Aiken A, Whaley J () Effective static race detection for Java. In: Schwartzbach MI, Ball T (eds) Programming language design and implementation (PLDI), ACM, pp – . O’Hearn PW () Resources, concurrency, and local reasoning. Theor Comput Sci (–):– . Siegel SF () Model checking nonblocking MPI programs. In: Cook B, Podelski A (eds) Verification, model checking, and abstract interpretation (VMCAI). Lecture Notes in Computer Science, vol , pp – . Sterling N () Warlock – a static data race analysis tool. In: USENIX Winter, pp – . Tofan B, Bäumler S, Schellhorn G, Reif W () Verifying linearizability and lock-freedom with temporal logic. Technical report, Fakultät für Angewandte Informatik der Universität Augsburg, . van der Aalst WMP, Hirnschall A, Verbeek HMWE () An alternative way to analyze workflow graphs. In: Pidduck AB, Mylopoulos J, Woo CC, Özsu MT (eds) Advanced information systems engineering (CAiSE). Lecture Notes in Computer Science, vol . Springer, pp – . Villard J, Lozes É, Calcagno C () Tracking heaps that hop with Heap-Hop. In: Esparza J, Majumdar R (eds) Tools and algorithms for the construction and analysis of systems (TACAS). Lecture Notes in Computer Science, vol . Springer, – . Visser W, Havelund K, Brat GP, Park S, Lerda F () Model checking programs. Autom Softw Eng ():– . Vo A, Vakkalanka S, DeLisi M, Gopalakrishnan G, Kirby RM, Thakur R () Formal verification of practical MPI programs. In: Principles and practice of parallel programming (PPoPP). ACM, pp – . Voung JW, Jhala R, Lerner S () Relay: static race detection on millions of lines of code. In: Crnkovic I, Bertolino A (eds) European software engineering conference and foundations of software engineering (ESEC/FSE). ACM, pp –
Fortran and Its Successors
Fortran and Its Successors Michael Metcalf Berlin, Germany
Definition Fortran is a high-level programming language that is widely used in scientific programming in the physical sciences, engineering, and mathematics. The first version was developed by John Backus at IBM in the s. Following an initial, rapid spread in its use, the language was standardized, first in and, after many intermediate standards, most recently in . The language is a procedural, imperative, compiled language with a syntax well suited to a direct representation of mathematical formulae. Individual procedures may either be grouped into modules or compiled separately, allowing the convenient construction of very large programs and of subroutine libraries. Fortran contains features for array processing, abstract and polymorphic data types, and dynamic data structures. Compilers typically generate very efficient object code, allowing an optimal use of computing resources. Modern language provide extensive facilities for object-oriented and parallel programming. Earlier versions have relied on auxiliary standards to facilitate parallel programming.
Discussion The Development of Fortran Historical Fortran Backus, at the end of , began the development of the Fortran programming language (the name being an acronym derived from FORmula TRANslation), with the objective of producing a compiler that would generate efficient object code. The first version, known as FORTRAN I, contained early forms of constructs that have survived to the present day: simple and subscripted variables, the assignment statement, a DO-loop, mixedmode arithmetic, and input/output (I/O) specifications. Many novel compiling techniques had to be developed. Based on the experience with FORTRAN I, it was decided to introduce a new version, FORTRAN II, in . The crucial differences between the two were the
F
introduction of subprograms, with their associated concept of shared data areas, and separate compilation. In , FORTRAN IV was released. It contained type statements, the logical-IF statement, and the possibility to pass procedure names as arguments. In , a standard for FORTRAN, based on FORTRAN IV, was published. This was the first programming language to achieve recognition as an American and subsequently international standard, and is now known as FORTRAN . The permissiveness of the standards led to a proliferation of dialects and a new revision was published in , becoming known as FORTRAN . It was adopted by the International Standards Organization (ISO) shortly afterwards. The new standard brought with it many new features, for instance the IF... THEN... ELSE construct, a character data type, and enhanced I/O facilities. Although only slowly adopted, this standard eventually gave Fortran a new lease of life that allowed it to maintain its position as the most widely used scientific programming language.
Modern Fortran Fortran
Fortran’s strength had always been in the area of numerical, scientific, engineering, and technical applications. In order that it be brought properly up to date, an American subcommittee, working as a development body for the ISO committee ISO/IEC JTC/SC/WG (or simply WG), prepared a further standard, now known as Fortran . It was developed in the heyday of the supercomputer, when large-scale computations using vectorized codes were being carried out on machines like the Cray and CDC Cyber , and the experience gained in this area was brought into the standardization process by representatives of computer hardware and software vendors, users, and academia. Thus, one of the main features of Fortran was the array language, built on whole array operations and assignments, array sections, intrinsic procedures for arrays, and dynamic storage. It was designed with optimization in mind. Another was the notion of abstract data types, built on modules and module procedures, derived data types (structures), operator overloading and generic interfaces, together with pointers. Also important were new facilities for
F
F
Fortran and Its Successors
numerical computation including a set of numeric inquiry functions, the parameterization of the intrinsic types, new control constructs (SELECT CASE and new forms of DO), internal and recursive procedures, optional and keyword arguments, improved I/O facilities, and new intrinsic procedures. The resulting language was a far more powerful tool than its predecessor and a safer and more reliable one too. Subsequent experience showed that Fortran compilers detected errors far more frequently than previously, resulting in a faster development cycle. The array syntax in particular allowed compact code to be written, in itself an aid to safe programming. Fortran was published by ISO in . Fortran
Following the publication of Fortran , the Fortran standards committees continued to operate, and decided on a strategy whereby a minor revision of Fortran would be prepared, establishing a principle of having major and minor revisions alternating. At the same time, the High-Performance Fortran Forum (HPFF) was founded. The HPFF was set up in an effort to define a set of extensions to Fortran such that it would be possible to write portable, single-threaded code when using parallel computers for handling problems involving large sets of data that can be represented by regular grids, allowing efficient implementations on SIMD (SingleInstruction-Multiple-Data) architectures. This version of Fortran was to be known as High Performance Fortran (HPF), and it was quickly decided, given its array features, that Fortran should be its base language. Thus, the final form of HPF was of a superset of Fortran , the main extensions being in the form of directives. However, HPF included some new syntax too and, in order to avoid the development of divergent dialects of Fortran, it was agreed to include some of this new syntax into Fortran . Apart from this syntax, only a small number of other pressing but minor changes were made, and Fortran was adopted as a formal standard in . It is currently the most widely used version of Fortran. Auxiliary Standards
With few exceptions, no Fortran standard up to and including Fortran included any feature intended
directly to facilitate parallel programming. Rather, this has had to be achieved through the intermediary of ad hoc industry standards, in particular HPF, MPI, OpenMP, and Posix Threads. The HPF directives, already introduced, take the form of Fortran comment lines that are recognized as directives only by an HPF processor. An example is !HPF$ ALIGN WITH b :: a1, a2, a3 to align three conformable arrays with a fourth, thus ensuring locality of reference. Further directives allow, for instance, aligned arrays to be distributed over a set of processors. MPI is a library specification for message passing. OpenMP supports multi-platform, shared-memory parallel programming and consists of a set of compiler directives, library routines, and environment variables that determine run-time behavior. Posix Threads is again a library specification, for multithreading. In any event, outside Japan, HPF met with little success, whereas the use of MPI and OpenMP has become widespread. Fortran
The following language standard, Fortran , was published in . Its main features were: the handling of floating-point exceptions, based on the IEEE model of floating-point arithmetic; allowing allocatable arrays as structure components, dummy arguments, and function results; interoperability with C; parameterized data types; object-orientation via constructors/destructors, inheritance, and polymorphism; derived type I/O; asynchronous I/O; and procedure variables. Although all this all represented a major upgrade, especially given the introduction of object-oriented programming, there was no new development specifically related to parallel programming. Fortran
Notwithstanding the fact that Fortran -conformant compilers had been very slow to appear, the standardization committees thought fit to proceed with yet another standard, Fortran . In contrast to the previous one, its single most important new feature is directly related to parallel processing – the addition of coarray handling facilities. Further, the DO CONCURRENT form of loop control and the CONTIGUOUS attribute
Fortran and Its Successors
are introduced. These are further described below. Other major new features include: sub-modules, enhanced access to data objects, enhancements to I/O and to execution control, and more intrinsic procedures, in particular for bit processing. Fortran was published in .
Selected Fortran Features Given that all the actively supported Fortran compilers on the market offer at least the whole of Fortran , that version is the one used as the basis of the descriptions in this section, except where otherwise stated. Only those features are described that are relevant to vectorization or parallelization. Under Fortran , achieving effective parallelization requires the exploitation of a combination of efficient Fortran serial execution on individual processors with, say, MPI, controlling the communication between processors.
The DO Constructs Many iterative calculations can be carried out in the form of a DO construct. A simplified but sufficient nested form is illustrated by: outer: DO inner: DO i = j, k, m ! from j to ! k in steps of m ! (m is optional) : IF (...) CYCLE ! take ! next iteration of ! construct ‘inner’ : IF (...) EXIT outer ! exit ! the construct ‘outer’ END DO inner END DO outer
(The Fortran language is case insensitive; however, to enhance the clarity of the code examples, Fortran keywords are distinguished by being written in upper case.) As shown, the constructs may be optionally named so that any EXIT or CYCLE statement may specify which loop is intended. Potential data dependencies between the iterations of a DO construct can inhibit optimization, including vectorization. To alleviate this problem, Fortran introduces the DO CONCURRENT form of the DO construct, as in
F
DO CONCURRENT (i = 1:m) a(k + i) = a(k + i) + scale * a(n + i) END DO
that asserts that there is no overlap between the range of values accessed (on the right-hand side of the assignment) and the range of values altered (on the left-hand side). The individual iterations are thus independent.
F
Array Handling Array Variables
Arrays are variables in their own right; typically they are specified using the DIMENSION attribute. Every array is characterized by its type, rank (dimensionality), and shape (defined by the extents of each dimension). The lower bound of each dimension is by default , but arbitrary bounds can be explicitly specified. The DIMENSION keyword is optional; if omitted, the array shape must be specified after the array-variable name. The examples REAL :: x(100) INTEGER, DIMENSION(0:1000, -100:100) :: plane
declare two arrays: x is rank- and plane is rank-. Fortran arrays are stored with elements in columnmajor order. Elements of arrays are referenced using subscripts, for example, x(n)
x(i*j)
Array elements are scalar. The subscripts may be any scalar integer expression. A section is a part of an array variable, referenced using subscript ranges, and is itself an array: x(i:j) plane(i:j, k:l:m) x(plane(7, k:l)) x(5:4)
! ! ! !
rank one rank two vector subscript zero zero length
Array-valued constants and variables (array constructors) are available, enclosed in (/ ... /)or, in Fortran , in square brackets, [ ]: (/ 1, 2, 3, 4, 5 /) (/ ( (/ 1, 2, 3 /), i = 1, 10) /) (/ (i, i = 1, limit, 2) /) [ (0, i = 1, 1000) ] [ (0.01*i, i = 1, 100) ]
F
Fortran and Its Successors
They may make use of an implied-DO loop notation, as shown, and can be freely used in expressions and assignments. A derived data type may contain array components. Given
REAL, DIMENSION(10, 20) :: x, y, z REAL, DIMENSION(5) :: a, b
it can be written:
[lower ] : [upper ] [:stride ]
! ! z = x/y ! ! ! z = 10.0 ! ! ! b = a + 1.3 ! ! ! b = 7/a + x(1:5, 5) ! ! ! ! z(1:8, 5:10) = x(2:9, z(1:8, ! ! ! a(2:5) = a(1:4) ! ! !
where [ ] indicates an optional item. Examples for an array x(10, 100) are
An array of zero size is a legitimate object, and is handled without the need for special coding.
x(i, 1:n) x(1:m, j) x(i, : ) x(i, 1::3)
Elemental Procedures
TYPE point REAL, DIMENSION(3) :: vertex END TYPE point TYPE(point), DIMENSION(10) :: points TYPE(point), DIMENSION(10, 10):: & points_array points(2) is a scalar (a structure) and points(2) %vertex is an array component of a scalar. A reference like points_array(n,2) is an element (a scalar) of type point. However, points_array(n,2) %vertex(2) is an array of type real, and points_array(n,2)%vertex(2) is a scalar element of it. An array element always has a subscript or subscripts qualifying at least the last name. Arrays of arrays are not allowed. The general form of subscripts for an array section is
x(i, 100:1:-1) x((/1, 7, 3/), 10) x(:, 1:7)
! ! ! ! ! ! ! ! ! !
part of one row part of one column whole row i every third element of row i row i in reverse order vector subscript rank-2 section
A given value in an array can be referenced both as an element and as a section: x(1, 1) x(1:1, 1)
! !
scalar (rank zero) array section (rank one)
depending on the circumstances or requirements. Operations and Assignments
As long as they are of the same shape (conformable), scalar operations and assignments are extended to arrays on an element-by-element basis. For example, given declarations of
x = y
whole array assignment whole array division and assignment whole array assignment of scalar value whole array addition to scalar value array division, and addition to section 5:10) + & 15:20) array section addition and assignment overlapping section assignment
In an assignment like y = SQRT(x)
an intrinsic function, SQRT, returns an array-valued result for an array-valued argument. It is an elemental procedure. Elemental procedures may be called with scalar or array actual arguments. The procedure is applied to array arguments as though there were a reference for each element. In the case of a function, the shape of the result is the shape of the array arguments. Most intrinsic functions are elemental and Fortran extended this feature to non-intrinsic procedures. It is a further an aid to optimization on parallel processors. An elemental procedure requires the ELEMENTAL attribute and must be pure (i.e., have no side effects, see below).
Fortran and Its Successors
Array-Valued Functions and Arguments
Users can write functions that are array valued; they require an explicit interface and are usually placed in a module. This example shows such a module function, accessed by a main program: MODULE show CONTAINS FUNCTION func(s, t) REAL, DIMENSION(:) :: s, t ! An assumed-shape array, see below REAL, DIMENSION(SIZE(s)) :: func func = s * t**2 END FUNCTION func END MODULE show PROGRAM example USE show REAL, DIMENSION(3) :: x = (/ 1., 2., 3./), y = (/ 3., 2., 1. /), s s = func(x, y) PRINT *, s END PROGRAM example
&
F
Fortran and Its Successors. Table Classes of intrinsic procedures, with examples Class of intrinsic array function Examples Numeric
abs, modulo
Mathematical
acos, log
Floating-point manipulation
exponent, scale
Vector and matrix multiply
dot_product, matmul
Array reduction
maxval, sum
Array inquiry
allocated, size
Array manipulation
cshift, eoshift
Array location
maxloc, minloc
Array construction
merge, pack
Array reshape
reshape
the corresponding dummy argument specification must define the type and rank of the array, but it may omit the shape. Thus in SUBROUTINE calc(d) REAL, DIMENSION(:, :) :: d
In order to allow for optimization, especially on parallel and vector machines, the order of expression evaluation in executable statements is not specified. However, some potential optimizations might be lost if it is not certain that an array is in contiguous storage; for instance, if the array is an assumed-shape dummy argument array or a pointer array. In Fortran , it is open to a programmer to assert that an array is contiguous by adding the CONTIGUOUS attribute to its specification, as in
it is as if d were dimensioned (, ). The shape, not the bounds, is passed, meaning that the default lower bound is and the default upper bound is the corresponding extent. However, any lower bound can be specified and the array maps accordingly. Assumed-shape arrays allow great flexibility in array argument passing.
REAL, CONTIGUOUS, DIMENSION(:, :) :: x
SUBROUTINE exchange(x, y) REAL, DIMENSION(:) :: x, y REAL, DIMENSION(SIZE(x)) :: work work = x x = y y = work END SUBROUTINE exchange
The many intrinsic functions that accept arrayvalued arguments should be regarded as an integral part of the array language. A brief summary of their categories appears in Table .
Assumed-Shaped Arrays
Given an actual argument in a procedure reference, as in: REAL, DIMENSION(0:10, 0:20) :: x : CALL calc(x)
Automatic Arrays
Automatic arrays are useful for defining local, temporary work space in procedures, as in
Here, the array work is created on entering the procedure and destroyed on leaving. The actual storage is typically maintained on a stack. Allocatable Arrays
Fortran provides dynamic allocation of storage, commonly implemented using a heap storage mechanism.
F
F
Fortran and Its Successors
An example, for establishing a work array for a whole program, is MODULE global INTEGER :: n REAL, DIMENSION(:,:), ALLOCATABLE :: & work END MODULE global PROGRAM analysis USE global READ (*, *) n ALLOCATE(work(n, 2*n),STAT=status) : DEALLOCATE (work)
The work array can be propagated throughout the whole program via a USE statement in each program unit. An explicit lower bound may be specified and several entities may be allocated in one statement. Deallocation of arrays is automatic when they go out of scope.
Masked Assignment
Often, there is a need to mask an assignment. This can be done using either a WHERE statement: WHERE (x /= 0.0) x = 1.0/x an example that avoids division by zero by replacing only non-zero values by their reciprocals, or a WHERE construct: WHERE (x /= 0.0) x = 1.0/x y = 0.0 ! all arrays assumed conformable ELSEWHERE x = HUGE(0.) y = 1.0 - y END WHERE
The ELSEWHERE statement may also have a mask clause, but at most one ELSEWHERE statement can be without a mask, and that must be the final one; WHERE constructs may be nested within one another.
The FORALL Statement and Construct
When a DO construct is executed, each successive iteration is performed in order and one after the other – an impediment to optimization on a parallel processor. To help in this situation, Fortran introduced the
FORALL statement and construct, which can be considered as an array assignment expressed with the help of indices. An example is FORALL(i = 1:n) x(i, i) = s(i)
where the individual assignments may be carried out in any order, and even simultaneously. In FORALL(i=1:n, j=1:n, y(i,j)/=0.) x(j,i) = 1.0/y(i,j)
the assignment is subject also to a masking condition. The FORALL construct allows several assignment statements to be executed in order, as in FORALL(i = 2:n-1, j = 2:n-1) x(i,j) = x(i,j-1) + x(i,j+1) + x(i-1,j) + x(i+1,j) y(i,j) = x(i,j) END FORALL
Assignment in a FORALL is like an array assignment: it is as if all the expressions were evaluated in any order, held in temporary storage, then all the assignments performed in any order. The first statement in a construct must fully complete before the second can begin. Procedures referenced within a FORALL must be pure (see below). In general, the FORALL has been poorly implemented and the DO CONCURRENT is seen as a better alternative: using DO CONCURRENT requires the programmer to ensure that the iterations are independent, whereas, with FORALL, the compiler is expected to determine whether a temporary copy is needed, which is often unrealistic.
Pure Procedures
This is another Fortran feature expressly for parallel computing, ensuring that execution of a procedure referenced from within a FORALL statement or construct (or, in Fortran , from within a DO CONCURRENT construct) cannot have any side effects (whereby the result of one reference could cause a change to the result of another). Any such side effects in a function could impede optimization on a parallel processor – the order of execution of the assignments could affect the results. To control this situation, a PURE keyword may be added to the SUBROUTINE or FUNCTION statement – an assertion that the procedure: alters no global variable;
Fortran and Its Successors
performs no input/output operations; has no saved variables; and, in the case of functions, alters no argument. These constraints are designed such that they can be verified by a compiler. All the intrinsic functions are pure. The FORALL statement and construct and the PURE keyword were adopted from HPF. Coarrays (Fortran Only)
The objective of coarrays is to distribute over some number of processors not only data, as in an SIMD model, but also work, using the SPMD (SingleProgram-Multiple-Data) model. The syntax required to implement this facility has been designed to make the smallest possible impact on the appearance of a program and to require a programmer to learn just a modest set of new rules. Data distribution is achieved by specifying the relationship among memory images using an elegant new syntax similar to conventional Fortran. Any object not declared using this syntax exists independently in all the images and can be accessed only from within its own image. Objects specified with the syntax have the additional property that they can be accessed directly from any other image. Thus, the statement REAL, DIMENSION()[∗] :: a, b specifies two coarrays, a and b, that have the same size () in each image. Execution by an image of the statement a(:) = b(:)[j]
causes the array b from image j to be copied into its own array a, where square brackets are the notation used to access an object on another image. On a shared-memory machine, an implementation of a coarray might be as an array of a higher dimension. On a distributed-memory machine with one physical processor per image, a coarray is likely to be stored at the same address in each physical processor. Work is distributed according to the concept of images, which are copies of the program that each have a separate set of data objects and a separate flow of control. The number of images is normally chosen on the command line, and its fixed value is available at execution time via an inquiry function, NUM_IMAGES. The images execute asynchronously and the execution
F
path in each may differ, possibly under the control of an image index to which the programmer has access (via the THIS_IMAGE function). When synchronization between two images is required, use can be made of a set of intrinsic synchronization procedures, such as SYNCH_ALL or LOCK. Thus, it is possible in particular to avoid race conditions whereby one image alters a value still required by another, or one image requires an altered value that is not yet available from another. Handling this is the responsibility of the programmer. Between synchronization points, an image has no access to the fresh state of any other image. Flushing of temporary memory, caches, or registers is normally handled implicitly by the synchronization mechanisms themselves. In this way, a compiler can safely take advantage of all code optimizations on all processors between synchronization points without compromising data integrity. Where it might be necessary to limit execution of a code section to just one image at a time, a critical section may be defined using a CRITICAL ... END CRITICAL construct. The codimensions of a coarray are specified in a similar way to the specifications of assumed-size arrays, and coarray sub-objects may be referenced in a similar way to sub-objects of normal arrays. The following example shows how coarrays might be used to read values in one image and distribute them to all the others: REAL :: val[*] ... IF(THIS_IMAGE() == 1) THEN ! Only image 1 executes this construct READ(*, *) val DO image = 2, NUM_IMAGES() val[image] = val END DO END IF CALL SYNC_ALL() ! Execution on all images ! pauses here until all ! images reach this point
Coarrays can be used in most of the ways that normal arrays can. The most notable restrictions are that they cannot be automatic arrays, cannot be used for a function result, cannot have the POINTER attribute, and cannot appear in a pure or elemental procedure.
F
F
Fortran, Connection Machine
Related Entries
Fortran, Connection Machine
Code Generation Data Distribution Dependences
Connection Machine Fortran
HPF (High Performance Fortran) Loop Nest Parallelization MPI (Message Passing Interface)
Fortress (Sun HPCS Language)
OpenMP Parallelization, Basic Block Run Time Parallelization
Bibliographic Notes and Further Reading The development of Fortran was very contentious and the entire issue of the journal [] is devoted to its various aspects. Fortran , is defined in the ISO publication []. Fortran was published in as []. Fortran and Fortran are informally but completely described in [] and [] and, with the latest additions contained in Fortran , in []. Coarrays have evolved from a programming model initially intended for the Cray-TD. They were extended to Fortran by Numrich and Steidel [] and subsequently defined for Fortran by Numrich and Reid []. Their formal definition is in the Fortran standard [] and they are described informally also in [].
Guy L. Steele Jr. , Eric Allen , David Chase , Christine Flood , Victor Luchangco , Jan-Willem Maessen , Sukyoung Ryu Oracle Labs, Burlington, MA, USA Oracle Labs, Austin, TX, USA Google, Cambridge, MA, USA Korea Advanced Institute of Science and Technology, Daejeon, Korea
Definition Fortress is a programming language designed to support parallel computing for scientific applications at scales ranging from multicore desktops and laptops to petascale supercomputers. Its generality and features for user extension also make it a useful framework for experimental design of domain-specific languages. The project originated at Sun Microsystems Laboratories in as part of their work on the DARPA HPCS (High Productivity Computing Systems) program [] and became an open-source project in .
Discussion Design Principles
Bibliography . Adams JC et al () The Fortran handbook. Springer, London/New York . () Comput Stand Interfaces . ISO/IEC -:. ISO, Geneva . ISO/IEC -:. ISO, Geneva . Metcalf M, Reid J, Cohen M () Fortran / explained. Oxford University Press, Oxford/New York . Metcalf M, Reid J, Cohen M () Modern Fortran explained. Oxford University Press, Oxford/New York . Numrich RW, Steidel JL () F−− : a simple parallel extension to Fortran . SIAM News ():– . Numrich RW, Reid JK () CoArray Fortran for parallel programming. ACM Fortran Forum : and Rutherford Appleton Laboratory report RAL-TR-–. ftp://ftp.numerical.rl. ac.uk/pub/reports/nrRAL.pdf
Fortress provides the programmer with a single global address space, an automatically managed heap, and automatically managed thread parallelism. Programs can also spawn threads explicitly. The three major design principles for Fortress are: ●
Mathematical syntax: Fortress syntax is inspired by mathematical tradition, using the full Unicode character set and two-dimensional notation where relevant. The goal is to simplify the desk-checking of code against what working scientists and mathematicians tend to write in their notebooks or on their whiteboards. ● Parallelism by default: Rather than grafting parallel features onto a sequential language design, Fortress
Fortress (Sun HPCS Language)
uses parallelism wherever possible and reasonable. The design intentionally makes sequential idioms a little more verbose than parallel idioms, and intentionally makes side effects a little more verbose than “pure” (side effect free) code. ● A growable language []: As few features as possible are built into the compiler, and as many language design decisions as possible are expressed at the source-code level. In many ways, Fortress is a framework for constructing domain-specific languages, and while the first such language is aimed at scientific programmers, the intent is that radically different design choices can also be supported. Fortress is a strongly typed language. The type system is object oriented, with parameterized polymorphic types but also with type inference so that programmers frequently need not declare the types of bound variables explicitly. The objects enjoy multiple inheritance of code but not of data fields. A Fortress trait is much like an interface in the JavaTM programming language but can contain concrete implementations of methods that can be inherited. A Fortress object can declare data fields and in other respects is also much like a Java class, but a Fortress object can extend (and therefore inherit from) only traits, not other objects. Methods may be overloaded; overload resolution is performed by multimethod dispatch using the dynamic types of all arguments. (In contrast, the Java programming language uses the dynamic type of the invocation target “before the dot” but the static types of the arguments in parentheses.) Fortress supports and favors the use of pure, functional object-oriented programming, but also supports side effects, including assignment to variables and to fields of objects that have been declared mutable. Atomic blocks are used to manage the synchronization of side effects. Fortress also favors divide-and-conquer approaches to parallelism (as opposed to, say, pipelining); generators split aggregate data structures and numeric ranges into portions that can be processed in parallel, and reducers combine or aggregate the results for further processing. Generators and reducers are classified according to their algebraic properties, such as associativity and commutativity; this classification is expressed through,
F
and enforced by, specific traits defined by a standard library, enabling a generator to choose from a variety of implementation strategies.
Salient Features Syntax Fortress syntax is designed to resemble standard mathematical notation as much as possible while integrating programming-language notions of variable binding, assignment, and control structures. All mathematical operator symbols in the Unicode character set [], such as ⊙ ⊕ ⊗ ≤ ≥ ∩ ∪ ⊓ ⊔ ⊆ ⊑ ≺ ≽ ∑ ∏ , may be defined and overloaded as user-defined infix, prefix, and postfix operators; a large number of bracketing symbols such as { } ⟨ ⟩ ⌜ ⌝ ⌞ ⌟ ⌊ ⌋ ⌈ ⌉ ∣ ∣ ∥ ∥ are likewise available for library or user (re-) definition. Relational operators may be chained, as in ≤ n ≤ ; the compiler treats it as if it had been written ( ≤ n) ∧ (n ≤ ) but arranges to evaluate the expression n just once. Simple juxtaposition is itself a binary operator that can be defined and overloaded by Fortress source code; the standard library defines juxtaposition to perform function application (as in f (x, y) or print x ), numerical multiplication (as in n and x y ), and string concatenation (as in “My name is” myName “.” ). Unicode characters may be used directly or represented in ASCII through a Wiki-inspired syntax that is reasonably readable in itself. Fortress also provides concise syntax to mimic mathematical notation: semicolons and parentheses may often be omitted, and curly braces { } denote sets (rather than blocks of statements). The notation is whitespace sensitive but not indentation sensitive. Fortress is expression oriented, and it favors the use of immutable bindings over mutable variables, requiring a slightly more verbose syntax for declaration of mutable variables and for assignment. The left-hand side of Fig. shows a sample Fortress function that performs a matrix/vector “conjugate gradient” calculation (based on the NAS “CG” conjugate gradient benchmark for parallel algorithms) []. The function conjGrad takes a matrix and a vector, performs a number of calculations, and returns two values: a result vector and a measure of the numerical error. The right-hand side of Fig. presents the original
F
F
Fortress (Sun HPCS Language)
conjGrad E extends Number, nat N, V extends VectorE, N, M extends MatrixE, N, N (A: M, x: V): (V, E) = do z: V := r: V := x ρ: E := r⋅r p: V := r for j ← seq( # ) do q=Ap α = ρ/(p⋅q) z += α p ρ = ρ r −= α q ρ := r⋅r β = ρ/ρ p := r + β p end (z, ∥x − A z∥) end
z= r=x ρ = rT r p=r DO i = , q =Ap α = ρ/(pT q) z=z+αp ρ = ρ r=r−αq ρ = rT r β = ρ/ρ p=r+βp ENDDO compute residual norm explicitly: ∥r∥ = ∥x − A z∥
Fortress (Sun HPCS Language). Fig. Sample Fortress code for a conjugate gradient calculation (left) and the original NAS CG pseudocode specification (right)
specification of the conjugate gradient calculation to show how the Fortress program resembles the problem specification. The principal differences are that the Fortress code provides type declarations – including a generic (type-parametric) function header – and distinguishes between binding and assignment of variables (within the loop body, q and α and ρ are readonly variables that are locally bound using = syntax, whereas z and r and ρ and p are mutable variables that are updated by the assignment operator := and the compound assignment operators += and −= ). Fortress is designed to grow over time to accommodate the changing needs of its users via a syntactic abstraction mechanism [] that, in effect, allows sourcecode declarations to augment the language grammar by specifying rewriting rules for new language constructs. Parsing of new constructs is done alongside parsing of primitive constructs, allowing programmers to detect syntax errors in use sites of new constructs early. Programs in domain-specific languages can be embedded in Fortress programs and parsed as part of their host programs. Moreover, the definition of many constructs
that are traditionally defined as core language primitives (such as for loops) can be moved into Fortress’ own libraries, thereby reducing the size of the core language.
Type System Fortress has a rich type system designed to give programmers the expressive power they need to write libraries that provide features often built into other languages. The Fortress type system is a trait-based, object-oriented system that supports parametrically polymorphic named types (also known as generic traits) organized as a multiple-inheritance hierarchy. Fortress code is statically type-checked: it is a static (“compiletime”) error if the type of an expression is not a subtype of the type required by its context. Objects may be declared to extend traits, from which they can inherit method declarations and definitions. If an object inherits an abstract method declaration, then it has the obligation to provide (or inherit) a concrete implementation of that method. A trait may also extend one or more other traits. Extension creates a subtype relationship; if type A extends type B, then every
Fortress (Sun HPCS Language)
F
object belonging to type A (that is, every instance of A) necessarily also belongs to type B. In addition to subtyping, Fortress allows the declaration of exclusion and comprises relationships among types. If type A excludes type B, then no value can be an instance of both A and B. If a type T comprises a set of types { U , U , . . . , Un } , then every instance of T is an instance of some Ui (possibly more than one). This allows a more flexible description of important relationships among types than simply the ability to declare that a class is final.
Tuples A tuple is written as a comma-separated
Types, values, and expressions Fortress provides several
Functions and Methods Functions are also first-class
kinds of types, including trait types, tuple types, arrow types. At the top of the type hierarchy is the type Any, which comprises three types, Object, Tuple, and () (pronounced “void”); at the bottom of the hierarchy is the type BottomType. Every expression in Fortress has a static type. Every value in Fortress is either an object, a tuple, or the value () , and has a (run-time) tag, or ilk, which is a “leaf ” in the type hierarchy: the only strict subtype of a value’s tag is BottomType. Fortress guarantees that when an expression is evaluated, the ilk of the resulting value is a subtype of the type of the expression. Traits Like an interface in the Java programming language, a trait is named program construct that declares a set of methods, and which may extend one or more other traits, inheriting their methods. However, unlike Java interfaces, whose methods are always abstract (i.e., they do not provide bodies), the methods of Fortress traits may be either abstract or concrete (having bodies). Like functions, methods may be overloaded, and a call to such a method is resolved using the run-time tags of its arguments. Objects Objects form the leaves of the trait hierarchy:
no trait may extend an object trait type. Thus object declarations are analogous to final classes in the Java programming language. In addition to methods, objects may have fields, in which state may be stored. Numbers and booleans and characters are all objects. This does not imply that they are always heapallocated. Objects declared with the value keyword have no object identity, must have only immutable fields, and may be freely copied by the implementation.
series of expressions within parentheses, for example, (a, , b + ) ; they are convenient for passing multiple arguments and for returning multiple results. Tuples are first-class values but are not objects. Tuple types are covariant: if type A extends type B, and C extends D, and E extends F, then the tuple type (A, C, E) extends the tuple type (B, D, F) . The type of a variable, of a field, or of array elements may be a tuple type. Tuples of variables may appear on the left-hand side of binding and assignment constructs.
values, and furthermore, they are objects. However, their types are not traits, but arrow types, constructed from other types using the arrow combinator → . A function that is declared to return an instance of type B whenever it is passed an instance of type A as an argument is an instance of the arrow type A → B . Because a function can be overloaded, having multiple definitions with different parameter types, it can be an instance of several different arrow types (Fig. ). Because arrow types are covariant in their return types and contravariant in their parameter types, every arrow type is a subtype of BottomType → Any and a supertype of Any → BottomType . Fortress provides two kinds of methods that may be declared in objects and traits: dotted methods, which are similar to methods in other objectoriented languages and are invoked using the syntax target.methodName(arguments), and functional methods, which are overloadings of global function names and therefore are invoked using the syntax methodName (arguments), but one of the arguments is treated as the target just as for a dotted method (Fig. ). Ignoring the possibility of automatic coercion, a function declaration is applicable to a call to that function if its parameter type is a supertype of (possibly equal to) the run-time tag of the argument, and it is more specific than another declaration if its parameter type is a strict subtype of the parameter type of the other. (If a declaration of a function or method takes more than one argument, then its “parameter type” (singular) is considered to be a tuple type.) The Java programming language allows any set of overloaded method declarations, resolves overloading based on the static types of the arguments and the
F
F
Fortress (Sun HPCS Language)
(* The function pad is an instance of the arrow type String → String and also of the arrow type Z → Z , neither of which is a subtype of the other. *) pad(x: String): String = “ ” x pad(y: Z): Z = y (* Arrow types can include the types of exceptions that can be thrown. The function pad′ in an instance of the arrow type String → String throws EmptyString and also of the arrow type Z → Z throws ZeroNumber . *) pad′ (x: String): String throws EmptyString = if x = “” then throw EmptyString else “ ” x “ ” end pad′ (y: Z): Z throws ZeroNumber = if y = then throw ZeroNumber else y end Fortress (Sun HPCS Language). Fig. Two examples of a function that belongs to more than one arrow type
(* This example object has one field, v, and four method definitions. The method add is a conventional “dotted method”; the method name maximum is overloaded with three functional method definitions. The parameter self indicates that a method is a functional method that is invoked using functional rather than dot syntax. *) object Example(v: Z) add(n: Z) = n + v maximum(self, n: Z) = if n > v then n else v end maximum(n: Z, self) = maximum(self, n) maximum(self, other: Example) = maximum(self, other.v) end
Dotted method (no self parameter) ⊛ Functional method ⊛
⊛
Functional method
⊛
Functional method
do o = Example() p = Example() ⊛ Examples of method invocation println o.add() ⊛ Prints “” println p.add() ⊛ Prints “” println maximum(o, ) ⊛ Prints “” println maximum(, o) ⊛ Prints “” println maximum(o, p) ⊛ Prints “” end Fortress (Sun HPCS Language). Fig. Example definition of a dynamically parameterized object
Fortress (Sun HPCS Language)
dynamic type of the target, and requires this resolution be unambiguous for calls that actually appear in the program (i.e., ambiguity is separately checked at every call site). In contrast, when a call to an overloaded Fortress function or method is executed, the most specific applicable declaration is selected based on the tags of the arguments passed to the function; that is, Fortress performs full multimethod dynamic dispatch on the target and all arguments. Furthermore, Fortress imposes rules on the set of overloaded declarations so that dispatch is always unambiguous (i.e., ambiguity is checked for each set of declarations and no separate check is needed at each call site). In particular, any pair of declarations of overloaded functions of method must satisfy one of three conditions: Subtype rule: The parameter type of one declaration is a strict subtype of the parameter type of the other, and the return type of the first declaration is a subtype of (possibly the same as) the return type of the second declaration. Exclusion rule: The parameter types of the two declarations exclude each other. Meet rule: The parameter types of the two declarations are incomparable (there is neither a subtype nor an exclusion relationship between them) and there is another declaration of that function whose parameter type “covers” the intersection of the parameter types of the two incomparable declarations. If all pairs of declarations satisfy one of these conditions, there can be no ambiguity for any particular call to that function or method: in the first case, one declaration is more specific than the other; in the second case, there is no call to which both declarations are applicable; and in the third case, whenever both declarations are applicable, there is another declaration that is more specific than both that is also applicable, so neither of the two incomparable declarations can be the most specific applicable declaration. These rules are how the design of Fortress solves the problem of multiple inheritance of methods. Access to fields of objects is abstract: the field access syntax x.fieldName actually invokes a getter method that takes no arguments but returns a value. Declaring a field by default provides a trivial getter method that simply fetches the contents of the field, but this method can be suppressed or displaced by
F
an explicit getter declaration. Similarly, the assignment operator := actually invokes a setter method, and the declaration of a mutable field by default provides a trivial setter method. In this manner, the behavior of field access and assignment is completely programmable.
Parametric polymorphism (or Generic types and functions) Fortress supports generic traits (and objects), as
well as generic functions (and methods). All these entities can have static parameters of various kinds: type parameters (for which any type can serve as an actual argument), nat and int and bool parameters (for which static arithmetic and Boolean expressions can serve as actual arguments), unit and dim parameters (for which declared physical dimensions and units can serve as actual arguments), and opr parameters (for which operator symbols can serve as actual arguments). Type parameters may be given one or more bounds, thereby requiring that they be instantiated only with subtypes of all the bounds. If no bound is given, it is implicitly assumed to be bound by Object. Note that Object is not the root of the Fortress type hierarchy; a type parameter that may be instantiated with a tuple type, for example, must be given an explicit bound (Fig. ). This unusual default for the bounds of type parameters helps to reduce confusion when functions and methods are overloaded with different numbers of parameters. The Java programming language has a system of generic types that erases type parameter information at run time. In contrast, Fortress types are not erased, and it is possible to perform the equivalent of an instanceof check to distinguish a list of strings from a list of booleans. Thus, there is no significant disparity between the static type system and the run-time tags associated with Fortress values: tags are simply a special class of types corresponding to leaves in the type hierarchy. In addition to standard parametric polymorphism, Fortress provides where clauses, in which hidden type variables can be declared and used within a trait (or object) or function declaration; they are called “hidden” because they are not static parameters of the trait or function – rather, they are universally quantified within their declared bounds.
F
F
Fortress (Sun HPCS Language)
(* The trait LinkedList can contain elements of type T. Because the declared bound on the type T is Any, the actual element type may be a tuple type. Because of the comprises clause, only instances of the object types ConsT and Empty can be instances of LinkedListT . *) trait LinkedList T extends Any comprises { ConsT, Empty } end (* A Cons instance has two fields, first and rest. This object declaration implicitly declares a constructor function called Cons that takes two arguments and creates a new instance with its two fields initialized to the given arguments. *) object Cons T extends Any (first: T, rest: ListT) extends LinkedListT end The single object Empty belongs to all LinkedList types. object Empty extends LinkedListT where { T extends Any } end ⊛
Fortress (Sun HPCS Language). Fig. Example definitions of a statically parameterized trait and two objects that implement it
Having to write out all static type arguments and complete type signatures for all functions and methods, especially local “helper” functions, can be tedious, and often results in code that is difficult to read and maintain. To mitigate these problems, Fortress (like ML and Haskell) has a type-inference mechanism that allows many types and static parameters to be elided in practice. Coercion A trait T may contain one or more coercion declarations; such a declaration specifies a computation that, when given any instance of another type U, will return a corresponding instance of type T. Such declarations should be used with care, only in situations where it is desirable that every instance of type U also be regarded as, in effect, an instance of type T in every computational context whatsoever. Typically this is used to convert among specialized numerical representations, and in particular to convert numeric literals to other numeric types. (Fortress in unusual in that the type of a literal such as is not any of the standard computational integer types such as Z or Z or Z , but a special numeric-literal trait.) Coercion comes into play during multimethod dispatch: if there is no applicable function or method declaration for a given call, then the net is cast wider by considering declarations that would be applicable if the arguments were automatically coerced to some other type; if there is a unique most specific applicable
declaration in this larger set, then that is the one that will be invoked. To help prevent surprises, coercions in Fortress are not transitive.
Parallelism Many constructs in the Fortress language give rise to multiple implicit tasks to execute subexpressions, such as the components of a tuple, blocks of the do -also construct, function arguments, operands of binary operators, and the target and arguments of a method call (Fig. ). This is fork-join parallelism: all tasks for a construct execute to completion before execution of the construct itself can complete. Tasks are implicitly scheduled and provide no fairness guarantees; while tasks may execute concurrently, a Fortress compiler or run-time may choose to schedule tasks for sequential execution instead. This allows flexible allocation of hardware resources. Comprehensions, reduction expressions, and for loops are all syntactic sugar for uses of generators and reducers. A generator is an object capable of supplying some number of separate values to a reducer through one or more library-defined protocols; a reducer is an object that can accept values from a generator and produce a combined or aggregated result. For example, the nested for loop in Fig. contains two generator expressions : and (i + ) : representing parallel counted ranges; the loop desugars into a nested pair of calls to their respective loop methods.
Fortress (Sun HPCS Language)
F
(* Five examples of situations in which expressions such as f (x) and g(y) and h(z) may be executed in parallel as concurrent tasks. *) (a, b, c) = ( f (x), g(y), h(z)) do f (x) also do g(y) also do h(z) end p( f (x), g(y), h(z)) f (x) + g(y) f (x).m(g(y), h(z))
Elements of a tuple ⊛ Separate blocks of a do -also construct ⊛
F Arguments of a function call ⊛ Operands of a binary operator ⊛ Target and arguments of a method call ⊛
Fortress (Sun HPCS Language). Fig. Implicit parallelism in Fortress
The actual parallelism is provided by using implicit tasks in the implementation of the loop method, generally using a divide-and-conquer coding style. Most Fortress data structures are generators and support structurerespecting parallel traversal. It is a library convention that generators are implicitly parallel by default; the seq(self) method conventionally constructs a sequential version of a generator. Definitions of BIG operators typically define the behavior of reductions (Fig. ) and comprehensions (Fig. ) in terms of an underlying monoid. Note that while generators run computations in parallel, there is still a well-defined spatial order to the results produced by a generator (Fig. ). Fortress also provides explicit parallelism using the spawn construct (Fig. ). Spawned threads are scheduled fairly with respect to one another, and implicit tasks of separately spawned threads are scheduled fairly with respect to one another. The programmer is responsible for wrapping concurrent accesses to shared data in an atomic block. The semantics of an atomic block is that either all the reads and writes to memory happen simultaneously, or none of them do; in effect, all the memory reads and memory writes of an atomic block appear to all other tasks to occur as a single atomic memory transaction. Such transactions are in fact strongly atomic (transactional and non-transactional accesses can coexist).
Implicit parallelism may be exploited inside of a transaction. Inner transactions may be retried independently of the outer transaction (“closed nesting”). Contention may be managed by the run-time system, or at the Fortress language level using the tryAtomic construct. The current implementation of Fortress uses an automatic dynamic load-balancing algorithm (based on work done at MIT as part of the Cilk project) that uses work-stealing to move pending tasks from one processor to another.
Programming in the Large Fortress supports a number of features for programming in the large. Components and APIs Fortress source code is encapsu-
lated in components, which contain definitions of program elements. Each component imports a collection of APIs (Application Programming Interfaces) that declare the program elements used by that component. Each component also exports a collection of APIs. If a component C exports an API A, then C must include a definition for every element declared in A. Components can be linked together into larger components. If a component C exports an API A that is imported by a component D, and C and D are linked, then references in D to the declarations in A are resolved to the corresponding definitions in C.
F
Fortress (Sun HPCS Language)
(* This for loop controls two index variables in nested fashion. The : operator, given two integers, produces a CountedRange object that can serve as a generator of integer index values. *) for i ← : , j ← (i + ) : do ai,j := i − j end (* The for loop is desugared into nested calls to higher-order methods that take functions as arguments. These functions bind the index variables. The fn syntax is analogous to Lisp’s lambda expression syntax for anonymous functions. *) ( : ).loop(fn (i) ⇒ ((i + ) : ).loop(fn ( j) ⇒ do ai,j := i − j end)) (* The loop method of the CountedRange generator object uses a recursive divide-and-conquer strategy that exploits implicit task parallelism (do -also ) to permit concurrent execution of loop iterations. *) object CountedRange(lo: Z, hi: Z) extends GeneratorZ ... loop(body: Z → ()): () = if lo = hi then body(lo) elif lo < hi then lo + hi ⌋ mid = ⌊ do CountedRange(lo, mid).loop(body) also do CountedRange(mid + , hi).loop(body) end end ... seq(self) = . . . end Fortress (Sun HPCS Language). Fig. Implementation of for loops through desugaring
Contracts Fortress functions and methods can be anno-
tated with contracts that declare preconditions and postconditions (Fig. ).
Test code, properties, and automated unit testing Fort-
ress includes support for defining tests inline in programs alongside the data and functionality they test. Similar to tests are property declarations, which document conditions that a program is expected to obey. The parameters in a property declaration denote the types of values over which the property is expected to hold.
When test data is provided with a property, the property is checked against all values in the test data when the program tests are run. One use of the design-by-contract features is to document and enforce algebraic relationships that are necessary for correct parallel execution of generators and reducers (Fig. ).
Innovations ●
Fortress introduced comprises and excludes clauses that allow the type checker to deduce that two types are disjoint; this matters for certain kinds
Fortress (Sun HPCS Language) ⊛
F
Summation is one kind of reduction expression. ai,j ∑
i←: j←(i+):
The summation is desugared into nested calls to higher-order methods. ∑(( : ).nest(fn (i) ⇒ ((i + ) : ).map(fn ( j) ⇒ ai,j ))) ⊛
(* The unary ∑ operator takes a generator and gives its reduce method an object that implements the standard reduction protocol using addition. The reduce method, like the loop method, provides implicit parallelism. This ∑ operator is generic; it can be used with any data type T that implements the Additive trait, meaning that type T implements a binary + operator and can supply an identity value named zero. *) opr ∑[[T extends AdditiveT]](g: GeneratorT): T = g.reduce(SumReductionT) object SumReduction[[T extends AdditiveT]] extends ReductionT, T empty(): T = zero ⊛ AdditiveT means type T has a zero value join(a: T, b: T): T = a + b ⊛ that is the identity for the + operator on type T. singleton(a: T): T = a ⊛ The sum of one number is the number itself. end Fortress (Sun HPCS Language). Fig. Implementation of reduction operations through desugaring
List comprehension ⟨ . . . ∣ . . . ⟩ is one kind of comprehension expression. ⊛ It produces an ordered list of values. ⟨ ai,j ∣ i ← : , j ← (i + ) : ⟩ ⊛
(* Other kinds of comprehension brackets construct arrays, sets, or multisets. They are desugared in much the same manner as reduction expressions. *) (* The meaning of a comprehension is defined by library code in the same way as for a reduction operator.
*)
opr BIG ⟨T g : GeneratorT ⟩ : ListT = g.reduce(ListReductionT) (* List aggregation is essentially reduction of singleton lists using the list concatenation operator ∥ . The library definitions of the ∥ operator and the (noncomprehension) list constructors ⟨ ⟩ and ⟨ x ⟩ are not shown here. *) object ListReductionT extends Reduction[[T, ListT]] empty(): ListT = ⟨ ⟩ ⊛ Produce an empty list. join(a: ListT, b: ListT): ListT = a ∥ b ⊛ Concatenate two lists. singleton(a : T): ListT = ⟨ a ⟩ ⊛ Produce a singleton list. end Fortress (Sun HPCS Language). Fig. Implementation of comprehension expressions through desugaring
F
F
Fortress (Sun HPCS Language)
of code optimization. Such declarations can also improve code maintainability. ● Lexical syntax is inspired in part by Wiki notation: the visual appearance of “rendered” code depends
(* The squares of the nine values , , , , , , , , may be computed concurrently or in any temporal order, but the logical or spatial order of the results within ordered list will always be ⟨ , , , , , , , , ⟩ . *) ⟨ k ∣ k ← : ⟩ Fortress (Sun HPCS Language). Fig. The distinction between logical order and temporal order
on whether an identifier is lowercase, uppercase, or mixed case, and whether underscore characters are used in particular ways. Many single-character operators and brackets can be represented by multicharacter ASCII sequences, but there is no single “escape” character (such as backslash) that introduces such sequences. As a rule, if an operator has a standard name in TEX or LATEX, then that same name converted to uppercase, omitting the backslash, is a Fortress name for that operator. See Fig. . Text in comments uses a markup syntax based on Wiki Creole . []. See Fig. . ● Most programming languages handle operator precedence by imposing a total order on operators (sometimes by mapping them to specific integer values). Fortress uses only precedence rules that would
do The spawn construct forks a fair concurrent thread ⊛ and immediately returns a handle object for the thread. ⊛
firstThread = spawn f (x) secondThread = spawn g(y) Other computations here will run concurrently with the spawned threads. The val method waits for a ⊛ spawned thread to complete and then produces whatever ⊛ value or exeception the thread may have produced. ⊛ ⊛
firstThread.val + secondThread.val end Fortress (Sun HPCS Language). Fig. Explicit spawning of fair concurrent threads
(* This definition of factorial has a contract that requires the argument to be nonnegative. The contract also guarantees that the result will be nonnegative. *) factorial (n) requires { n ≥ } ensures { outcome ≥ } = if n = then else n factorial (n − ) end (* This test-only code (indicated by the keyword test ) calls the factorial function for all values from to , and checks that the result for each is not less than the argument. *) test factorialNotLessThanInput [x ← : ] = (x ≤ factorial (x)) (* This property declaration requires that the result of factorial be not less than the argument for every argument of type Z . Test data for spot-checking a property may be provided separately. *) property factorialNeverLessThanInput = ∀(x: Z) (x ≤ factorial(x)) Fortress (Sun HPCS Language). Fig. Examples of contracts, test code, and declared properties
Fortress (Sun HPCS Language)
F
(* This trait imposes upon any trait T that extends BinaryOperator T, ⊙ the requirement to provide a concrete implementation of the specified binary operator ⊙ . Note that both T and ⊙ are parameter names, for which an actual type and actual operator are substituted by an invocation such as BinaryOperator Q, + or BinaryOperator Z, MAX . *) trait BinaryOperator T, opr ⊙ comprises T opr ⊙(self, other: T): T end An associative operator is a binary operator with the associative property. trait Associative T, opr ⊙ comprises T extends { BinaryOperator T, ⊙ , EquivalenceRelation T, = } property ∀(A; T, b: T, c: T) ((a ⊙ b) ⊙ c) = (a ⊙ (b ⊙ c)) end ⊛
A monoid is a type with an operator that is associative and has an identity. trait Monoid T, opr ⊙ comprises T extends { Associative T, ⊙ , HasIdentity T, ⊙ } end ⊛
Fortress (Sun HPCS Language). Fig. Use of declared properties to enforce algebraic relationships
be familiar from high-school or college mathematics courses. A consequence is that operator precedence in Fortress is not transitive (Fig. ). ● The “meet rule” eliminates the need for arbitrary “tie-break” or “ordering” rules found in other multiple-inheritance type systems. ● Fortress addresses the “self-type problem” common to many type systems with inheritance by using generic traits that each have a static parameter that is a type that must extend the generic trait; the comprises clause is used to enforce this requirement. The parameter name then serves as a self-type adequate to solve the “binary method problem.” ● The Fortress libraries use the self-type idiom to encode and enforce (through use of design-bycontract features) a hierarchy of algebraic properties of data types and operators ranging from commutativity and associativity all the way up to partial and total orders, monoids and groups, and Boolean algebras. Type-checking ensures that, for example, a comparison operator given to a sort method actually implements a correct ordering criterion; overloaded method dispatch allows the sort method to use different algorithms depending on whether the order is partial or total.
●
Fortress supports units and dimensions with a relatively small amount of built-in mechanism. All the Fortress compiler knows is that units and dimensions may be declared (Fig. ), that they form a free Abelian group under multiplication, that these abstract values may be used as static arguments to generic types, and that such a generic type may optionally be declared to absorb units. That you can multiply apples and oranges, that apples may be added only to apples, and whether you can exclusive-or apples and apples, are all at the discretion of the library programmer. Dimensioned types can ensure that data is correctly scaled (Fig. ). ● The Maybe type (instances are either Just x or Nothing), borrowed from Haskell, can be used as a generator of at most one item. A syntactic convenience allows it to be used in an if statement so as to bind a variable to the item in just the then clause (Fig. ). ● The Java programming language allows a method of the superclass to be called using the super keyword. A language with multiple inheritance requires a more general feature. Using a type assumption expression x asif T as an argument causes method dispatch to use the specified static
F
F
Fortress (Sun HPCS Language)
ASCII
rendered
z
z
normal variable (italic)
z_
z
roman
_z
z
bold
_z_
z
bold italic
ZZ
Z
blackboard calligraphic
ZZ_ _ZZ
Z
sans serif
z_bar
z¯
math accent
z_hat
zˆ
math accent
z_dot
z˙
math accent
z’
z′
math accent
_z_vec
⃗z
math accent
z_max
zmax
z13
z
numeric subscript
_z13_bar’
z¯′
combination of features
and
and
normal variable (italic)
And
And
type name (roman)
AND
∧
operator name
OPLUS
⊕
operator name as in TEX
SQCAP
⊓
operator name as in TEX
<=
≤
less than or equal to
>=
≥
greater than or equal to
<-
←
left arrow
[\ \]
white brackets
<| |>
⟨ ⟩
angle brackets
roman subscript
Fortress (Sun HPCS Language). Fig. Examples of rendering ASCII identifiers as mathematical symbols
Fortress (Sun HPCS Language)
F
This ASCII Fortress source code includes Wiki markup in the comment: (* Compute the logical exclusive OR of two Boolean values. =Arguments ; ‘self‘ :: a first Boolean value ; ‘other‘ :: a second Boolean value =Returns > The exclusive OR of the two arguments. This is ‘true‘ if **either** argument is ‘true‘, but not if **both** arguments are ‘true‘. > | ||||| ‘other‘ ||| | |||||-------------------|| | ‘OPLUS‘ ||||| false | true || |=========================================|| | ‘self‘ || false ||| false | true || | || true ||| true | false || |-----------------------------------------|| *) opr OPLUS(self, other: Boolean): Boolean and it is rendered as follows: (* Compute the logical exclusive OR of two Boolean values. Arguments self
a first Boolean value
other
a second Boolean value
Returns The exclusive OR of the two arguments. This is true if either argument is true, but not if both arguments are true. other ⊕
false true
self false
false true
true
true false
*) opr ⊕(self, other: Boolean): Boolean Fortress (Sun HPCS Language). Fig. Example of rendering Wiki-style markup in Fortress comments
type T for the argument expresssion x during method lookup. This general mechanism solves the multiple supertype problem as a special case (Fig. ).
●
Fortress provides an object-oriented implicit coercion feature. It differs from a conventional method such as toString in that it is declared in the type converted to rather the type converted from, and
F
F
Fortress (Sun HPCS Language)
a+b>c+d p>q∧r >s w+x∧y+z (w + x) ∧ (y + z) w + (x ∧ y) + z
Correct: + has higher precedence than > ⊛ Correct: > has higher precedence than ∧ ⊛ Incorrect: no defined precedence between + and ∧ ⊛ Correct: parentheses specify desired grouping ⊛ Correct: parentheses specify desired grouping ⊛
Fortress (Sun HPCS Language). Fig. Operator precedence in Fortress is not transitive
Declarations of several physical dimensions and their default units dim Mass default kilogram dim Length default meter dim Time default second dim ElectricCurrent default ampere ⊛
Declarations of derived physical dimensions dim Velocity = Length/Time dim Acceleration = Length/Time dim Force = Mass⋅Acceleration dim Energy = Force⋅Length ⊛
Declarations of basic units and their abbreviations unit kilogram kg: Mass unit meter m: Length unit second s: Time unit ampere A: ElectricCurrent ⊛
Declarations of derived units unit newton n: Force = meter⋅kilogram/second unit joule J: Energy = newton meter ⊛
Declarations of derived units that require scaling factors unit gram g: Mass = − kilogram unit kilometer km: Length = meter unit centimeter cm: Length = − meter unit nanosecond ns: Time = − second ⊛
unit inch inches in: Length = . cm unit foot feet ft: Length = inches
Fortress (Sun HPCS Language). Fig. Example declarations of physical dimensions and units
in that it is invoked implicitly by method dispatch if necessary. Among method overloadings, one that requires no coercion to be applicable is always considered more specific than one that requires one or more coercions. Coercion declarations may also
be marked as widening; automatic rewriting rules on parts of the expression tree that include widening coercions provide the effect of “widest need evaluation” [], but in a manner that is fully under user (or library) control.
Fortress (Sun HPCS Language)
F
Declarations in terms of dimensions implicitly use default units. m v kineticEnergy(m: R Mass, v: R Velocity): R Energy = ⊛ Alternatively, declarations may specify units explicitly. m v kineticEnergy(m: R kg, v: R m/s): R J = do mySpeed = . feet per second ⊛ The function kineticEnergy requires a speed measured in ⊛ meters per second, so mySpeed by itself is not a valid argument. ⊛ The in operator multiplies by the necessary scale factor, − ⊛ which in this case is × . × . The scale factor ⊛ is determined automatically from the declarations of the units. kineticEnergy( kg, mySpeed in m/sec) end
⊛
Fortress (Sun HPCS Language). Fig. Examples of the use of declared physical dimensions and units
Assume the position method returns a value of type MaybeZ . if p ← myString.position(‘=’) then myString[ : p ] ⊛ Here p is bound to a value of type Z . else myString end ⊛
Fortress (Sun HPCS Language). Fig. Use of a generator expression in an if statement
●
Fortress uses where clauses to bind additional type parameters and to constrain relationships among static type parameters. In particular, the constraint may be a subtype relationship. This general mechanism suffices to express covariance and contravariance (Fig. ), which are important type relationships that require more specialized mechanisms in other languages. ● MPI allows a program to inquire how many hardware processors there are, but little else about the execution environment. High Performance Fortran provides alignment and distribution directives that describe how arrays should be laid out across multiple processors, but the processors must be organized as a flat Cartesian grid, and the class of distributions (such as block, cyclic, and blockcyclic) is fixed.
Fortress defines a hierarchical data structure (the region) that can describe hardware resources (both processors and memory) and relative communication costs, and can be queried by a running program. A Fortress distribution maps the parts of an aggregate data structure to one or more regions. The class of distributions is open ended and library programmers can define new distributions.
Influences from Other Programming Languages Aspects of the Fortress type system, including inheritance and overloaded multimethod dispatch, were influenced by the ML, Haskell, Java, NextGen, Scala, CLOS, Smalltalk, and Cecil programming languages, among others. Inspirations for the use of mathematical syntax and a character set beyond ASCII include
F
F
Fortress (Sun HPCS Language)
trait FloorWax advertise(self) = print “Great shine!” end trait DessertTopping advertise(self) = print “Great taste!” end trait Shimmer extends { FloorWax, DessertTopping } advertise(self) = do ⊛ Type assumption controls which “super-method” to call. advertise(self asif FloorWax) advertise(self asif DessertTopping) end end Fortress (Sun HPCS Language). Fig. Example use of asif expressions
List is covariant in its type parameter T. trait ListT extends ListU where { T extends U } ... end ⊛
OutputStream is contravariant in its type parameter E. trait OutputStreamE extends OutputStreamG where { G extends E } ... end ⊛
Fortress (Sun HPCS Language). Fig. Use of where clauses to express covariance and contravariance of type parameters
APL, MADCAP, MIRFAC, the Klerer-May System, and COLASL. The idea of having “publication” (beautifully rendered) textual form as well as one or more “implementation” forms goes back to Algol . Notions of data encapsulation, comprehensions, generators, and reducers can be traced to CLU, Alphard, KRC, Miranda, and Haskell. In the area of functional programming, higher-order functions, and divide-andconquer parallelism, many languages had an influence, but especially Lisp, Haskell, and APL. The approach to data distribution was strongly influenced by experience with High Performance Fortran and its predecessors.
Future Directions The current Fortress compiler targets the Java Virtual Machine (JVM) and uses a custom class loader to instantiate generic types as they are needed. The JVM provides good multiplatform native code generation, threads, and garbage collection, and hosts libraries implementing both work-stealing and transactions. The Fortress type system is embedded into the Java type system where it is convenient, but tuple types and arrow types are handled specially. Fortress value types are eligible for compilation into an unboxed (primitive) representation. Fortress traits compile to Java interfaces, and Fortress objects compile to Java final classes.
Fujitsu Vector Computers
A straightforward analysis of traits comprising object singletons can demonstrate finiteness, which permits the general application of unboxing to new types. A specialized virtual machine (VM) and run-time system may someday be desirable, but at this time the benefits of targeting the JVM far outweigh the overhead. The design of regions and distributions for managing the mapping of data and threads to hardware resources has been worked out on paper, but has not yet been implemented. The original design of Fortress includes a form of where clause that conditionally gates such type relationships as extension, comprising, and exclusion. For example, one might define an generic aggregate data structure AE with a multiplication operator that is defined in terms of multiplication for the element type E, and one might well wish to declare that the multiplication of aggregates is commutative if and only if multiplication of the elements is commutative. This may be achieved by declaring trait AE extends { CommutativeAE, × where { E extends CommutativeE, × } }
.
.
.
.
.
. .
The implications of such conditional type relationships for the implementation of a practical type checker are still a subject of research. An object-oriented pattern-matching facility for easily binding multiple variables to the fields of an object, similar to the pattern-matching facilities in Haskell and Scala but with more provisions for user extensibility, has been sketched out but only recently implemented.
F
Version .. http://research.sun.com/projects/plrg/fortress.pdf, March . See also http://research.sun.com/projects/plrg/ Publications/fortress.beta.pdf, March Allen E, Culpepper R, Nielsen JD, Rafkind J, Ryu S () Growing a syntax. In: ACM SIGPLAN Foundations of Object-Oriented Languages workshop, Savannah, Bailey DH, Barszcz E, Barton JT, Browning DS, Carter RL, Fatoohi RA, Frederickson PO, Lasinski TA, Simon HD, Venkatakrishnan V, Weeratunga SK () The NAS parallel benchmarks. Technical report, Int J Supercomput Appl ():– Corbett RP () Enhanced arithmetic for Fortran. ACM SIGPLAN Notices ():–, http://doi.acm.org/./. High Productivity Computer Systems program of the Defense Advanced Research Projects Agency (United States Department of Defense). http://www.highproductivity.org/ and see also //www. darpa.mil/IPTO/programs/hpcs/hpcs.asp Steele Jr GL () Growing a Language. Invited talk. Abstract in OOPSLA ’ Addendum: Addendum to the Proceedings of the Conference on Object-Oriented Programming, Systems, Languages, and Applications. ISBN ---. ACM, New York, http://doi.acm.org/./. Transcript in Higher-Order and Symbolic Computation , (October ), –. Video at http://video.google.com/videoplay? docid=-# The Unicode Consortium () The Unicode Standard, Version .. Addison-Wesley, Boston Wiki Creole project and website. Sponsored by the Wiki Symposium. http://www.wikicreole.org/wiki/Creole.
Forwarding Routing (Including Deadlock Avoidance)
Related Entries Chapel (Cray Inc. HPCS Language) HPF (High Performance Fortran) PGAS (Partitioned Global Address Space) Languages
Bibliography . Allen E, Chase D, Flood C, Luchangco V, Maessen JW, Ryu S, Steele Jr GL () Project Fortress Community website. http:// projectfortress.java.net . Allen E, Chase D, Hallett J, Luchangco V, Maessen JW, Ryu S, Steele Jr GL, Hochstadt ST () The Fortress Language Specification
Fujitsu Vector Computers Kenichi Miura National Institute of Informatics, Tokyo, Japan
Synonyms Fujitsu vector processors; Fujitsu VPP systems; SIMD (single instruction, multiple data) machines
F
F
Fujitsu Vector Computers
Definition Fujitsu Vector Machines are the series of supercomputers developed by Fujitsu from to . They are based on the pipeline architecture, and VPP systems have also incorporated a high degree of parallelism with distributed memory in the system architecture, utilizing the full crossbar network for non-blocking system interconnect.
Discussion FACOM – Array Processor Unit (APU) The first vector machine in Japan was FACOM – Array Processing Unit (APU) which was installed at the National Aerospace Laboratory (later became the Japanese Aerospace Exploration Agency (JAXA) in October ) in []. This system was organized as an asymmetric multiprocessor system which consists of the mainframe computer FACOM – and the Array Processor Unit. The clock was ns, and the peak performance of APU is Mflop/s for addition/subtraction and Mflop/s for multiplication. This system was a pipeline-based array processor with data registers (scalar) and one vector register with , word ( bits/word) for vector operations. The capacity of the main memory was M words with -way interleave. AP-Fortran, an extended Fortran language, was developed for the – APU to allow parallelism description.
FACOM VP/ Series The vector processor FACOM VP and FACOM VP [–] were announced in July . The major goals of the VP/ Series were to achieve a very high performance, ease-of-use, and a good affinity with general-purpose computing environment with the FACOM M-Series mainframe computers. The state-ofthe-art semiconductor technology was incorporated in the system; the ECL LSI with a gate delay time of ps (Fig. ) and and , gates per chip, and high-speed RAM with an access time of . ns for vector registers. As for the memory technology, the main memory unit with high capacity and high data transfer capability was realized by using the Kbit SRAM with access time of ns. Figure illustrates the architecture of VP/ Series system. It consists of the scalar unit, vector unit,
Fujitsu Vector Computers. Fig. VP LSI packaging technology. (Photo Credit: Fujitsu Limited)
and main storage unit. The scalar unit fetches and decodes all the instructions and executes the scalar instructions. The vector unit consists of six functional pipeline units, vector registers, and mask registers. The functional pipes are add/logical pipe, multiply pipe, divide pipe, mask pipe, and two load/store ports, which are also pipelined. The first three pipes are for arithmetic operations. The load/store pipes support the contiguous access, strided access, and the indirect addressing (gather/scatter) for flexible vector operations. The mask pipes are used to handle the conditional branches within loops. One of the most unique features of the VP/ Series is the dynamically reconfigurable vector registers. The total capacity of the vector registers for VP is K words ( bits), but they can take such configurations as (vector length) × (number of vector registers), × , × , . . . , × . The major features of the VP/ Series are summarized in Table . Figure shows the cabinet of the VP system. The first delivery of the VP-Series system took place in January , that is, VP- at the Institute for Plasma Physics in Nagoya. Later, the high-end model FACOM VP-, with improved performance, was added, and a processing performance exceeding Gflop/s was achieved. Also, the market was expanded by adding lower-end models: the FACOM VP and FACOM VP . The VP-Series E Models, enhanced versions of the VP / Series, were announced in July .
Fujitsu Vector Computers
Vector Unit (570MFLOPS)
F
Mask Registers (1KB) Mask
Add/Logical Load/Store (2.3GB/S)
Main Storage Unit
Multiply Reconfigurable Vector Registers (64KB)
Load/Store (2.3GB/S)
(Max. 256MB)
Scalar Execution Unit
Cache (64KB) Scalar Unit
Channels
Divide
General Purpose Registers (4B x 16) Floating Point Registers (8B x 8)
Fujitsu Vector Computers. Fig. VP architectural block diagram
Fujitsu Vector Computers. Table VP Series specifications Model
VP
VP
VP
VP
Announcement
April,
July,
July,
April,
Machine cycle
. ns
. ns
. ns
. ns
Max. performance
Mflop/s
Mflop/s
Mflop/s
, Mflop/s
Total vector regs. size
KB
KB
KB
KB
Basic vector length
elements
elements
elements
elements
Total Mask regs.
kbits
kbits
kbits
kbits
Add/logical
×
×
×
×
Multiply
×
×
×
×
Divide
×
×
×
×
Mask
×
×
×
×
Load/Store
×
×
×
×
Ckt. technology (ECL) & delay
gates/chip ps
gates/chip ps
gates/chip ps
gates/chip ps
Memory technology
kbits SRAM ns access
kbits SRAM ns access
kbits SRAM ns access
kbits SRAM ns access
Main memory size
, , MB
, , MB
, , MB
, , MB
Logical no. times multiplicity of vector pipes
F
F
Fujitsu Vector Computers
Fujitsu Vector Computers. Fig. VP system. (Photo Credit: Fujitsu Limited)
Vector processor
Mask registers
Mask
Mask
System storage unit
Main storage unit
Load/ store*
Multiplication & addition/logical calculation*
Vector registers
Multiplication & addition/logical calculation*
Load/ store*
Division
Scalar execution unit
Buffer storage Channel processor
Vector unit
Scalar unit **
General registers Floating registers * Model VP2000 provides separate pipelines for loading/storing and addition/subtraction/ logic calculations. **Uni-processor models have one scalar unit.
Fujitsu Vector Computers. Fig. VP architectural block diagram
Fujitsu Vector Computers
F
Fujitsu Vector Computers. Table VP Series specifications Models
VP/ VP/
VP/ VP/
VP/ VP/
VP/ VP/
Announcement
Dec.
Dec.
Dec.
Dec.
Max. performance
Mflop/s
Gflop/s
Gflop/s
Gflop/s
No. scalar unit/
–
–
–
–
vector unit
Total capacity of vector registers
KB
KB
KB
KB
Total capacity of mask registers
kbits
kbits
kbits
kbits
No. vector pipes
& multiplicity
Ckt. technology
K gates
K gates
K gates
K gates
(ECL) & delay/gate
ps
ps
ps
ps
Ckt technology
kbit SRAM
kbit SRAM
kbit SRAM
kbit SRAM
(Capacity & delay)
. ns
. ns
. ns
. ns
Capacity of main memory
MB– GB
MB– GB
MB- GB
MB– GB
Memory technology
Mbit SRAM/
Mbit SRAM/
Mbit SRAM/
Mbit SRAM/
/delay
ns
ns
ns
ns
Capacity of system storage
,,,, GB
,,,, GB
,,,, GB
,,,, GB
I/O capabilities
Max. GB/s
Max. GB/s
Max. GB/s
Max. GB/s
No. channels
–
–
–
–
Block Mpx Ch./
. MB/s
. MB/s
. MB/s
. MB/s
Optical Ch.
. MB/s
MB/s
. MB/s
. MB/s
There were five models : VP E, VP E, VP E,VP E, and VP E. These E models achieved performance which is . times more powerful than that of the previous VP / Series machines by installing high-speed vector memory and an additional arithmetic pipeline for the fused multiply-add operations.
sharing one vector unit. This feature, called Dual Scalar Processors (DSP), may be regarded as two-threaded system. The objective of DSP is to utilize the vector unit more efficiently when the vectorization ratio of users’ application programs is not very high []. The key features of VP Series system are summarized in Table .
FUJITSU VP Series The VP Series [–] was announced in December as the successor to Fujitsu’s VP-E Series supercomputers. Figure illustrates the architecture of VP Series systems. It carries most of the VP/ architectural features with various enhancements in the vector unit, such as two sets of fused multiply-add pipelines, doubled mask pipelines, and faster division circuit. The most notable feature of the VP Series is that it can be configured with one or two scalar units
Numerical Wind Tunnel and VPP-Series Numerical Wind Tunnel The National Aerospace Laboratory (NAL) of Japan developed the Numerical Wind Tunnel (NWT) [] parallel supercomputer system jointly with Fujitsu. The system architecture was a distributed memory vectorparallel computer, in which vector supercomputers
F
F
Fujitsu Vector Computers
Fujitsu Vector Computers. Fig. NWT system. (Photo Credit: JAXA)
were connected with the non-blocking crossbar network. The NWT’s final specifications were processing elements, each with . Gflop/s of processing power to give a peak performance of Gflop/s with . GB of the total main memory. The NWT’s processing element consisted of a CPU and memory. The CPU is a vector supercomputer by itself, comprising three types of LSIs: GaAs, ECL, and BiCMOS, in total. It is one of the very few commercially successful GaAs-based system. The primary cooling method was water cooling, but the forced air-cooling method was also used for the memory units. Figure shows the NWT as installed at NAL. The NWT, which went into operation in January , was rated as the most powerful supercomputer site in the world in the TOP list from to . It should be noted that the NWT was the technology precursor to the VPP, which will be described in more detail in the next section.
FUJITSU VPP The VPP was Fujitsu’s third generation supercomputer system [, –]. As stated in the previous section, it was a commercial version of NWT. Fujitsu announced the VPP in September as the world’s most powerful supercomputer at that point in
time with a speed of Gflop/s. Fujitsu has developed processor elements (PEs), each being capable of vector processing and also can operate in highly parallel fashion to obtain this performance. The maximum configuration was with processing elements to reach Gflop/s. As shown in Fig. , each processing element consisted of a scalar unit, a vector unit, a main storage unit, and a Data Transfer Unit (DTU), built with Gallium Arsenide LSI’s, Bipolar LSI’s and BiCMOS LSI’s, and ultrahigh-density circuit boards, and was able to deliver a maximum vector performance of . Gflop/s. The scalar unit is VLIW architecture and may execute up to four operations in parallel. In order to achieve very high system performance, PEs were connected with a highspeed crossbar network which allows non-blocking data transfer. The data transfer rate was MB/s from one PE to another. Since a PE can simultaneously send and receive data, the aggregate data transfer rate per PE was MB/s. Figure illustrates the system configuration of VPP, based around the crossbar network. In order to fully utilize the powerful capability of the interconnecting network, the Data Transfer Unit (DTU) was incorporated, which allows the mapping between the local and the global addresses, a very fast synchronization across PEs, and actual data transfer with proprietary message passing protocol. Various
Fujitsu Vector Computers
N E T W O R K
F
Mask Registers Mask
Data Transfer Unit
Vector Unit
Multiply
(DTU) Load
Add/Logical
Vectors Registers
Store Main Storage Unit
Divide
F Scalar Unit
Scalar Execution Unit
Cache General Purpose Registers Floating Point Registers
Fujitsu Vector Computers. Fig. VPP architectural block diagram
VU D T U
M E M
SU
PE1
PE(n-1)
PEn
VU
VU D T U
M E M
SU
PE0
...
D T U
M E M
SU
VU D T U
M E M
IOP
SU IOP
Crossbar Interconnect
Fujitsu Vector Computers. Fig. VPP system configuration
data transfer modes were supported by the DTU. They were: contiguous, constant-strided, sub-array, and indirect addressing (random gather/scatter). Figure shows the inside of one cabinet of VPP, which contains PEs. Plumbing for water cooling is also visible in this photo. The specifications of the VPP system and the follow-on products are summarized in Table .
the distributed memory units of VPP. The ability to define the array data in Fortran programs, distributed across multiple processing elements as a single global space, greatly simplified the porting and tuning of existing codes to VPP VPP Fortran may be regarded as an early implementation of PGAS (Partitioned Global Address Space). More details are described in [].
New Parallel Language Fujitsu developed a new parallel language, called VPP Fortran, which can define a single name space across
VPP/VPP/VPP Series After a great success with VPP, Fujitsu developed the CMOS versions of the vector-parallel system The
F
Fujitsu Vector Computers
oxide semiconductor) technology, were introduced in February , the Fujitsu VPP Series in March , and the VPP Series in April . All VPP Series supercomputers after the VPP Series used the CMOS technology. Packaging technology of VPP is shown in Fig. . The VPP was the successor to the former VPP/VPPE systems (with E for extended, i.e., the clock cycle . instead of ns). The clock cycle has been halved, and the vector pipes are able to deliver fused multiply-add results. With a multiplicity of for these vector pipes, floating-point results per clock cycle can be generated. In this way, a fourfold increase in speed per processor can be attained with respect to the VPPE. The architecture of the VPP nodes is almost identical to that of the VPP, but one of the major enhancements is in the performance of noncontiguous memory access by incorporating more address matching circuits.
Fujitsu Vector Computers. Fig. VPP cabinet. (Photo Credit: Fujitsu Limited)
Fujitsu VPP Series and VX Series (the single CPU model), which used CMOS (Complementary metal
Conclusion While the vector processing with very sophisticated hardware pipelines together with highly interleaved memory subsystem was the dominant supercomputer architecture in the s and s, the wave of highly (or massively) parallel scalar systems have been gaining
Fujitsu Vector Computers. Table VPP Series specifications (typical models) Models
VPP
VPPE
VPP
Announcement
September,
February,
April,
Ckt. technology
GaAs/ECL/Bipolar
CMOS(.μ)
CMOS(.μ)
Level of integration
K Tx’s (GaAs)
M Tx’s/chip
M Tx’s/chip
Delay time/gate
ps
ps
ps
Cooling method
Liquid (water)
Forced air
Forced air
Clock frequency
MHz
MHz
MHz
Max. perf./PE
. Gflop/s
. Gflop/s
. Gflop/s
Mem. capacity/PE
GB
GB
– GB
Memory technology
Mbit SRAM
Mbit SDRAM
Mbit SDRAM, Mbit SSRAM
I/O capability/PE
MB ×
MB ×
. GB ×
Max. configuration
PEs
PEs
PEs
Max. system perf.
Gflop/s
. Tflop/s
. Tflop/s
CPU architecture
bit
bit
bit
Fujitsu Vector Computers
F
F
Fujitsu Vector Computers. Fig. VPP PE/Memory board. (Photo Credit: Fujitsu Limited)
more popularities in this century. The major reasons may be stated as; . Pipelining has become the common techniques even in the scalar microprocessor design. . Larger market opportunities have arisen with the advent of high volume produced the powerful scalar microprocessors. . The highly interleaved memory structure in the vector architecture has become a more expensive implementation as compared with the cache-based approach in the scalar architecture, for the purpose of coping with the widening speed gap between CPU and memory. . More and more application programs have been tailored toward scalar architecture. Thus, Fujitsu decided to change direction from vector architecture to highly parallel scalar Symmetric Multiprocessor (SMP) and its constellations. The VPP system was Fujitsu’s last vector product. But it should also be pointed out that the recent trends in SIMD instructions in the general purpose microprocessors and the popularity of GPGPU (General Purpose Graphic Processing Unit) could be regarded as “the return of vectors,” in the sense that the
programming models and compiler technology developed for the vector architecture continue to be vital technologies in the new approaches.
Related Entries PGAS (Partitioned Global Address Space) Languages TOP VLIW Processors
Bibliography . Uchida K, Seta Y, Tanakura Y () The FACOM - Array Processor System. In: Proc rd USA-JAPAN Computer Conference, San Francisco, CA. AFIPS Press, Montvale, NJ, pp – . Miura K, Uchida K () FACOM vector processor VP-/VP. In: Kowalik JS (ed) High-speed computation, NATO ASI Series F: computer and systems sciences, vol . Springer, Berlin, pp – . Miura K () Fujitsu’s supercomputer: FACOM vector processor system. In: Fernbach S (ed) Supercomputers: class VI systems, hardware and software. North-Holland, Amsterdam, pp – . Matsuura T, Miura K, Makino M () Supervector performance without toil: Fortran implemented vector algorithms on the VP/. Comput Phys Commun :–
F
Fujitsu Vector Processors
. Miura K () Vectorization and parallelization of transport Monte Carlo simulation codes. In: NATO ASI Series vol F, pp – . Uchida N et al () Fujitsu VP Series. In: Proceedings of COMPCON Spring’, San Francisco, CA. IEEE Computer Society Press, Washington, DC, pp – . Miura K, Nagakura H, Tamura H () VP Series dual scalar and quadruple scalar models supercomputer systems – A new concept in vector processing. In: Proceedings of COMPCON Spring’, San Francisco, CA. IEEE Computer Society Press, Washington, DC, pp – . Hwang K () Section .. Fujitsu VP and VPP. In: Advanced computer architecture: parallelism, scalability, programmability. McGraw-Hill, Inc., New York, pp – . Miyoshi H et al () Development and achievement of NAL numerical wind tunnel (NWT) for CFD computations. In: Proceedings of ACM/IEEE conference on supercomputing (SC’), Washington, DC. IEEE Computer Society Press (), Washington, DC, pp – . Utsumi T, Ikeda M, Takamura M () Architecture of the VPP parallel supercomputer. In: Proceedings of supercomputing’, Washington, DC. IEEE Computer Society Press, Washington, DC, pp – . Nakashima Y, Kitamura T, Tamura H, Takiuchi M, Miura K () The scalar processor of the VPP parallel supercomputer. In: Proceedings of the th ACM international conference on supercomputing (ICS’), Manchester, England. ACM, New York, pp – . Nakanishi M, Ina H, Miura K () A high performance linear equation solver on the VPP parallel supercomputer. In: Proceedings of ACM/IEEE conference on supercomputing (SC’), Washington, DC. IEEE Computer Society Press, Washington DC, pp – . Iwashita H et al () VPP Fortran and parallel programming on the VPP supercomputer. In: Proceedings of international symposium on parallel architectures, algorithms and networks (ISPAN’), Kanazawa, Japan, – Dec , pp – . Nodomi A, Ikeda M, Takamura M, Miura K () Hardware performance of the VPP parallel supercomputer. In: Dongarra J, Grandinetti L, Joubert G, Kowalik J (eds) High performance computing: technology, method and applications, vol . Advances in Parallel Computing. Elsevier, Amsterdam, pp –
Fujitsu Vector Processors Fujitsu Vector Computers
Fujitsu VPP Systems Fujitsu Vector Computers
Functional Decomposition Domain Decomposition
Functional Languages Philip Trinder , Hans-Wolfgang Loidl , Kevin Hammond Heriot-Watt University, Edinburgh, UK University of St. Andrews, St. Andrews, UK
Definition Parallel functional languages are parallel variants of functional languages, that is, languages that treat computation as the evaluation of mathematical functions and avoid state and mutable data. Functional languages are founded on the lambda calculus. The majority of parallel functional languages add a small number of high-level parallel coordination constructs to some functional language, although some introduce parallelism without changing the language.
Discussion Introduction The potential of functional languages for parallelism has been recognized for over years, for example []. The key advantage of functional languages is that they mainly, or solely, contain stateless computations. Subject to data dependencies between the computations, the evaluation of stateless computations can be arbitrarily reordered or interleaved while preserving the sequential semantics. Parallel programs must not only specify the computation, that is, a correct and efficient algorithm, it must also specify the coordination, for example, how the program is partitioned, or how parts of the program are placed on processors. As functional languages provide high level constructs for specifying computation, for example, higher-order functions and sophisticated type systems, they typically provide correspondingly high level coordination sublanguages. A wide range of parallel paradigms and constructs have been used,
Functional Languages
for example, data-parallelism [], and skeleton-based parallelism []. As with computation, the great advantage of highlevel coordination is that it frees the programmer from specifying low-level coordination details. Where low-level coordination constructs encourage programmers to construct static, simple, or regular coordination, higher-level constructs encourage more dynamic and irregular coordination. The challenges of highlevel coordination are that automatic coordination management complicates the operational semantics, makes the performance of programs opaque, requires a sophisticated language implementation, and is frequently less effective than hand-crafted coordination. Despite these challenges many new nonfunctional parallel programming languages also support highlevel coordination, and examples include Fortress [], Chapel [] or X [] that also support high-level coordination. Coordination may be managed statically by the compiler as in PSML [], dynamically by the runtime system as in GpH [], or by both as in Eden []. Whichever mechanism is chosen, the implementation of sophisticated automatic coordination management is arduous, and there have been many more parallel language designs than well-engineered implementations. The combination of a high-level computation and coordination language requires specialized parallel development tools and methodologies, as discussed below.
Strict and Non-strict Parallel Functional Languages Most programming languages strictly evaluate the arguments to a function before evaluating the function. However, as purely functional languages have significant freedom of evaluation order, some use non-strict or lazy evaluation. Here the arguments to a function are evaluated only if and when they are demanded by the function body. In consequence more programs terminate, it is easy to program with infinite structures, and programming concerns are separated []. Like most languages, many parallel functional languages are strict, for example, NESL [] or Manticore []. However there are many parallel variants of lazy
F
languages like Haskell [] and Clean []. At first glance this is surprising as lazy evaluation is sequential and performs minimum work, but non-strict languages have a number of advantages for parallelism []. They can be easily married to many different coordination languages as the execution order of expressions is immaterial; they naturally support highly dynamic coordination where evaluation is performed and data communicated on demand; their implementations already have many of the mechanisms required for parallel execution. Laziness also exists in the language implementations, for example, distributing work and data on demand [].
Paradigms Parallel functional languages have adopted a range of paradigms. For the main parallel paradigms this section outlines the correspondence with the functional paradigm, and describes a representative language. Some languages support more than one paradigm, for example, Manticore [] combines data and task parallelism, and Eden [] combines semi-explicit and skeleton parallelism.
Skeleton-Based Languages Algorithmic Skeletons [, ] are a popular parallel coordination construct, and as higher-order functions fit naturally in the functional model. Often these higher order functions work over compound data structures like lists or vectors and consequently the resulting parallel code often resembles data parallel code as discussed in section. Example skeleton-based functional languages include SCL [], PL [], Eden [], PSML [], and HDC. HDC [] is a strictly evaluated subset of the Haskell language with skeleton-based coordination. HDC programs are compiled using a set of skeletons for common higher-order functions, like fold and map, and several forms of divide-and-conquer. The language supports two divide-and-conquer skeletons and a parallel map, and the system relies on the use of these higher-order functions to generate parallel code. Data Parallel Languages Data parallel languages [] focus on the efficient implementation of the parallel evaluation of every element
F
F
Functional Languages
in a collection. Functional languages fit well with the data parallel paradigm as they provide powerful operations on collection types, and in particular lists. Indeed, all of the languages discussed here use some parallel extension of list comprehensions and implicitly parallel higher order functions such as map. Compared to other approaches to parallelism, the data parallel approach makes it easier to develop good cost models, although, it is notoriously difficult to develop cost models for languages with a non-strict semantics. Example data parallel functional languages include NESL [] and Data Parallel Haskell []. NESL is a strict, strongly typed, language with implicit parallelism and implicit thread interaction. It has been implemented on a range of parallel architectures, including several vector computers. A wide range of algorithms have been parallelized in NESL, including a Delaunay algorithm for triangularization [], several algorithms for the n-body problem [], and several graph algorithms.
Semi-explicit Languages Semi-explicit parallel languages provide a few highlevel constructs for controlling key coordination aspects, while automatically managing most coordination aspects statically or dynamically. Historically, annotations were commonly used for semi-explicit coordination, but more recent languages provide compositional language constructs. As a result, the distinction between semiexplicit coordination and coordination languages is now rather blurred, but the key difference in the approach is that semi-explicit languages aim for minimal explicit coordination and minimal change to language semantics. Example semi-explicit parallel functional languages include Eden [] and GpH []. GpH is a small extension of Haskell with a parallel (par) composition primitive. par returns its second argument and indicates that the first argument may be executed in parallel. In this model the programmer only has to expose expressions in the program that can usefully be evaluated in parallel. The runtime-system manages the details of the parallel execution such as thread creation, communication, etc. Experience in implementing large programs in GpH shows that the unstructured use of par and seq operators often leads to
obscure programs. This problem is overcome by evaluation strategies: lazy, polymorphic, higher-order functions controlling the evaluation degree and the parallelism of a Haskell expression. They provide a clean separation between coordination and computation. The driving philosophy behind evaluation strategies is that it should be possible to understand the computation specified by a function without considering its coordination.
Coordination Languages Parallel coordination languages [] are separate from the computation language and thereby provide a clean distinction between coordination and computation. Historically, Linda [] and PCN [] have been the most influential coordination languages, and often a coordination language can be combined with many different computation languages, typically Fortran or C. Other systems such as SCL [] and PL [] focus on a skeleton approach for introducing parallelism and employ sophisticated compilation technology to achieve good resource management. Example coordination languages using functional computation languages include Haskell-Linda [] and Caliban [, ]. Caliban has constructs for explicit partitioning of the computation into threads, and for assigning threads to (abstract) processors in a static process network. Communication between processors works on streams, that is, eagerly evaluated lists, similar to Eden.
Dataflow Languages Historically a number of functional languages adopted the dataflow paradigm and some, like SISAL [], achieved very impressive performance. Example dataflow functional languages include SISAL [], pHLuid system []. SISAL [] is a first-order, strict functional language with implicit parallelism and implicit thread interaction. Its implementation is based on a dataflow model and it has been ported to a range of parallel architectures. Comparisons of SISAL code with parallel Fortran code show that its performance is competitive with Fortran, without adding the additional complexity of explicit coordination [].
Functional Languages
Explicit Languages In contrast to the high-level coordination found in the majority of parallel functional languages, a number of languages support low-level explicit coordination. For example, there are several bindings for explicit message passing libraries, such as PVM [] and MPI [] for languages like Haskell and OCaml [, ]. These languages use an open system model of explicit parallelism with explicit thread interaction. Since the coordination language is basically a stateful (imperative) language, stateful code is required at the coordination level. Although the high availability and portability of these systems are appealing, the language models suffer from the rigid separation between the stateful and purely functional levels. Other functional languages support explicit coordination, for example, Erlang []. Erlang is probably the most commercially successful functional language, and was developed in the telecommunications industry for constructing distributed, real-time fault tolerant systems. Erlang is strict, impure, weakly typed, and relatively simple: omitting features such as currying and higher order functions. However the language has a number of extremely useful features, including the OTP libraries, hot loading of new code into running applications, explicit time manipulation to support soft real-time systems, message authentication, and sophisticated fault tolerance.
F
coordination. As a consequence the parallel behaviour of programs is often far from obvious, and hence hard to tune. Hence suites of profiling and visualization tools are very important for many parallel functional languages, for example [, , ]. Implicit parallelism, often promised in the context of functional languages, offers the enticing vision of parallel execution without changes to the program. In reality, however, the program must be designed with parallelism in mind to avoid unnecessary sequentiality. In theory, program analyses such as granularity, sharing, and usage analysis can be used to automatically generate parallelism. In practice, however, almost all current systems rely on some level of programmer control. Current development methodologies, like [], have several interesting features. The combination of languages with high-level coordination and good profiling tools facilitates the prototyping of alternative parallelisations. Obtaining good coordination at an early stage of parallel software development avoids expensive redesigns. In later development stages, detailed control over small but crucial parts of the program may be required, and profiling tools can help locate expensive parallel computations. During performance tuning the high level of abstraction may obscure key lowlevel features that could be usefully controlled by the programmer.
Parallel Functional Languages Today Tools and Methodologies Development methodologies for parallel functional programs exploit the amenability of the languages to derivation and analysis. For example, a programmer can reason about costs during program design using abstract cost models like BMF-PRAM []. Guided by the cost imformation the program design can relatively easily be transformed to reduce resource consumption before implementation, for example, []. Similarly, automated static resource analysis can provide information to improve coordination, for example, predicted task execution time can improve scheduling. Parallel functional languages favour high-level and often dynamic coordination in contrast to detailed static
Parallel functional languages are used in a range of domains including numeric, symbolic, and data intensive []. Commercially Erlang has been enormously successful for developing substantial telecoms and banking applications [] and parallel Haskell for providing standardized parallel computational algebra systems. The high-level coordination and computation constructs in parallel functional languages have been very influential. For example, algorithmic skeletons appear in MPI [] and are the interface for cloud parallel search engines like Google MapReduce. Similarly, stateless threads or comprehensions occur in modern parallel languages like Fortress [], Chapel [], or X []. Moreover the dominance of multicore and emergence
F
F
Functional Languages
of many-core architectures has focused attention on the importance of stateless computation. Some trends are as follows. As clock speeds plateaux but register width increases, data parallelism becomes attractive for functional, and other, parallel languages. In the functional community there are often multiple parallelizations of a single base language, for example, GpH and Eden are both Haskell variants, moreover, a single compiler often supports more than one parallel version, for example, the Glasgow Haskell Compiler has four experimental multicore implementations [].
Related Entries Data Flow Graphs Data Flow Computer Architecture MPI (Message Passing Interface) Parallel Skeletons Profiling
Languages Chapel (Cray inc. HPCS Language) Concurrent ML Eden Fortress (Sun HPCS Language) Glasgow Parallel Haskell Multilisp NESL
Bibliographic Notes and Further Reading A sound introduction to parallel functional language concepts and research can be found in []. A comprehensive survey of parallel and distributed languages based on Haskell is available in []. A recent series of workshops studies declarative aspects of multicore programming, for example [].
Bibliography . Allen E, Chase D, Flood C, Luchangco V, Maessen J-W, Steele S, Ryu GL () Project fortress: a multicore language for multicore processors. Linux Magazine, September . Armstrong J () Programming Erlang: software for a concurrent world. The Pragmatic Bookshelf, Raleigh, NC
. Bacci B, Danelutto M, Orlando S, Pelagatti S, Vanneschi M () P L: a structured high level programming language and its structured support. Concurrency Practice Exp ():– . Berthold J, Loogen R () Visualizing parallel functional program runs – case studies with the eden trace viewer. In: ParCo’: Proceedings of the parallel computing: architectures, algorithms and applications, Jülich, Germany . Blelloch GE () Programming parallel algorithms. Commun ACM ():– . Breitinger S, Loogen R, Priebe S () Parallel programming with Haskell and MPI. In: IFL’ – International Workshop on the implementation of functional languages, University College London, UK, September . Draft proceedings, pp – . Blelloch GE, Miller GL, Talmor D () Developing a practical projection-based parallel Delaunay algorithm. In: Symposium on Computational Geometry. ACM, Philadelphia, PA . Blelloch GE, Narlikar G () A practical comparison of N-body algorithms. In: Parallel Algorithms, volume of Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society, University of Tennessee, Knoxville, TN . Cann D () Retire Fortran? A debate rekindled. Commun ACM ():– . Chamberlain BL, Callahan D, Zima HP () Parallel programmability and the chapel language. Int J High Perform Comput Appl ():– . Charles P, Donawa C, Ebcioglu K, Grothoff C, Kielstra A, von Praun C, Saraswat V, Sarkar V () X: An object-oriented approach to non-uniform cluster computing. In: OOPSLA’. ACM Press, New York . Carriero N, Gelernter D () How to write parallel programs: a guide to the perplexed. ACM Comput Surv ():– . Chakravarty MMT (ed) () ACM SIGPLAN Workshop on Declarative Aspects of Multicore Programming (DAMP’). ACM Press, New York . Cole M () Algorithmic skeletons. In: Hammond K, Michaelson G (eds) Research directions in parallel functional programming. Springer-Verlag, New York, pp – . Darlington J, Guo Y, To HW () Structured parallel programming: theory meets practice. In Milner R, Wand I (eds) Research Directions in Computer Science. Cambridge University Press, Cambridge, MA . Ellmenreich N, Lengauer C () Costing stepwise refinements of parallel programs. Comp Lang Syst Struct (–):– (special issue on semantics and cost models for high-level parallel programming) . Flanagan C, Nikhil RS () pHluid: the design of a parallel functional language implementation on workstations. In: ICFP’ – International Conference on functional programming, ACM Press, Philadelphia, PA, pp – . Foster I, Olson R, Tuecke S () Productive parallel programming: The PCN approach. J Scientific Program ():– . Fluet M, Rainey M, Reppy J, Shaw A () Implicitly-threaded parallelism in Manticore. In: ICFP’ – International Conference on functional programming, ACM Press, Victoria, BC
Futures
. Glasgow Haskell Compiler. WWW page, . http://www.haskell.org/ghc/ . Herrmann C, Lengauer C () HDC: a higher-order language for divide-and-conquer. Parallel Processing Lett (–): – . Hammond K, Michaelson G () Research directions in parallel functional programming. Springer-Verlag, New York . Hughes J () Why functional programming matters. Computer J ():– . Jones Jr D, Marlow S, Singh S () Parallel performance tuning for haskell. In: Haskell ’: Proceedings of the nd ACM SIGPLAN symposium on Haskell. ACM Press, New York . Kelly PHJ () Functional programming for loosely-coupled multiprocessors. Research monographs in parallel and distributed computing. MIT Press, Cambridge, MA . Kelly P, Taylor F () Coordination languages. In: Hammond K, Michaelson G (eds) Research directions in parallel functional programming. Springer-Verlag, New York, pp – . Sisal Performance Data. WWW page, January . http://www.llnl.gov/sisal/PerformanceData. . Loogen R, Ortega-Mallen Y, Pena R () Parallel functional programming in Eden. J Funct Program ():– . MPI-: Extensions to the message-passing interface. Technical report, University of Tennessee, Knoxville, TN, July . Chakravarty MMT, Leshchinskiy R, Jones SP, Keller G () Partial vectorisation of haskell programs. In DAMP’: ACM SIGPLAN workshop on declarative aspects of multicore programming, San Francisco, CA . Michaelson G, Horiguchi S, Scaife N, Bristow P () A parallel SML compiler based on algorithmic skeletons. J Funct Program ():– . Nöcker EGJMH, Smetsers JEW, van Eekelen MCJD, Plasmeijer MJ () Concurrent clean. In: PARLE’ — parallel architectures and languages Europe, LNCS . Springer-Verlag, Veldhoven, the Netherlands, pp – . O’Donnell J () Data parallelism. In: Hammond K, Michaelson G (eds) Research directions in parallel functional programming. Springer-Verlag, New York, pp – . Jones SLP, Hughes J, Augustsson L, Barton D, Boutel B, Burton W, Fasel J, Hammond K, Hinze R, Hudak P, Johnsson T, Jones M, Launchbury J, Meijer E, Peterson J, Reid A, Runciman C, Wadler P () Haskell : a non-strict, purely functional language. Electronic document available on-line at http://www.haskell.org/, February . Peterson J, Trifonov V, Serjantov A () Parallel functional reactive programming. In: PADL’ – practical aspects of declarative languages, LNCS . Springer-Verlag, New York, pp – . Parallel Virtual Machine Reference Manual. University of Tennessee, August . Rabhi FA, Gorlatch S (eds) () Patterns and skeletons for parallel and distributed computing. Springer-Verlag, New York, . Skillicorn DB, Cai W () A cost calculus for parallel functional programming. J Parallel Distrib Comput ():–
F
. Taylor FS () Parallel functional programming by partitioning. PhD thesis, Department of Computing, Imperial College, London . Trinder PW, Hammond K, Loidl H-W, Jones SLP () Algorithm + Strategy = Parallelism. J Funct Program (): – . Trinder PW, Hammond K, Mattson Jr JS, Partridge AS, Jones SLP () GUM: a portable implementation of Haskell. In: PLDI’ — programming language design and implementation, Philadephia, PA, May . Trinder PW, Loidl H-W, Pointon RF () Parallel and distributed Haskells. JFP, . Special issue on Haskell . Weber M () hMPI — Haskell with MPI. WWW page, July . http://www-i.informatik.rwth-aachen.de/~michaelw/ hmpi.html . Wegner P () Programming languages, information structures and machine organisation. McGraw-Hill, New York
Futures Cormac Flanagan University of California at Santa Cruz, Santa Cruz, CA, USA
Synonyms Eventual values; Promises
Definition The programming language construct “future E” indicates that the evaluation of the expression “E” may proceed in parallel with the evaluation of the rest of the program. It is typically implemented by immediately returning a future object that is a proxy for the eventual value of “E.” Attempts to access the value of an undetermined future object block until that value is determined.
Discussion Introduction The construct “future E” permits the run-time system to evaluate the expression “E” in parallel with the rest of the program. To initiate parallel evaluation, the runtime system forks a parallel task that evaluates “E,” and also creates an object known as a future object that will eventually contain the result of that computation of “E.” The future object is initially undetermined, and becomes
F
F
Futures
determined or resolved once the parallel task evaluates “E” and updates (or resolves/fulfils/binds) the future object to contain the result of that computation. Parallel evaluation arises because the future object is immediately returned as the result of evaluating “future E,” without waiting for the future object to become determined. Thus, the continuation (or enclosing context) of “future E” executes in parallel with the evaluation of “E” by the forked task. The result of performing a primitive operation on a future object depends on whether that operation is considered strict or not. Some program operations, such as assignments, initializing a data structure, passing an argument to a procedure, returning a result from a procedure, are non-strict. These operations can manipulate (determined or undetermined) future objects in an entirely transparent fashion, and do not need to know the underlying value of that future object. In contrast, strict program operations, such as addition, need to know the value of their arguments, and so cannot be directly applied to future objects. Instead, the arguments to these operations must first be converted to a regular (non-future) value, by forcing or touching that argument. Forcing a future object that has been determined extracts its underlying value (which may in turn be a future and so the force operation must operate recursively). Forcing an undetermined future object blocks until that future is determined, and so force operations introduce synchronization between parallel tasks.
Implicit Versus Explicit Futures A central design choice in a language with futures is whether forcing operations should be explicit in the source code, or whether they should be performed implicitly by strict operations, such as addition, that need to know the values of their argument. Explicit futures can be implemented as a library that introduces an additional data type. For example, the polymorphic type Future
introduce additional overheads, since every strict primitive operation must now check whether its operands are futures. As an optimization, a static analysis can be used to translate from a language with implicit futures into one with explicit futures. Essentially, the translation introduces explicit touch operations on those primitive operations that, according to a static analysis, may be applied to futures. In the context of the Gambit compiler, this optimization reduces the overhead of implicit futures from roughly % to less than % []. As an orthogonal optimization, a copying garbage collector could replace a determined future object by its underlying value, saving space and reducing the cost of subsequent touch operations.
Task Scheduling Future annotations are often used to introduce parallel evaluation into recursive, divide-and-conquer style computations such as quicksort. In this situation, there may be exponentially more tasks than processors, and consequently task scheduling has a significant impact on overall performance. A naïve choice of roundrobin scheduling would interleave the executions of all available tasks and result in an essentially breadthfirst exploration of the computation tree, which may requires exponentially more memory than an equivalent sequential execution. Unfair scheduling policies significantly reduce this memory overhead, by better matching the number of concurrently active tasks to the number of processors. In the Multilisp implementation of this technique [], each task has two possible states: active and pending, and each processor maintains a LIFO queue of pending tasks. When evaluating “future E,” the newly forked task evaluating “E” is marked active, while the parent task is moved to the pending queue. In the absence of idle processors, the parent task remains in the pending queue until the task evaluating “E” terminates, resulting in the same evaluation order (and memory footprint) as sequential execution, and the pending task queue functions much like the control stack of a sequential execution. The benefit of the pending task queue is that if another processor becomes idle, it can steal and start evaluating one of these pending tasks. In this manner, the number of active tasks in the system is roughly
Futures
dynamically balanced with the number of available processors. Each processor typically has just one active task. Once that task terminates, it removes and makes active the next task in its pending queue. If the processor’s pending task queue becomes empty, it tries to steal and activate a pending task from one of the other processors in the system. An active task may block by trying to force an undetermined future object. In this situation, the task is typically put on a waiting list associated with that future. Once the future is later determined, all tasks on the waiting list are marked as pending, and will become active once an idle processor is available.
Lazy Task Creation Lazy task creation [, ] provides a mechanism for reducing task creation overhead. Under this mechanism, the evaluation of “future E” does not immediately create a new task. Instead, the expression “E” is evaluated as normal, except that the control stack is marked to identify that the continuation of this future expression is implicitly a pending task. Later, an idle processor can inspect this control stack to find this implicit pending task, create the appropriate future object, and start executing this pending task. The control stack is also modified so that after the evaluation of “E” terminates, its value is used to resolve that future object. In the common case, however, the evaluation of “E” will terminate without the implicit pending task having been stolen by a different processor. In this situation, the result of “E” will be returned directly to the context of “future E” which still resides on the control stack, with little additional overhead and without any need to create a new task or future object.
Futures in Purely Functional Languages Programs in purely functional languages offer numerous opportunities for executing program components in parallel. For example, the evaluation of a function application could spawn a parallel task for the evaluation of each argument expression. Applying such a strategy indiscriminately, however, leads to many parallel tasks whose overhead outweighs any benefits of parallel execution.
F
Futures provide a simple method for taming the implicit parallelism of purely functional programs. A programmer who believes that the parallel evaluation of some expression outweighs the overhead of creating a separate task may annotate the expression with the keyword “future.” In a purely functional language, these future annotations are transparent in that they have no effect on the final values computed by the program, and only influence the amount of parallelism exposed during the program’s computation. Consequently, purely functional programs with future are deterministic, just like their sequential counterparts.
Futures in Mostly Functional Languages Futures have been used effectively in “mostly functional” languages such as Multilisp [] and Scheme [], where most computation is performed in a functional style and assignment statements are limited to situations where they provide increased expressiveness. In this setting, futures are not transparent, as they introduce parallelism that may change the order in which assignments are performed. To preserve determinism, the programmer needs to ensure there are no race conditions between parallel computations, possibly by introducing additional forcing operations that block until a particular task has terminated and all of its side effects have been performed. Mostly functional languages include a side effect– free subset with substantial expressive power, and so the future construct is still effective for exposing parallelism within functional sub-computations written in these languages.
Deterministic Futures in Imperative Languages Futures have been combined with transactional memory techniques to guarantee determinism even in the presence of side effects. Essentially, each task is considered to be a transaction, and these transactions must commit in a fixed, as-if-sequential order. Each transaction tracks its read and write sets. If one transaction accesses data that another transaction is mutating, then the later transaction is rolled back and restarted [].
F
F
Futures
Futures in Distributed Systems Future objects have been used in distributed systems to hide the latency associated with a remote procedure call (RPC). Instead of waiting for a message send and a matching response from a remote machine, an RPC can immediately return a future object as a proxy for the eventual result of that call. Note that this future could be returned even before the first RPC message is sent over the network. Consequently, if one machine performs multiple RPCs to the same remote machine, then these calls can all be batched into a single message; this technique is referred to as promise pipelining.
Thread-Specific Futures, Resolvable Futures, and I-vars The programming language construct “future E” described above does not expose the ability to resolve the future to an arbitrary value; it can only be resolved to the result of evaluating the expression “E.” An I-var is analogous to a future object in that it functions as a proxy for a value that may not yet be determined. The key difference is that an I-var does not have an explicit associated thread that is computing its value (as in the case for future objects). Instead, any thread can update or resolve the I-var. Thus, an I-var is essentially a single entry, one-shot queue. Attempts to access an I-var before it is determined block, as with future objects. Somewhat confusingly, the terms “future” and “promise” are sometimes used to mean an I-var object that also exposes this resolve operation. The terms “thread-specific future” and “resolvable future” help disambiguate these distinct meanings.
Futures and Lazy Evaluation Futures are closely related to lazy evaluation, in that both immediately return a placeholder for the result of a pending computation. A major difference is that future expressions typically must be evaluated before program termination, but lazy expressions need not be evaluated before program termination. This semantic difference means that preemptive evaluation of lazy expressions is considered speculative, whereas evaluation of future expressions is considered mandatory. Sometimes, as in the language Alice ML, the term “lazy future” is used to denote a future whose expression
is evaluated lazily, that is, when the future object is first touched.
Futures and Exceptions In the construct “future E,” an interesting design choice is how to handle any exceptions that are raised during the evaluation of “E,” since enclosing exception handlers from the context of “future E” may be no longer active. A more general version of this question is how the future construct should interact with call/cc (call-withcurrent-continuation). This question has been studied by a number of researchers (see, e.g., Moreau []), with the goal of providing determinism guarantees in the presence of these constructs. Some languages take a more pragmatic choice, whereby if “E” raises an exception, then any attempt to touch the corresponding future object will also raise that exception.
Related Entries Cilk
Bibliographic Notes and Further Reading Friedman and Wise introduced the term “promise” in []. Peter Hibbard described a form of explicit futures, called “eventual values,” in the context of Algol []. Baker and Hewitt [] proposed implicit futures for the parallel evaluation of functional languages. Implicit futures were implemented in the Multilisp version of Scheme in the mid-s [], and futures have since been adapted to a variety of other languages and compilers, including the Gambit Scheme compiler [], Mul-T [], Butterfly Lisp, Portable Standard Lisp, Act , Alice ML, Habanero Java, and many others. Libraries supporting futures have been implemented for several languages, including (to name just a few of many examples) Java, C++x, OCaml, python, Perl, and Ruby. The single-assignment I-var structure originated in Id [] and was included in subsequent languages such as parallel Haskell and Concurrent ML.
Bibliography . Halstead R (Oct ) Multilisp: A language for concurrent symbolic computation. ACM Trans Program Lang Syst ():–
Futures
. Feeley M () An efficient and general implementation of futures on large scale shared-memory multiprocessors. PhD thesis, Department of Computer Science, Brandeis University . Flanagan C, Felleisen M () The semantics of future and its use in program optimization. In: Proceedings of the ACM SIGPLAN-SIGACT symposium on the principles of programming languages. POPL’, pp – . Moreau L () Sound evaluation of parallel functional programs with first-class continuations. PhD thesis, Universite de Liege . Kranz D, Halstead R, Mohr E () Mul-T: A high performance parallel lisp. In: Proceedings of the ACM SIGPLAN Conference on Programming language design and implementation, Portland, pp – . Knueven P, Hibbard P, Leverett B (June ) A language system for a multiprocessor environment. In: Proceedings of the th international conference on design and implementation of algorithmic languages, Courant Institute of mathematical Studies, New York, pp – . Friedman DP, Wise DS () The impact of applicative programming on multiprocessing. International conference on parallel processing, pp –
F
. Welc A, Jagannathan S, Hosking A () Safe futures for Java. In: Proceedings of the th annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications (OOPSLA ), ACM, pp – . Baker H, Hewitt C (Aug ) The incremental garbage collection of processes. In: Proceedings of symposium on AI and programming languages, ACM SIGPLAN Notices (): – . Moreau L () The semantics of scheme with future. In: Proceedings of the first ACM SIGPLAN international conference on functional programming, pp – . Mohr E, Kranz D, Halstead R () Lazy task creation: a technique for increasing the granularity of parallel programs. In: Proceedings of the ACM conference on LISP and functional programming, pp – . Arvind, Nikhil RS, Pingali KK () I-Structures: Data structures for parallel computing. ACM Trans Program Lang Syst ():–
F
G GA Global Arrays Parallel Programming Toolkit
Gather Collective Communication
Gather-to-All Allgather
Gaussian Elimination Dense Linear System Solvers Sparse Direct Methods
GCD Test Banerjee’s Dependence Test Dependences
Gene Networks Reconstruction Systems Biology, Network Inference in
Generalized Meshes and Tori Hypercubes and Meshes
Genome Assembly Ananth Kalyanaraman Washington State University, Pullman, WA, USA
Synonyms Genome sequencing
Definition Genome assembly is the computational process of deciphering the sequence composition of the genetic material (DNA) within the cell of an organism, using numerous short sequences called reads derived from different portions of the target DNA as input. The term genome is a collective reference to all the DNA molecules in the cell of an organism. Sequencing generally refers to the experimental (wetlab) process of determining the sequence composition of biomolecules such as DNA, RNA, and protein. In the context of genome assembly, however, the term is more commonly used to refer to the experimental (wetlab) process of generating reads from the set of chromosomes that constitutes the genome of an organism. Genome assembly is the computational step that follows sequencing with the objective of reconstructing the genome from its reads.
Discussion Introduction
Gene Networks Reverse-Engineering Systems Biology, Network Inference in
Deoxyribonucleic acid (or DNA) is a double-stranded molecule which forms the genetic basis in most known organisms. The DNA along with other molecules, such as the ribonucleic acid (or RNA) and proteins, collectively constitute the subject of study in various
David Padua (ed.), Encyclopedia of Parallel Computing, DOI ./----, © Springer Science+Business Media, LLC
G
Genome Assembly
branches of biology such as molecular biology, genetics, genomics, proteomics, and systems biology. A DNA molecule is made up of two equal length strands with opposite directionality, with the ends labeled from ′ to ′ on one and ′ to ′ on the other. Each strand is a sequence of smaller molecules called nucleotides, and each nucleotide contains one of the four possible nitrogenous bases – adenine (a), cytosine (c), guanine (g), and thymine (t). Therefore for computational purposes, each strand can be represented in the form of a string over the alphabet {a, c, g, t}, expressed always in the direction from ′ to ′ . Furthermore, the base at a given position in one strand is related to the base at the corresponding position in the other strand by the following base-pairing rule (referred to as “base complementarity”): a ↔ t, c ↔ g. Therefore, the sequence of one strand can be directly deduced from the sequence of the other. For example, if one strand is ′ agaccagttac ′ , then the other is ′ gtaactggtct ′ . The length of a genome is measured in the number of its base pairs (“bp”). Genomes range in length from being just a few million base pairs in microbes to several billions of base pairs in many eukaryotic organisms. However, all sequencing technologies available till date, since the invention of Sanger sequencing in late s, have been limited to accurately sequencing DNA molecules no longer than ∼ Kbp. Consequently, scientists have had to deploy alternative sequencing strategies for extending the reach of technology to genome scale. The most popular strategy is the whole genome shotgun (or WGS) strategy, where multiple copies of a single long target genome are shredded randomly into numerous fragments of sequenceable length and the corresponding reads sequenced individually using any standard technology. Another popular albeit more expensive strategy is the hierarchical approach, where a library of smaller molecules called Bacterial Artificial Chromosomes (or BACs) is constructed. Each BAC is ∼ Kbp in length and they collectively provide a minimum tiling path over the entire length of the genome. Subsequently, the BACs are sequenced individually using the shotgun approach. In both approaches, the information of the relative ordering among the sequenced reads is lost during sequencing either completely or almost completely. Therefore, the primary information that genome assemblers should rely upon is the end-to-end sequence
overlap preserved between reads that were sequenced from overlapping regions along the target genome. To increase the chance of overlap, the target genome is typically sequenced in a redundant fashion. This is referred to as genome coverage. A higher coverage typically tends to provide information for a more accurate assembly, although at increased costs of generation and analysis. The fact that reads could have originated from an arbitrary strand adds another dimension to the complexity of the reconstruction process. The process is further complicated by other factors such as errors introduced during read sequencing and the presence of genomic repeats that could ambiguate overlap detection and read placement. To partially aid the resolving of the genomic repeat regions, sequencing is sometimes performed in pairs from clonal insert libraries, where the genomic distance between the two reads of a pair can be estimated, typically in the – Kbp range. This technique is called pair-end sequencing and genome assemblers could take advantage of this information to resolve repeats that are smaller than the specified range. Genome assembly is a classical computational problem in the field of bioinformatics and computational biology. More than two decades of research has yielded a number of approaches and algorithms and yet the problem continues to be actively pursued. This is because of several reasons. While there are different ways of formulating the genome assembly problem, all known formulations are NP-Hard. Therefore, researchers continue to work on developing improved, efficient approximation and heuristic algorithms. However, even the best serial algorithms take tens of thousands of CPU hours for eukaryotic genomes owing to their: large genomic sizes, which increase the number of reads to be analyzed; and high genomic repeat complexity, which adds substantial processing overheads. For instance, in the Celera’s WGS version of the human genome assembly published in , over million reads representing .x coverage over the genome were assembled in about , CPU h. A paradigm shift in sequencing technologies has further aggravated the data-intensiveness of the problem and thereby the need for continued algorithmic development. Until the mid-s, the only available technology for use in genome assembly projects was Sanger sequencing, which produces reads of approximate length Kbp each. Since then, a slew of
Genome Assembly
high-throughput sequencing technologies, collectively referred to as the next-generation sequencing technologies, have emerged, significantly revitalizing the sequencing community. Examples of next-generation technologies include Roche Life Sciences system’s pyrosequencing (read length ∼ bp), Illumina Genome Analyzer and HiSeq (read length ∼– bp); Life Technologies SOLiD (read length ∼ bp) and Helicos HeliScope (read length ∼ bp). While these instruments generate much shorter reads than Sanger, they do so at a much faster rate effectively producing several hundred millions of reads in a single experiment, and at significantly reduced costs (about – times cheaper). These attractive features are essentially democratizing the sequencing process and broadening community contribution to sequenced data. From a genome assembly perspective, a shorter read length could easily deteriorate the assembly quality because the reads are more likely to exhibit false or insufficient overlaps. To offset this shortcoming, sequencing is required at a much higher coverage (x–x) than for Sanger sequencing. The higher coverage has another desirable effect in that, because of its built-in redundancy, it could aid in a more reliable identification of real genomic variations and their differentiation from experimental artifacts. Detecting genomic variations such as single nucleotide polymorphisms (SNPs) is of prime importance in comparative and population genomics. This combination of high coverage and short read lengths makes the short read genome assembly problem significantly more data-intensive than for Sanger reads. For instance, any project aiming to reconstruct a mammalian genome (e.g., human genome) de novo using any of the current next-generation technologies would have to contend with finding a way to assemble several billion short reads. This changing landscape in technology and application has led to the development of a new generation of short read assemblers. Although traditional developmental efforts have been targeting serial computers, the increasing data-intensiveness and the increasing complexity of genomes being sequenced have gradually pushed the community toward parallel processing, for what has now become an active branch within the area. In what follows, an overview of the assembly problem, with its different formulations and algorithmic solutions is presented. This entry is not intended to be
G
a survey of tools for genome assembly. Rather, it will focus on the parallelism in the problem and related efforts in parallel algorithm development. In order to set the stage, the key ideas from the corresponding serial approach will also be presented as necessary. The bulk of the discussion will be on de novo assembly, which is typically the harder task of reconstructing a genome from its reads assuming no prior knowledge about the sequence of the genome except for its reads and as available, their pair-end sequencing information.
Algorithmic Formulations of Genome Assembly The genome assembly problem: Let R = {r , r . . . rm } denote a set of m reads sequenced from an unknown target genome G of length g. The problem of genome assembly is to reconstruct the sequence of genome G from R. As a caveat, for nearly all input scenarios, the outcome expected is not a single assembled sequence but a set of assembled sequences for a couple of reasons. Typically, the genome of a species comprises of multiple chromosomes (e.g., pairs in the human genome) and therefore each chromosome can be treated as an individual sequence target. Note that, however, the information about the source chromosome for a read is lost during a sequencing process such as WGS and it is left for the assembler to detect and sort these reads by chromosomes. Furthermore, any shotgun sequencing procedure tends to leave out “gaps” along the chromosomal DNA during sampling, and therefore it is possible to reconstruct the genome only for those sampled sections. It is expected that, through incorporation of pair-end information, at least a subset of these assembled products (called “contigs” in genome assembly parlance) can be partially ordered and oriented. Notation and terminology: Let s denote a sequence over a fixed alphabet Σ. Unless otherwise specified, a DNA alphabet is assumed – i.e., Σ = {a, c, g, t}. Let ∣s∣ denote the length of s; and s[i . . . j] denote the substring of s starting and ending at indices i and j respectively, with the convention that string indexing starts at . A prefix i (alternatively, suffix i) of s is s[ . . . i] (alternatively, n s[i . . . ∣s∣]). Let n = Σ m i= ∣ri ∣ and ℓ = m . The sequencing g coverage c is given by n . The terms string and sequence are used interchangeably. Let p denote the number of processors in a parallel computer.
G
G
Genome Assembly
The most simplistic formulation of genome assembly is that of the Shortest Superstring Problem (SSP). A superstring is a string that contains each input read as a substring. The SSP formulation is NP-complete. Furthermore, its assumptions are not realistic in practice. Reads can contain sequencing errors and hence they need not always appear preserved as substrings in the genome; and genomes typically are longer than the shortest superstring due to presence of repeats and nearly identical genic regions (called paralogs). Among the more realistic models for genome assembly, graph theoretic formulations have been in the forefront. In broad terms, there are three distinct ways to model the problem (refer to Fig. ):
were gaps during sequencing, then one Hamiltonian Path is sought for every connected component in G (if it exists) and the genome can be recovered as an unordered set of pieces corresponding to the sequenced sections. This formulation using the overlap graph model has been shown to be NP-Hard. () De Bruijn graph: Let a k-mer denote a string of length k. Construct a De Bruijn graph, where the vertex set is the set of all k-mers contained inside the reads of R; and a directed edge is drawn from vertices i to j, iff the k − length suffix of the k-mer i is identical to the k − length prefix of the k-mer j. Put another way, each edge in E represents a unique k + -mer that is found in at least one of the reads in R. Given a De Bruijn graph G, genome assembly is the problem of finding a shortest Euler tour in which each read is represented by a sub-path in the tour. While finding an Euler tour is polynomially solvable, the optimization problem is still NP-Hard by reduction from SSP. () String graph: This model is a variant of the overlap graph in which the edges represent reads, and vertices represent branching of end-to-end overlaps of the adjoining reads. For example, if a read ri overlaps with both rj and rk then it is represented by a vertex
() Overlap graph: Construct graph G(V, E) from R, where each read is represented by a unique vertex in V and an edge is drawn between (ri , rj ) iff there is a significant suffix–prefix overlap between ri and rj . Overlap is typically defined using a pairwise sequence alignment model (e.g., semi-global alignment) and related metrics to assess the quality of an alignment. Given G, the genome assembly problem can be reduced to the problem of finding a Hamiltonian Path, if one exists, provided that there were no breaks (called sequencing gaps) along the genome during sequencing. If there Unknown genome Repeat
Repeat
Reads
Overlap graph
cagtcaac agtcagt cagttcgg tcaa caac
De Bruijn graph agtc gtca tcag cagt
agtt gttc ttcg tcgg String graph r1
r3
r2
r4
Genome Assembly. Fig. Illustration of the three different models for genome assembly. The top section of the figure shows a hypothetical unknown genome and the reads sequenced from it. The dotted box shows a repetitive region within the genome. In the overlap graph, solid lines indicate true overlaps and dotted lines show overlaps induced by the presence of the repeat. The De Bruijn graph shows the graph for an arbitrary section involving three reads. The string graph shows an example of a branching vertex with multiple possible entry and exit paths
Genome Assembly
which has an inbound edge corresponding to ri and two outbound edges representing rj and rk . Furthermore, the graph is pruned off transitively inferable edges – e.g., if both ri and rj overlaps with rk , whereas ri also overlaps with rj , then the overlap from rj to rk is inferable and is therefore removed. Given a string graph G, the assembly problem becomes one that is equivalent of finding a constrained least cost Chinese Postman tour of G. This formulation has also been shown to be NP-Hard. All the three formulations incorporate features to handle sequencing errors. In the overlap graph model, the pairwise alignment formulation used to define edges automatically takes into account character substitutions, insertions, and deletions. In the other two models, an “error correction” procedure is run as a preprocessing step to mask off anomalous portions of the graph prior to performing the assembly tours.
Parallelization for the Overlap Graph Model Algorithms that use the overlap graph model deploy a three-stage assembly approach of overlap–layout– consensus. In the first stage, the overlap graph is constructed by performing all-against-all pairwise comparison of the reads in R. As a result, a layout for the assembly is prepared using the pairwise overlaps and in the third stage, a multiple sequence alignment (MSA) of the reads is performed as dictated by the layout. Given the large number of reads generated in a typical sequencing project, the overlap computation phase dominates the run-time and hence is the primary target for parallelization and performance optimization. A brute-force way of implementing this step will be to compare all (m ) possible pairs. This approach can also be parallelized easily, as individual alignment tasks can be distributed in a load-balanced fashion given the uniformity expected in the read lengths. However, an implementation may not be practically feasible for larger number of reads due to the quadratic increase in alignment workload. Each alignment task executed serially could take milliseconds of run-time. In fact, for the assembly problem, the nature of sampling done during sequencing guarantees that performing all-againstall comparisons is destined to be highly wasteful. This is because of the random shotgun approach to sequencing
G
which is expected to sample roughly equal number of reads covering each genomic base and it is typically only reads that originate from the same locus that tend to overlap with one another, with the exception of reads from repetitive regions of the genome. A more scalable approach to detect pairwise overlaps is to first shortlist pairs of sequences based on the presence of sufficiently long exact matches (i.e., a filter built on a necessary-but-not-a-sufficient condition) and then compute pairwise comparisons only on them. Methods vary in the manner in which these “promising pairs” are shortlisted. One way of generating promising pairs is to use a string lookup table data structure that is built to record all k-mer occurrences within the reads in linear time. Sequence pairs that share one or more k-mers can be subsequently considered for alignmentbased evaluation. While this approach supports a simple implementation, it poses a few notable limitations. The size complexity of the lookup table data structure contains an exponential term O(∣Σ∣k ). This restricts the value of k in practice; for DNA alphabet, physical memory constraints demand that k ≤ . Whereas, reads could share arbitrarily long matches, especially if the sequencing error rates are low. Another disadvantage of the lookup table is that it is only suited to detect pairs with fixed length matches. An exact match of an arbitrary length q will reveal itself as q − k + smaller k-mer matches, thus increasing computation cost. Furthermore, the distribution of entries within the lookup table is highly data dependent and could scatter the same sequence pair in different locations of the lookup table, making it difficult for parallelization on distributed memory machines. Methods that use lookup tables simply use a high end shared memory machine for serial generation and storage, and focus on parallelizing only the pairwise sequence comparison step. An alternative albeit more efficient way of enumerating promising pairs based on exact matches is to use a string indexing data structure such as suffix tree or suffix array that will allow detection of variable length matches. A match α between two reads ri and rj is called left-maximal (alternatively, right-maximal) if the characters preceding (alternatively, following) it in both strings, if any, are different. A match α between reads ri and rj is said to be a maximal match between the two reads if it is both left- and right-maximal. Pairs containing maximal matches of a minimum length cutoff, say
G
G
Genome Assembly
ψ, can be detected in optimal run-time and linear space using a suffix tree data structure. A generalized suffix tree of a set of strings is the compacted trie of all suffixes in the strings. The pair enumeration algorithm first builds a generalized suffix tree over all reads in R and then navigates the tree in a bottom-up order.
Parallel Generalized Suffix Tree Construction For constructing a suffix tree, there are several linear time construction serial algorithms and optimal algorithms for the CRCW-PRAM model. However, optimal construction under the distributed memory machine model, where memory per processor is assumed to be too small to store the input sequences in R, is an open problem. Of the several approaches that have been studied, the following approach is notable as it has demonstrated linear scaling to thousands of processors on distributed memory supercomputers. The major steps of the underlying algorithm are as follows. Each step is a parallel step. The steps marked as comm are communication-bound and the steps marked comp are computation-bound. S. (comp) Load R from I/O in a distributed fashion such that each processor receives O( np ) characters and no read is split across processor boundaries. S. (comp) Slide a window of length k ≤ ψ over the set of local reads and bucket sort locally all suffixes of reads based on their k length prefix. Note that for DNA alphabet, even a value as small as for k is expected to create over a million buckets, sufficient to support parallel distribution. An alternative to local generation of buckets is to have a master– worker paradigm where the reads are scanned in small-sized batches and have a dedicated master distribute batches to workers in a load-balanced fashion. S. (comm) Parallel sort the buckets such that each processor receives approximately ∼ np suffixes and no bucket is split across processor boundaries. While the latter cannot be theoretically guaranteed, real-world genomic data sets typically distribute the suffixes uniformly across buckets thereby virtually ensuring that no bucket exceeds the O( np ) bound. If a bucket size becomes too big to fit in the local memory, then an alternative strategy can be deployed
whereby the prefix length for bucketing is iteratively extended until the size of the bucket fits in the local memory. S. (comm, comp) Each bucket in the output corresponds to a unique subtree in the generalized suffix tree rooted exactly k characters below the root. The subtree corresponding to each bucket is then locally constructed using a recursive bucket sort–based method that compares characters between suffixes. However, this step cannot be done strictly locally because it needs the sequences of reads whose suffixes are being compared. This can be achieved by building the local set of subtrees in small-sized batches and performing one round of communication using Alltoallv() to load the reads required to build each batch from other processors. In order to reduce the overhead due to read fetches, observe that the aggregate memory on a relatively small group of processors is typically sufficient to accommodate the read set for most inputs in practice. For example, even a set of million reads each of length Kbp needs only of the order of GB. Even assuming GB per processor, this means the read set will fit within processors. Higher number of processors is required only to speedup computation. One could take advantage of this observation by partitioning the processor space into subgroups of fixed, smaller size, determined by the input size, and then have the on-the-fly read fetches to seek data from processors from within each subgroup. This improved scheme could distribute the load of any potential hot spots and also the overall communication could be faster as the collective transportation primitive now operates on a smaller number of processors. The above approach can be implemented to run in time and O( np ) space, where ℓ is the mean read length. The output of the algorithm is a distributed representation of the generalized suffix tree, with each processor storing roughly np suffixes (or leaves in the tree). ) O( nℓ p
The cluster-then-assemble Approach A potential stumbling block in the overlap–layout– consensus approach is the large amount of memory required to store and retrieve the pairwise overlaps so that they can be used for generating an assembly layout. For genomes which have been sequenced at a uniform
Genome Assembly
coverage c, the expectation is that each base along the genome is covered by c reads on an average, thereby implying (c ) overlapping read pairs for every genomic base. However, this theoretical linear expectation holds for only a fraction of the target genome, whereas factors such as genomic repeats and oversampled genic regions could increase the number of overlapping pairs arbitrarily beyond the expected level. An example case in point is the gene-enriched sequencing of the highly repetitive ∼. billion bp maize genome. One way to overcome this scalability bottleneck is to use clustering as a preprocessing step to assembly. This approach, referred to as cluster-then-assemble, builds upon the assumption that any genome-scale sequencing effort is likely to undersample the genome, leaving out sequencing gaps along the genome length and thereby allowing the assemblers to reconstruct the genome in pieces – one piece for every contiguous stretch of the genome (aka “contig” or “genomic island”) that is fully sampled through sequencing. This assumption is not unrealistic as tens of thousands of sequencing gaps have always resulted in almost all eukaryotic genomes sequenced so far using the WGS approach. The cluster-then-assemble takes advantage of this assumption as follows: The set of m reads is first partitioned into groups using a single-linkage transitive closure clustering method, such that there exists a sequence of overlapping pairs connecting every read to every other read in the same cluster. Ideally, each cluster
G
should correspond to one genomic island, although the presence of genomic repeats could collapse reads belonging to different islands together. Once clustered, each cluster represents an independent subproblem for assembly, thereby breaking a large problem with m reads into numerous, disjoint subproblems of significantly reduced size small enough to fit in a serial computer. In practice, tens of thousands of clusters are generated making the approach highly suited for trivial parallelization after clustering. The primary challenge is to implement the clustering step in parallel, in a time- and space-efficient manner. In graph-theoretic terms, the output produced by a transitive closure clustering is representative of connected components in the overlap graph, thereby allowing the problem to be reduced to one of connected component detection. However, generating the entire graph G prior to detection would contradict the purpose of clustering from a space complexity standpoint. Figure outlines an algorithm to perform sequence clustering. The algorithm can be described as follows: Initialize each read in a cluster of its own. In an iterative process, generate promising pairs based on maximal matches in a non-increasing order of their lengths using suffix trees (as explained in the previous section). Before assigning a promising pair for further alignment processing, a check is made to see whether the constituent reads are in the same cluster. If they are part of the same cluster already, then the pair is discarded; otherwise it
Algorithm 1 Read Clustering Input: Read set R = {r1 ; r2 ; : : : rm } Output: A partition C = {C1 ; C2 ; : : : Ck } of R, 1 ≤ k ≤ m 1. Initialize Clusters: C ← { {ri } | 1 ≤ i ≤ m} 2. FOR each pair (ri ; rj ) with a maximal match of length ≥ Ã generated in non-increasing order of maximal match length Cp ← F ind(ri ) Cq ← F ind(rj ) IF Cp = Cq THEN overlap quality ← Align(ri ; rj ) IF overlap quality is significant THEN U nion(Cp ; Cq ) 3. Output C
⇒ (master) ⇒ (worker) ⇒ (master) ⇒ (master) ⇒ (master) ⇒ (worker) ⇒ (master) ⇒ (worker) ⇒ (master)
Genome Assembly. Fig. Pseudocode for a read clustering algorithm suited for parallelization based on the master-worker model. The site of computation is shown in brackets. Operations on the set of clusters are performed using the Union-Find data structure
G
G
Genome Assembly
is aligned. If the alignment results show a satisfactory overlap (based on user-defined alignment parameters), then merge the clusters containing both reads into one larger cluster. The process is repeated until all promising pairs are exhausted or until no more merges are possible. Using the union–find data structure would ensure the Find and Union calls to run in amortized time proportional to the Inverse Ackerman function – a small constant for all practical inputs. There are several advantages to this clustering strategy. Clustering can be achieved in at most m − merging steps. Checking if a read pair is already clustered before alignment is aimed at reducing the number of pairs aligned. Generating promising pairs in a nonincreasing order of their maximal match lengths is a heuristic that could help identify pairs that are more likely to succeed the alignment test sooner during execution. Furthermore, promising pairs are processed as they are generated, obviating the need to store them. This coupled with the use of the suffix tree data structure implies an O(n) serial space complexity for the clustering algorithm. The parallel algorithm can be implemented using a master–worker paradigm. A dedicated master processor can be responsible for initializing and maintaining the clusters and also for distributing alignment workload to the workers in a load-balanced fashion. The workers at first can generate a distributed representation of the generalized suffix tree in parallel (as explained in the previous section). Subsequently, they can generate promising pairs from their local portion of the tree and send them to the master. To reduce communication overheads, pairs can be sent in arbitrary-sized batches as demanded by the situation of the work queue buffer at the master. The master processor can check the pairs against the current set of clusters, filter out pairs that do not need alignment, and add only those pairs that need alignment to its work queue buffer. The pairs in the work queue buffer can be redistributed to workers in fixed size batches, to have the workers compute alignments and send back the results. Communication overheads can be masked by overlapping alignment computation with communication waits using non-blocking calls. The PaCE software suite implements the above parallel algorithm, and it has demonstrated linear scaling to thousands of processors on a distributed memory supercomputer.
Short Read Assembly The landscape of genome assembly tools has significantly transformed itself over the last few years with the advent of next-generation sequencing technologies. A plethora of new-generation assemblers that collectively operate under the banner of “short read assemblers” is continuing to emerge. Broadly speaking, these tools can be grouped into two categories: () those that follow or extend from the classical overlap graph model; and () those that deploy either the De Bruijn graph or the string graph formulation. The former category includes programs such as Edena, Newbler, PE-Assembler, QSRA, SHARCGS, SSAKE, and VCAKE. The programs that fall under the latter category are largely inspired by two approaches – the De Bruijn graph formulation used in the EULER assembler; and the string graph formulation proposed by Myers. As of this writing, these programs include EULER-SR, ALLPATHS, Velvet, SOAPdenovo, ABySS, and YAGA. Either of these lists is likely to expand in the coming years, as new implementations of short read assemblers are continuing to emerge as are new technologies. In principle, the techniques developed for assembling Sanger reads do not suit direct application for short read assembly due to a combination of factors that include shorter read length, higher sequencing coverage, an increased reliance on pair-end libraries, and idiosyncrasies in sequencing errors. A shorter read length implies that the method has to be more sensitive to differences to avoid potential mis-assemblies. This also means providing a robust support for incorporating the knowledge available from pair-end libraries. In case of next-generation sequencing, pair-end libraries are typically available for different clone insert sizes, which translates to a need to implement multiple sets of distance constraints. A high sequencing coverage introduces complexity at two levels. The number of short reads to assemble increases linearly with increased coverage, and this number can easily reach hundreds of millions even for modest-sized genomes because of shorter read length. For instance, a mid-size genome such as that of Arabidopsis (∼ Mbp) or fruit fly (∼ Mbp) sequenced at x coverage either using Illumina or SOLiD would generate a couple of hundred million reads. Secondly, the average number of overlapping read
Genome Assembly
pairs at every given genomic location is expected to grow quadratically (∝ (c )) with the coverage depth (c). This particularly affects the time and memory scalability of methods that use the overlap graph model not only because they operate at the level of pairwise read overlaps, but also because many tools assume that such overlaps can be stored and retrieved from local memory. Consequently, such methods take between – h and several gigabytes of memory even for assembling small bacterial genomes. From a parallelism perspective, a high coverage in sampling could also potentially mean fewer sequencing gaps, as the probability of a genomic base being captured by a read improves with coverage by the Lander–Waterman model. Therefore, divide-and-conquer–based techniques such as the cluster-then-assemble may not be as effective in breaking the initial problem size down. Short read assemblers also have to deal with technology specific errors. The error rates associated with next-generation technologies are typically in the –% range, and cannot be ignored as differentiating them from real differences could be key to capturing natural variations such as near identical paralogs. In the current suite of tools specifically built for short read assembly, only a handful of tools support some degree of parallelism. These tools include PEAssembler, ABySS, and YAGA. Others are serial tools that work on desktop computers and rely on highmemory nodes for larger inputs. Even the tools that support parallelism do so to varying degrees. PEAssembler, which implements a variant of the overlap graph model, limits parallelism to node level and assumes that the number of parallel threads is small (< ). It deploys a “seed-extend” strategy in which a subset of “reliable” reads are selected as seeds and other reads that overlap with each seed are incrementally added to extend and build into a consensus sequence in the ′ direction. Parallelism is supported by selecting multiple seeds and launching multiple threads that assume responsibility of extending these different reads in parallel. While the algorithm has the advantage of not having to build and manage large graphs, it performs a conservative extension primarily relying on pair-end data to collapse reads into contigs. From a parallelism perspective, the algorithm relies on shared memory access for managing the read set, which works well if the number of threads is very small. Furthermore, not
G
all steps in the method are parallel and some steps are disk-bound. Experimental results show that the parallel efficiency drops drastically beyond three threads. ABySS and YAGA are two parallel methods that implement the De Bruijn graph model and work for distributed memory computers. The De Bruijn and string graph formulations are conceptually better suited for short read assembly. These formulations allow the algorithm to operate at a read or read’s subunit (i.e., k-mer) level rather than the pairwise overlap level. Repetitive regions manifest themselves in the form of special graph patterns, and error correction mechanism could detect and reconcile anomalous graph substructures prior to performing assembly tours, thereby reducing the chance of misassemblies. The assembly itself manifests in the form of graph traversals. Constructing these graphs and traversing them to produce assembly tours (although not guaranteed to be optimal due to intractability of the problem) are problems with efficient heuristic solutions on a serial computer. The methods ABySS and YAGA provide two different approaches to implement the different stages in parallel using De Bruijn graphs. While these methods differ in their underlying algorithmic details and in the degree offered for parallelism, the structural layout of their algorithms is similar consisting of these four major steps: () parallel graph construction; () error correction; () incorporation of distance constraints due to pair-end reads; and () assembly tour and output. In what follows, approaches to parallelize each of these major steps are outlined, with the bulk of exposition closely mirroring the algorithm in YAGA because its algorithm more rigorously addresses parallelization for all the steps, and takes advantage of techniques that are more standard and well understood among distributed memory processing codes (e.g., sorting, list ranking). By contrast, parallelization is described only for the initial phase of graph construction in ABySS. Therefore, where appropriate, variations in the ABySS algorithmic approach will be highlighted in the text. Other than differences in their parallelization strategies, the two methods also differ particularly in the type of De Bruijn graph they construct (directed vs. bidirected) and their ability to handle multiple read lengths. Such details are omitted in an attempt to keep the text focused on the parallel aspects of the problem. An interested reader can refer to the individual papers for details
G
G
Genome Assembly
pertaining to the nuances of the assembly procedure and the output quality of these assemblies.
Parallel De Bruijn Graph Construction and Compaction Given m reads in set R, the goal is to construct in parallel a distributed representation of the corresponding De Bruijn graph built out of k-mers, for a userspecified value of k. Note that, once a De Bruijn graph representation is generated, it can be transformed into a corresponding string graph representation by compressing paths whose label when concatenated spells out the characters in a read, and by accordingly introducing branch nodes that capture overlap continuation between adjoining reads. This can be done in a successive stage of graph compaction, a minor variant of which is described in the later part of this section. The computing model assumed is p processors (or equivalently, processes), each with access to a local RAM, and connected through a network interconnect and with access to a shared file system where the input is made available. To construct the De Bruijn graph, the reads in R are initially partitioned and loaded in a distributed manner such that each processor receives O( np ) input characters. Let Ri refer to the subset of reads in processor pi . Through a linear scan of the reads in Ri , each pi enumerates the set of k-mers present in Ri . However, instead of storing each such k-mer as a vertex of the De Bruijn graph, the processor equivalently generates and stores the corresponding edges connecting those vertices in the graph. In other words, there is a bijection between the set of edges and the set of distinct k + -mers in Ri . In this edge-centric representation, the vertex information connecting each edge is implicit and the count of reads containing a given k + -mer is stored internally at that edge. Note that after this generation process, the same edge could be potentially generated at multiple processor locations. To detect and merge such duplicates, the algorithm simply performs a parallel sort of the edges using the k + -mers as the key. Therefore, in one sorting step, a distributed representation of the De Bruijn graph is constructed. Standard parallel sort routines such as sample sort can be used here to ensure even redistribution of the edges across processors due to sorting.
An alternative to this edge-centric representation is a vertex-centric representation, which is followed in ABySS and in an earlier version of YAGA. Here, the k-mer set corresponding to each Ri is generated by the corresponding pi . The vertices adjacent to a given vertex could be generated remotely by one or more processors. Therefore, this approach necessitates processors to communicate with one another in order to check the validity of all edges that could be theoretically drawn from its local set of vertices. While the number of such edge validation queries is bounded by per vertex ({a, c, g, t} on either strand orientation), the method runs the risk of generating false edges, e.g., if a k − -mer, say α, occurs in exactly two places along the genome as aαc and gαt, then the vertex-centric approach will erroneously generate edges for aαt and gαc, which are k + -mers nonexistent in the input. Besides this risk, care must be taken to guarantee an even redistribution of vertices among processors by the end of the construction process. For instance, a static allocation scheme in which a hash function is used to map each k-mer to a destination processor, as it is done in ABySS, runs the risk of producing unbalanced distribution as the k-mer concentration within reads is input dependent. Graph compaction: Once a distributed edge-centric representation of the De Bruijn graph is generated, the next step is to simplify it by identifying maximal segments of simple paths (or “chains”) and compact each of them into a single longer edge. Edges that belong to a given chain could be distributed and therefore this problem of removing chains becomes a two-step process: First, to detect the edges (or equivalently, the vertices) which are part of chains and then perform compaction on the individual chains. To mark the vertices that are part of some chain, assume without loss of generality that each edge is stored twice as < u, v > and < v, u >. Sorting the edges in parallel by their first vertex it would bring all edges incident on a vertex together on some processor. Only those vertices that have a degree of two can be part of chains and such vertices can be easily marked with a special label. In the next step, the problem of compacting the vertices along each chain can be treated as a variant of segmented parallel list ranking. A distributed version of the list ranking algorithm can be used to calculate the relative distances of each marked vertex from the start and end of its respective chain. The same procedure can also be used to determine the
Genome Assembly
vertex identifiers of those two boundary vertices for each marked vertex. Compaction then follows through the operations of concatenating the edge labels along the chain at one of the terminal edges, removing all internal edges, and aggregating an average k + -mer frequency of the constituent edges along the chain. All of these operators are binary associative to allow being implemented using calls to a segmented parallel prefix routine. The output of this step is a distributed representation of the compacted graph.
Error Correction and Variation Detection In the De Bruijn graph representation, errors due to sequencing manifest themselves as different subgraph motifs which could be detected and pruned. For ease of exposition, let us informally call an edge in the compacted De Bruijn graph as being “strongly supported” (alternatively, “weakly supported”) if its average k + -mer frequency is relatively high (alternatively, low). Examples of motifs are follows: () Tips are weakly supported dead ends in the graph created due to a base miscall occurring in a read at one of its end positions. Because of compaction such tips occur as single edges branching out of a strongly supported path, and can be easily removed; () Bubbles are detours that provide alternative paths between two terminal vertices. Due to compaction, these detours will also be single edges. There are two types of bubbles – weakly supported bubbles are manifestations of single base miscall occurring internal to a read and need to be removed; whereas, bubbles that originate from the same vertex and are supported roughly to the same degree could be the result of natural variations such as near identical paralogs (i.e., copies of the same gene occurring at different genomic loci). Such bubbles need to be retained. () Spurious links connect two otherwise disparate paths in the graph. Weakly supported links are manifestations of erroneous k + -mers that happen to match the k + -mer present at a valid genomic locus, and they can be severed by examining the supports of the other two paths. The first pass of error correction on the compacted graph could reveal new instances of motifs that can be pruned through iterative passes subsequently until no new instances are observed.
G
Incorporation of Distance Constraints Using Pair-End Information The goal of this step is to map the information provided by read pairs that are linked by the pair-end library onto the compacted and error corrected De Bruijn graph. Once mapped, this information can be used to guide the assembly tour of the graph consistent (to the extent possible) with the distance constraints imposed by the pair-end information. To appreciate the value added by this step to the assembly procedure, recall that the pairend information consists of a list of read pairs of the form < ri , rj > that have originated from the same clonal insert during sequencing. In genomic distance parlance, this implies that the number of bases separating the two reads is bounded by a minimum and maximum. Also recall that an assembly from a De Bruijn graph corresponds to a tour of the graph (possibly multiple tours if there were gaps in the original sequencing). Now consider a scenario where a path in the De Bruijn graph branches into multiple separate subpaths. A correct assembly tour would have to decide which of those branches reflect the sequence of characters along the unknown genome. It should be easy to see how pair-end information can be used to resolve such situations. To incorporate the information provided in the form of read pairs by such pair-end libraries, the YAGA algorithm uses a cluster summarization procedure, which can be outlined as follows: First, the list of read pairs provided as input by the pair-end information is delineated into a corresponding list of constituent k + -mer pairs. Note that two k + -mers from two reads of a pair can map to two different edges on the distributed graph. Furthermore, different positions along the same edge could be paired with positions emanating from different edges. To this effect, the algorithm attempts to compute a grouping of edge pairs based on their best alignment with the imposed distance constraints. To achieve this, an observed distance interval is computed between every edge pair on the graph linked by a read pair, and then overlapping intervals that capture roughly the same distances along the same orientation are incrementally clustered using a greedy heuristic. This last step is achieved using a two-phase clustering step that primarily relies on several rounds of parallel sorting tuples containing edge pair and interval distance information. The formal details pertaining to this algorithmic step has been omitted here for brevity.
G
G
Genome Assembly
Completing the Assembly Tour An important offshoot of the above summarization procedure is that redundant distance information captured by edge pairs in the same cluster can be removed, thereby allowing significant compression in the level of information needed to store relative to the original graph. This compression, in most practical cases, would allow for the overall tour to be performed sequentially on a single node. The serial assembly touring procedure begins by using edges that have significantly longer edge labels than the original read length as “seeds,” and then by extending them in both directions as guided by the pair-end summarization traversal constraints. The YAGA assembler has demonstrated scaling on IBM BlueGene/L processors and performs assembly of over a billion synthetically generated reads in under h. The run-time is dominated by the pair-end cluster summarization phase. As of this writing, performancerelated information is not available for ABySS.
Future Trends Genome sequencing and assembly is an actively pursued, constantly evolving branch of bioinformatics. Technological advancements in high-throughput sequencing have fueled algorithmic innovation and along with a compelling need to harness new computing paradigms. With the adoption of ever faster, cheaper, and massively parallel sequencing machines, this application domain is becoming increasingly dataand compute-intensive. Consequently, parallel processing is destined to play a critical role in genomic discovery. Numerous large-scale sequencing projects including the assembly of the human genome to the most recent assembly of the maize genome have benefited from the use of parallel processing, although in different ad hoc ways. Heterogeneous clusters comprising of a mixture of a few high-end shared memory machines along with numerous compute nodes have been used to “farm” out tasks and accelerate the overlap computation phase in particular. This is justified because nearly all of these large-scale projects used the more traditional Sanger sequencing. A few specialpurpose projects such as the maize gene-enriched sequencing used more strongly coupled parallel codes such as PaCE. However, with an aggressive adoption of next-generation sequencing for genome sequencing
and more complex genomes in the pipeline for sequencing (e.g., wheat, pine, metagenomic communities), this scenario is about to change, and more strongly coupled parallel codes are expected to become part of the mainstream computing in genome assembly. In addition to de novo sequencing, next-generation sequencing is also increasingly being used in genome resequencing projects, where the goal is to assemble a genome of a particular variant strain (or subspecies) of an already sequenced genome. The type of computation that originates is significantly different from that of de novo assembly. For resequencing, the reads generated from a new strain are compared against a fully assembled sequence of a reference strain. This process, sometimes called read mapping, only requires comparison of reads against a much larger reference. Approaches that capitalize on advanced string data structures such as suffix trees are likely to play an active role in these algorithms. Also, as this branch of science becomes more data-intensive, reaching the petascale range, different parallel paradigms such as MapReduce need to be explored, in addition to distributed memory and shared memory models. The developments in genome sequencing can be carried over to other related applications that also involve large-scale sequence analysis, e.g., in transcriptomics, metagenomics, and proteomics. Fine-grain parallelism in the area of string matching and sequence alignment has been an active pursued topic over the last decade and there are numerous hardware accelerators for performing sequence alignment on various platforms including General Purpose Graphical Processing Units, Cell Broadband Engine, Field-Programmable Gate Arrays, and Multicores. These advances are yet to take their place in mainstream sequencing projects. Genome sequencing is at the cusp of revolutionary possibilities. With a rapidly advancing technology base, the possibility of realizing landmark goals such as personalized medicine and “$, genome” do not look distant or far-fetched. The well-advertised $ million Archon Genomics X PRIZE will be awarded to the first team that sequences human genomes in days, at a recurring cost of no more than $, per genome. In , the cost for sequencing a human genome plummeted below $, using technologies from Illumina and SOLiD. Going past these nextgeneration technologies, however, companies such as
Genome Assembly
Pacific Biosciences are now releasing a third-generation (“gen-”) sequencer that uses an impressive approach called single-molecule sequencing (or “SMS”) [], and have proclaimed grand goals such as sequencing a human genome in min for less than $ by . These are exciting times for genomics, and the field is likely to continue serving as a rich reservoir for new problems that pose interesting compute- and data-intensive challenges which can be addressed only through a comprehensive embrace of parallel computing.
Related Entries Homology to Sequence Alignment, From Suffix Trees
Bibliographic Notes and Further Reading Even though DNA sequencing technologies have been available since the late s, it was not until the s that they were applied at a genome scale. The first genome to be fully sequenced and assembled was the ∼. Mbp bacterial genome of H. influenzae in []. The sequencing of the more complex, ∼ billion bp long human genome followed []. Several other notable large-scale sequencing initiatives followed in the new millennium including that of chimpanzee, rice, and maize (to cite a few examples). All of these used either the WGS strategy or hierarchical strategy coupled with Sanger sequencing, and their assemblies were performed using programs that followed the overlap–layout–consensus model. The National Center for Biotechnology Information (NCBI) (http://www. ncbi.nlm.nih.gov) maintains a comprehensive database of all sequenced genomes. More than a dozen programs exist to perform genome assembly under the overlap– layout–consensus model. Notable examples include PCAP [], Phrap (http://www.phrap.org/), Arachne [], Celera [], and TIGR assembler []. For a detailed review of fragment assembly algorithms, refer to [, ]. The parallel algorithm that uses the clusterthen-assemble approach along with the suffix tree data structure for assembly was implemented in a program called PaCE and was first described in [] in the context of clustering Expressed Sequence Tag data and then later adapted for genome assembly []. Experiments
G
conducted as part of the maize genome sequencing consortium demonstrated scaling of this method to over a million reads generated from gene-enriched fractions of the maize genome on a , node BlueGene/L supercomputer []. The parallel suffix tree construction algorithm described in this entry was first described in [] and a variant of this method was later presented in []. An optimal algorithm to detect maximal matching pairs of reads in parallel using the suffix tree data structure is presented in []. The NP-completeness of the Shortest Superstring Problem (SSP) was shown by Gallant et al. []. The De Bruijn graph formulation for genome assembly was first introduced by Idury and Waterman [] in the context of a sequencing technique called sequencingby-hybridization, and later extended to WGS based approaches in the EULER program by Pevzner et al. []. The string graph formulation was developed by Myers []. The proof of NP-Hardness for the overlap– layout–consensus is due to Kececioglu and Myers []. The proofs of NP-Hardness for the De Bruijn and string graphs models of genome assembly are due to Medvedev et al. []. Since the later part of s, various next-generation sequencing technologies such as Roche (http://www.genome-sequencing.com/), SOLiD (http:// www.appliedbiosystems.com/), Illumina (http://www. illumina.com/), and HeliScope (http://www.helicosbio. com/) have emerged along side serial assemblers. A “third” generation of machines that promise a brand new way of sequencing (by single-molecule sequencing) are also on their way (e.g., Pacific Biosciences (http://www.pacificbiosciences.com/)). Consequently, the development of short read assemblers continue to be in hot pursuit. Edena, Newbler (http:// www..com), PE-Assembler [], QSRA [], SHARCGS [], SSAKE [], and VCAKE [] are all examples of programs that operate using the overlap graph model. EULER-SR [], ALLPATHS [], Velvet [], SOAPdenovo [], ABySS [], and YAGA [] are programs that use the De Bruijn graph formulation. Of these tools, PE-Assembler, ABySS, and YAGA are parallel implementations, although to varying degrees as described in the main text.
Acknowledgment Study supported by NSF grant IIS-.
G
G
Genome Sequencing
Bibliography . Ariyaratne P, Sung W () PE-assembler: de novo assembler using short paired-end reads. Bioinformatics ():– . Batzoglou S, Jaffe DB, Stanley K, Butler J et al () ARACHNE: a whole-genome shotgun assembler. Genome Res ():– . Bryant D, Wong W, Mockler T () QSRA – a quality-value guided de novo short read assembler. BMC Bioinform (): . Butler J, MacCallum L, Kleber M, Shlyakhter IA et al () ALLPATHS: de novo assembly of whole-genome shotgun microreads. Genome Res :– . Chaisson MJ, Pevzner PA () Short read fragment assembly of bacterial genomes. Genome Res :– . Dohm J, Lottaz C, Borodina T, Himmelbaurer H () SHARCGS, a fast and highly accurate short-read assembly algorithm for de novo genomic sequencing. Genome Res ():– . Emrich S, Kalyanaraman A, Aluru S () Chapter : algorithms for large-scale clustering and assembly of biological sequence data. In: Handbook of computational molecular biology. CRC Press, Boca Raton . Fleischmann R, Adams M, White O, Clayton R et al () Wholegenome random sequencing and assembly of Haemophilus influenzae rd. Science ():– . Flusberg BA, Webster DR, Lee JH, Travers KJ et al () Direct detection of DNA methylation during single-molecule, real-time sequencing. Nat Methods :– . Gallant J, Maier D, Storer J () On finding minimal length superstrings. J Comput Syst Sci :– . Ghoting A, Makarychev K () Indexing genomic sequences on the IBM blue gene. In: Proceedings ACM/IEEE conference on supercomputing. Portland . Huang X, Wang J, Aluru S, Yang S, Hiller L () PCAP: a wholegenome assembly program. Genome Res :– . Idury RM, Waterman MS () A new algorithm for DNA sequence assembly. J Comput Biol ():– . Jackson BG, Regennitter M, Yang X, Schnable PS, Aluru S () Parallel de novo assembly of large genomes from highthroughput short reads. In: IEEE international symposium on parallel distributed processing, pp – . Jeck W, Reinhardt J, Baltrus D, Hickenbotham M et al () Extending assembly of short DNA sequences to handle error. Bioinformatics :– . Kalyanaraman A, Aluru S, Brendel V, Kothari S () Space and time efficient parallel algorithms and software for EST clustering. IEEE Trans Parallel Distrib Syst ():– . Kalyanaraman A, Emrich SJ, Schnable PS, Aluru S () Assembling genomes on large-scale parallel computers. J Parallel Distrib Comput ():– . Kececioglu J, Myers E () Combinatorial algorithms for DNA sequence assembly. Algorithmica (–):– . Li R, Zhu H, Ruan J, Qian W et al () De novo assembly of human genomes with massively parallel short read sequencing. Genome Res ():– . Medvedev P, Georgiou K, Myers G, Brudno M () Computability of models for sequence assembly. Lecture notes in computer science, vol . Springer, Heidelberg, pp –
. Myers EW () The fragment assembly string graph. Bioinformatics, (Suppl ):ii–ii . Myers EW, Sutton GG, Delcher AL, Dew IM et al () A WholeGenome assembly of drosophila. Science ():– . Pevzner PA, Tang H, Waterman M () An eulerian path approach to DNA fragment assembly. In: Proceedings of the national academy of sciences of the United States of America, vol , pp – . Pop M () Genome assembly reborn: recent computational challenges. Briefings in Bioinformatics ():– . Simpson J, Wong K, Jackman S, Schein J et al () ABySS: a parallel assembler for short read sequence data. Genome Res :– . Sutton GG, White O, Adams MD, Kerlavage AR () TIGR assembler: a new tool for assembling large shotgun sequencing projects. Genome Sci Technol ():– . Venter C, Adams MD, Myers EW, Li P et al () The sequence of the human genome. Science ():– . Warren P, Sutton G, Holt R () Assembling millions of short DNA sequences using SSAKE. Bioinformatics :– . Zerbino DR, Velvet BE () Algorithms for de novo short read assembly using de bruijn graphs. Genome Res :–
Genome Sequencing Genome Assembly
GIO PCI Express
Glasgow Parallel Haskell (GpH) Kevin Hammond University of St. Andrews, St. Andrews, UK
Synonyms GpH (Glasgow Parallel Haskell)
Definition Glasgow Parallel Haskell (GpH) is a simple parallel dialect of the purely functional programming language, Haskell. It uses a semi-explicit model of parallelism, where possible parallel threads are marked by the programmer, and a sophisticated runtime system then
Glasgow Parallel Haskell (GpH)
decides on the timing of thread creation, allocation to processing elements, migration. There have been several implementations of GpH, covering platforms ranging from single multicore machines through to computational grids or clouds. The best known of these is the GUM implementation that targets both sharedmemory and distributed-memory systems using a sophisticated virtual shared-memory abstraction built over a common message-passing layer, but there is a new implementation, GHC-SMP, which directly exploits shared-memory systems, and which targets multicore architectures.
Discussion Purely Functional Languages and Parallelism Because of the absence of side effects in purely functional languages, it is relatively straightforward to identify computations that can be run in parallel: any sub-expression can be evaluated by a dedicated parallel task. For example in the following very simple function definition, each of the two arguments to the addition operation can be evaluated in parallel. f x = fibonacci x + f act o ri al x
This property was already realized by the mid-s, when there was a surge of interest both in parallel evaluation in general, and in the novel architectural designs that it was believed could overcome the sequential “von-Neumann bottleneck.” In fact, the main issue in a purely functional language is not extracting enough parallelism – it is not unusual for even a short-running program to produce many tens of thousands of parallel threads – but is rather one of identifying sufficiently large-grained parallel tasks. If this is not done, then thread creation and communication overheads quickly eliminate any benefit that can be obtained from parallel execution. This is especially important on conventional processor architectures, which historically provided little, if anything, in the way of hardware support for parallel execution. The response of some parallel functional language designers has therefore been to provide very explicit parallelism mechanisms. While this usually avoids the problem of excessive parallelism, it places a significant burden on the programmer, who must understand the details of parallel execution as well as
G
the application domain, and it can lead to code that is specialized to a specific parallel architecture, or class of architectures. It also violates a key design principle for most functional languages, which is to provide as much isolation as possible from the underlying implementation. GpH is therefore designed to allow programmers to provide information about parallel execution while delegating issues of placement, communication, etc. to the runtime system.
History and Development of GpH Glasgow Parallel Haskell (GpH) was first defined in . GpH was designed to be a simple parallel extension to the then-new nonstrict, purely functional language Haskell [], adding only two constructs to sequential Haskell: par and seq. Unlike most earlier lazy functional languages, Haskell was always intended to be parallelizable. The use of the term “non-strict” rather than lazy reflects this: while lazy evaluation is inherently sequential, since it fixes a specific evaluation order for sub-expressions, under Haskell’s non-strict evaluation model, any number of sub-expressions can be evaluated in parallel provided that they are needed by the result of the program. The original GpH implementation targeted the GRIP novel parallel architecture, using direct calls to low-level GRIP communications primitives. GRIP was a shared-memory machine, using custom microcoded “intelligent memory units” to share global program data between off-the-shelf processing elements (Motorola processors, each with a private MB memory), developed in an Alvey research project which ran from to . Initially, GpH built on the prototype Haskell compiler developed by Hammond and Peyton Jones in . It was subsequently ported to the Glasgow Haskell Compiler, GHC [], that was developed at Glasgow University from onward, and which is now the de-facto standard compiler for Haskell (since the focus of the maintenance effort was moved to Microsoft Research in , GHC has also become known as the Glorious Haskell Compiler). In , the communications library was redesigned to give what became the highly portable GUM implementation []. This allowed a single parallel implementation to target both commercial shared-memory systems and the then-emerging class of cost-effective loosely coupled networks of workstations. Initially, GUM targeted
G
G
Glasgow Parallel Haskell (GpH)
system-specific communication libraries, but it was subsequently ported to PVM. There are now UDP, PVM, MPI, and MPICH-G instances of GUM, as well as system-specific implementations, and the same GUM implementation runs on multicore machines, shared-memory systems, workstation clusters, computational grids, and is being ported to large-scale highperformance systems, such as the ,-core HECToR system at the Edinburgh Parallel Computer Centre. Key parts of the GUM implementation have also been used to implement the Eden [] parallel dialect of Haskell, and GpH is being incorporated into the latest mainstream version of GHC. Although the two parallelism primitives that GpH uses are very simple, it became clear that they could be packaged using higher-order functions to give much higher-level parallel abstractions, such as parallel pipelines, data-parallelism, etc. This led ultimately to the development of evaluation strategies []: high-level parallel structures that are built from the basic par and seq primitives using standard higher-order functions. Because they are built from simple components and standard language technology, evaluation strategies are highly flexible: they can be easily composed or nested; the parallelism structure can change dynamically; and the applications programmer can define new strategies on an as-need basis, while still using standard strategies.
The GpH Model of Parallelism GpH is unusual in using only two parallelism primitives: par and seq. The design of the par primitive dates back to the late s [], where it was used in a parallel implementation of the Lazy ML (LML) compiler. The primitive, higher-order, function, par is used by the programmer to mark a sub-expression as being suitable for parallel evaluation. For example, a variant of the function f above can be parallelized using two instances of par. f x = let r = fibonacci x ; r = f a c t o r i a l x in l e t r e s u l t = ( r , r ) in r ‘ par ’ ( r ‘ par ’ r e s u l t )
r1 and r2 can now both be evaluated in parallel with the construction of the result pair (r1, r2). There is no need to specify any explicit communication, since
the results of each computation are shared through the variables r1 and r2. The runtime system also decides on issues such as when a thread is created, where it is placed, how much data is communicated, etc. Very importantly, it can also decide whether a thread is created. The parallelism model is thus semi-explicit: the programmer marks possible sites of parallelism, but the runtime system takes responsibility for the underlying parallel control based on information about system load, etc. This approach therefore eliminates significant difficulties that are commonly experienced with more explicit parallel approaches, such as raw MPI. The programmer needs to make sure there is enough scope for parallelism, but does not need to worry about issues of deadlock, communication, throttling, load-balancing, etc. In the example above, it is likely that only one of r1 or r2 (or neither) will actually be evaluated in parallel, since the current thread will probably need both their values. Which of r1 or r2 is evaluated first by the original thread will, however, depend on the context in which it is called. It is therefore left unspecified to avoid unnecessary sequentialization.
Lazy Thread Creation Several different versions of the par function have been described in the literature. The version used in GpH is asymmetric in that it marks its first argument as being suitable for possible execution (the expression is sparked), while continuing sequential execution of its second argument, which forms the result of the expression. For example, in par s e, the expression s will be sparked for possible parallel evaluation, and the value of e will be returned as the result of the current thread. Since the result expression is always evaluated by the sparking thread, and since there can be no side effects, it follows that it is completely safe to ignore any spark. That is, unlike many parallel systems, the creation of threads from sparks is entirely optional. This fact can be used to throttle the creation of threads from sparks in order to avoid swamping the parallel machine. This is a lazy thread creation approach (the term “lazy task creation” was coined later by Mohr et al. [] to describe a similar mechanism in MultiLisp). A corollorary is that, since sparks do not carry any execution state, they can be very lightweight. Generally, a single pointer is adequate to record a sparked expression in most implementations of GpH.
Glasgow Parallel Haskell (GpH)
Sparks may be chosen for conversion to threads using a number of different strategies. One common approach is to use the oldest spark first. If the application is a divide-and-conquer program, this means that the spark representing the largest amount of work will be chosen. Alternatively, an approach may be used where the youngest (smallest) spark is executed locally and the oldest (largest) spark is offloaded for remote execution. This will improve locality, but may increase thread creation costs, since many of the locally created threads might otherwise be subsumed into their parent thread.
Parallel Graph Reduction A second key issue that must be dealt with is that of thread synchronization. Efficient sequential non-strict functional language implementations generally use an evaluation technique called graph reduction, where a program builds a graph data structure representing the work that is needed to give the result of a program and gradually rewrites this using the functional rules defined in the program until the graph is sufficiently complete to yield the required result. This rewriting process is known as reducing the graph to some normal form (in fact weak head normal form). Each node in the graph represents an expression in the original program. Initially, this will be a single unevaluated expression corresponding to the result of the program (a “thunk”). As execution proceeds, thunks are evaluated, and each graph node is overwritten with its result. In this way, results are automatically shared between several consumers. Only graph nodes that actually contribute to the final result need to be rewritten. This means that unnecessary work can be avoided, a process that, in the sequential world, allows lazy evaluation. The same mechanism naturally lends itself to parallel evaluation. Each shared node in the graph represents a possible synchronization point between two parallel threads. The first thread to evaluate a graph node will lock it. Any thread that evaluates the node in parallel with the thread that is evaluating it will then block when it attempts to read the value of the node. When the evaluating thread produces the result, the graph node will be updated and any blocked threads will be awoken. In this way, threads will automatically synchronize through the graph representing the computation, not
G
just at the root of each thread, but whenever they share any sub-expressions. Figure shows a simple example of a divide-andconquer parallel program, where the root of the computation, f 9, is rewritten using three threads: one for the main computation and two sub-threads, one to evaluate f 8 and one to evaluate f 7. These thunks are linked into the addition nodes in the main computation. Having evaluated the sparked thunks, the second and third threads update their root nodes with the corresponding result. Once the main thread has evaluated the remaining thunk (another call to f 8, which we have assumed is not shared with Thread #), it will incorporate these results into its own result. The final stage of the computation is to rewrite the root of the graph (the result of the program) with the value 89.
Evaluate-and-Die As a thread evaluates its program graph, it may encounter a node that has been sparked. There are three possible cases, depending on the evaluation status of the sparked node. If the thunk has already been evaluated, then its value can be used as normal. If the thunk has not yet been evaluated, then the thread will evaluate it as normal, first annotating it to indicate that it is under evaluation. Once a result is produced, the node will be overwritten with the result value, and any blocked threads will be reawakened. Finally, if the node is actually under evaluation, then the thread must be blocked until the result is produced and the node updated with this value. This is the evaluate-and-die execution model []. The advantage of this approach is that it automatically absorbs sparks into already executing threads, so increasing their granularity and avoiding thread creation overheads. If a spark refers to a thunk that has already been evaluated then it may be discarded without a thread ever being created. The seq Primitive While par is entirely adequate for identifying many forms of parallelism, Roe and Peyton Jones discovered [] that by adding a sequential combining function, called seq, much tighter control could be obtained over parallel execution. This could be used both to reduce the need for detailed knowledge of the underlying parallel implementation and to encode more sophisticated patterns of parallel execution. For example, one
G
G
Glasgow Parallel Haskell (GpH)
f
+
10 Thread #1
f
Thread #2
9
f
8
+ Thread #2
Thread #1
+
f
8
+
f
7
Thread #3
+
+ Thread #2
Thread #1
+
f
8
+
+
f
34
7
Thread #3
55
34
21 Thread #3
+ Thread #1
Thread #2
Thread #1
89 Thread #2
34
Glasgow Parallel Haskell (GpH). Fig. Parallel graph reduction
of the par constructs in the example above can be replaced by a seq construct, as shown below. f x = let r = fibonacci x ; r = f a c t o r i a l x in l e t r e s u l t = ( r , r ) in r ‘ par ’ ( r ‘ seq ’ r e s u l t ) −− was r
‘ par ’ r ‘ par ’ r e s u l t
The seq construct acts like ; in a conventional language such as C: it first evaluates its first argument (here r2), and then returns its second argument (here result). So in the example above, rather than creating two sparks as before, now only one is created. Previously, depending on the order in which threads were scheduled, either the parent thread would have blocked when it evaluated r1 or r2, because these
were already under evaluation, or one or both of the sparked threads would have blocked, because they were being evaluated (or had been evaluated) by the parent thread. Now, however, the only possible synchronization is between the thread evaluating r1 and the parent thread. Note that it is not possible to use either of the simpler forms of par r1 (r1, r2) or par r2 (r1, r2) to achieve the same effect, as might be expected. Because the implementation is free to evaluate the pair (r1, r2) in whichever order it prefers, there is a % probability that the spark will block on the parent thread. This is not the only use of seq. For example, evaluation strategies (discussed below) also make heavy use of this primitive to give precise control over evaluation order. While not absolutely essential for parallel evaluation, it is thus very useful to provide a finer
Glasgow Parallel Haskell (GpH)
degree of control than can be achieved using seq alone. However, there are two major costs to the use of seq. The first is that the strictness properties of the program may be changed – this means that some thunks may be evaluated that were previously not needed, and that the termination properties of the program may therefore be changed. The second is that parallel programs may become speculative.
Speculative Evaluation As described above, in GpH it is safe to spark any subexpession that is needed by the result of the parallel computation. However, nothing prevents the programmer from sparking a sub-expression that is not known to be needed. This therefore allows speculative evaluation. Although there was some early experimentation with mechanisms to control speculation, these were found to be very difficult to implement, and current implementations of GpH do not provide facilities to kill created threads, change priorities dynamically, etc., as is necessary to support safe speculation. For example, s p e c x y = x ‘ par ’ y
defines a new function spec that sparks x and continues execution of y. If x is not definitely needed by y, or is a completely independent expression, this will spark x speculatively. If a thread is created to evaluate x, it will terminate either when it has completed evaluation of x, or when the main program terminates, having produced its own result, or if it evaluates an undefined value. In the latter case, it may cause the entire program to fail. Moreover, it is theoretically possible to prevent any progress on the main computation by flooding the runtime system with useless speculative threads. For these reasons, any speculative evaluation has to be treated carefully: if the program is to terminate, speculative sub-expressions should terminate in finite time without an error, and there should not be too many speculative sparks.
Evaluation Strategies As discussed above, it is possible to construct higherlevel parallel abstractions from basic seq/par constructs using standard Haskell programming constructs. These parallel abstractions can be associated with computation functions using higher-order definitions. This is the evaluation strategy approach. The key idea of evaluation
G
strategies to separate what is to be evaluated from how it could be evaluated in parallel []. For example, a simple sequential ray tracing function could be defined as follows: r a y t r a c e r : : I n t −> I n t −> S c e n e −> [ L i g h t ] −> [ [ V e c t o r ] ] r a y t r a c e r xlim ylim scene l i g h t s = map t r a c e l i n e [ . . ylim − ] where t r a c e l i n e y = [ t r a c e p i x e l s c e n e l i g h t s x y | x <− [ . . xlim − ] ]
Given a visible image with maximum x and y dimensions of xlim and ylim, the raytracer function applies the traceline function to each line in the image (by mapping it across each value of y in the range 0..ylim-1), and hence applies the tracepixel function to each pixel using the specified scene and lighting model. The raytracer function may be parallelized, for example, by adding the parMap strategy to parallelize each line as follows: r a y t r a c e r : : I n t −> I n t −> S c e n e −> [ L i g h t ] −> [ [ V e c t o r ] ] r a y t r a c e r x l i m y l i m s c e n e l i g h t s = map t r a c e l i n e [ . . ylim − ] ‘ u s i n g ’ parMap r n f
Here, parMap is a strategy that specifies that each element of its value argument should be evaluated in parallel. It is parameterized on another strategy that specifies what to do with each element. Here, rnf indicates that each element should be evaluated as far as possible (rnf stands for “reduce to normal form”). The using function simply applies its second argument (an evaluation strategy) to its first argument (a functional value), and returns the value. It can be easily defined using higher-order functions and the seq primitive as follows: u s i n g : : a −> S t r a t e g y a −> a x ‘ u s i n g ’ s = s x ‘ seq ’ x
So, when a strategy is applied to an expression by the using operation, the strategy is first applied to the value of the expression, and once this has completed, the value is returned. This allows the use of both parallel and sequential strategies. In fact, any list strategy may be used to parallelize the raytracer function instead of parMap. For example, a task farm could be used, or the list could be divided into equally sized chunks as shown below.
G
G
Glasgow Parallel Haskell (GpH)
r a y t r a c e r : : I n t −> I n t −> S c e n e −> [ L i g h t ] −> [ [ V e c t o r ] ] r a y t r a c e r xlim ylim scene l i g h t s = . . . ‘ using ’ parListChunk chunkSize r nf
Evaluation strategies have proved to be very powerful and flexible, having been successfully applied to several large programs, including symbolic programs with irregular patterns of parallelism []. Typically, only a few lines need to be changed at key points by adding appropriate using clauses. In abstracting parallel patterns from sequential computations, evaluation strategies have some similarities with algorithmic skeletons. The key differences between evaluation strategies and typical skeleton approaches is that evaluation strategies do not mandate a specific parallel implementation (since they rely on semi-explicit parallelism, they provide hints to the runtime system rather than directives); and that they are completely user-programmable – everything above the par and seq primitives is programmed using standard Haskell code. Unlike most skeleton approaches, they may also be easily composed to give irregular nested parallel structures or phased parallel computations, for example.
shared with other PEs. These nodes are given global addresses, which identify the owner of the graph node. The advantage of this model is that it allows completely independent memory management: by keeping tables of global in-pointers, which are used as garbage collection roots into the local heap, each PE can garbage-collect its own local heap independently of all other PEs. This local garbage collection is conservative, as required, since even if a global in-pointer is no longer referenced by any other PE, it will still be treated as a garbage collection root. In order to collect global garbage, a separate scheme is used, based on distributed reference counting. Since the majority of the program graph never needs to be shared, this approach brings major efficiency gains over typical single-level memory management. In addition to the reduced need for synchronization during garbage collection, there is also no need to maintain global locks across purely local graph. Within each PE, GUM uses the same efficient (and sequential) garbage collector as the standard GHC implementation. This is currently a stop-and-copy generational collector based on Appel’s collector.
Visualizing the Behavior of GpH Programs The GUM Implementation of GpH GUM is the original and most general implementation of GpH. It provides a virtual shared-memory abstraction for parallel graph reduction, using a message-passing implementation to target both physically shared-memory and distributed-memory systems. In the GUM model, the graph representing the program that is being evaluated is distributed among the set of processor elements (PEs) that are available to execute the program. These PEs usually correspond to the cores or processors that are available to execute the parallel program, but it is possible to map PEs onto multiple processes on the same core/processor, if required. Each PE manages its own execution environment, with its own local pools of sparks and threads, which it schedules as required. Sparks and threads are offloaded on demand to maintain good work balance. A key feature of GUM is that it uses a two-level memory model: each PE has its own private heap that contains unshared program graph created by local threads. Within this heap, some graph nodes may be
Good visualization is an essential part of understanding parallel behavior. Runtime information can be visualized at a variety of levels to give progressively more detailed information. For example, Fig. visualizes the overall and per-PE parallel activity for a -PE system running the soda application (a simple crosswordpuzzle solver). Apart from the clear phase-transition about % into the execution, very few threads are blocked; and there are very few runnable, but not running threads. Those that are runnable are migrated to balance overall system load. The profile also reveals that relatively little time is spent fetching nonlocal data. The per-PE profile gives a more detailed view. It is obvious that the workload is well balanced and that the workstealing mechanism is effective in distributing work. The example also shows where PEs are idle for repeated periods, and that only the main PE is active at the end of the computation (collating results to be written as the output of the program). It thus clearly identifies places where improvements could be made to the parallel program.
Glasgow Parallel Haskell (GpH)
tasks
GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim
soda_mg +RTS -bP -bp32 -bM -b-G -by1 -bl400
G
Average Parallelism = 14.8
45
40
35
30
25
G
20
15
10
5
0
0
50.0 k
100.0 k
running
GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim GrAnSim
150.0 k
runnable
200.0 k
250.0 k
fetching
300.0 k
blocked
350.0 k
400.0 k
migrating
soda_mg +RTS -bP -bp32 -bM -b-G -by1 -bl400
450.0 k
Runtime = 493.2 k cycles
Mon Oct 21 22:59 1996
31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 0
50.0 k
100.0 k
150.0 k
200.0 k
250.0 k
300.0 k
350.0 k
400.0 k
450.0 k
Glasgow Parallel Haskell (GpH). Fig. Sample overall and per-PE Activity Profiles for a -PE machine
493251
G
Glasgow Parallel Haskell (GpH)
The GHC-SMP Implementation of GpH A recent development is the GHC-SMP [] implementation of GpH from Microsoft Research Labs, Cambridge, UK, which is integrated into the standard GHC distribution. This provides an implementation of GpH that specifically targets shared-memory and multicore systems. It uses a similar model of PEs and threads to the GUM implementation, including spark pools and runnable thread pools, and implements a similar sparkstealing model. The key difference from GUM is that GHC-SMP uses a physically shared heap, rather than a virtual shared heap with an underlying message-passing implementation. This heap is garbage-collected using a global stop-and-copy collector that requires the synchronization of all PEs, but which may itself be executed in parallel using the available processor cores. Also, unlike GUM, GHC-SMP exports threads to remote PEs based on the load of the local core, rather than on demand.
GRID-GUM and the SymGrid System By replacing the low-level communications library with MPICH-G, it is possible to execute GUM (or Eden – see below) not only on standard clusters of workstations, but also on wide-area computational grids, coordinated by the standard Globus grid middleware. The GRID-GUM and grid-enabled Eden implementations form the basis for the SymGrid-Par middleware [], which aims to provide high-level support for computational grids for a variety of symbolic computing systems. The middleware coordinates symbolic computing engines into a coherent parallel system, providing high-level skeletons. The system uses a high-level data-exchange protocol (SCSCP) based on the standard OpenMath XML format for mathematical data. Results have been very promising to date, with superlinear performance being achievable for some mathematical applications, without changing any of the sequential symbolic computing engines.
The GranSim Simulator The GranSim parallel simulator was developed in [] as an efficient and accurate simulator for GpH running on a sequential machine, specifically to expose granularity issues. It is unusual in modifying the sequential GHC runtime system so that evaluating graph nodes also triggers the addition of sparks,
and simulates inter-PE communication. It thus follows the actual GUM implementation very precisely. It allows a range of communication costs to be simulated for a specific parallel application, ranging from zero (ideal communication) to costs that are similar to those Program execution times are also simulated, using a cost model that takes architectural characteristics into account. One of the key uses of GranSim is as the core of a parallel program development methodology, where a GpH program is first simulated under ideal parallel conditions, and then under communication cost settings for specific parallel machines: shared-memory, distributed-memory, etc. This allows the program to be gradually tuned for a specific parallel setting without needing access to the actual parallel machine, and in a way that is repeatable, can be easily debugged, and which provides detailed metrics.
GdH and Mobile Haskell GdH [] is a distributed Haskell dialect that builds on GpH. It adds explicit constructs for task creation on specific processors, with inter-processor communication through explicit mutable shared variables, that can be written on one processor and read on another. Internally, each processor runs a number of threads. The GdH implementation extends the GUM implementation with a number of explicit constructs. The main construct is revalIO, which constructs a new task on a specific procesor. In conjunction with mutable variables, this can be used to construct higher-level constructs. For example, the showSystem definition below prints the names of all available PEs. The getAllHostNames function maps the hostName function over the list of all processor identifiers, returning an IO action that obtains the environment variable HOST on the specified PE. This is used in the showSystem IO operation which first obtains a list of all host names, then outputs them as a sorted list, with duplicates eliminated using the standard nub function. g e t A l l H o s t N a m e s : : IO [ S t r i n g ] g e t A l l H o s t N a m e s = mapM hostName a l l P E I d hostName : : PEId −> IO S t r i n g hostName pe = r e v a l I O ( g e t E n v ‘ ‘ HOST ’ ’ ) pe showSystem = do { h o s t n a m e s <=
Glasgow Parallel Haskell (GpH)
g e t A l l H o s t N a m e s ; showHostNames h o s t n a m e s } where showHostNames names = p u t S t r L n ( show ( s o r t ( nub names ) ) )
At the language level, GdH is similar to Concurrent Haskell, which is designed to execute explicit concurrent threads on a single PE. The key language difference is the inclusion of an explicit PEId to allocate tasks to PEs, supported by a distributed runtime environment. GdH is also broadly similar to the industrial Erlang language, targeting a similar area of distributed programming, but is non-strict rather than strict and, since it is a research language, does not have the range of telecommunications-specific libraries that Erlang supports in the form of the Erlang/OTP development platform. Mobile Haskell [] similarly extends Concurrent Haskell, adding higher-order communication channels (known as Mobile Channels), to support mobile computations across dynamically evolving distributed systems. The system uses a bytecode implementation to ensure portability, and serializes arbitrary Haskell values including (higher-order) functions, IO actions, and even mobile channels.
Eden Eden [] (described in a separate encyclopedia entry) is closely related to GpH. Like GpH, it uses an implicit communication mechanism, sharing values through program graph nodes. However, it uses an explict process construct to create new processes and explicit process application to pass streams of values to each process. Unlike GpH, all data that is passed to an Eden process must be evaluated before it is communicated. There is also no automatic work-stealing mechanism, task migration, granularity agglomeration, or virtual shared graph mechanism. Eden, does, however, provide additional threading support: each output from the process is evaluated using its own parallel thread. It also provides mechanisms to allow explicit communication channels to be passed as first-class objects between processes. Eden has been used to implement both algorithmic skeletons and evaluation strategies, where it has the advantage of providing more controllable parallelism but the disadvantage of losing adaptivity. One
G
particularly useful technique is to use Eden to implement a master–worker evaluation strategy, where a set of dynamically generated worker functions is allocated to a fixed set of worker processors using an explicitly programmed scheduler. This allows Eden to deal with varying thread granularities, without using a lazy thread creation mechanism, as in GpH. While it does not need the advanced adaptivity mechanisms that GUM provides, the Eden implementation shares several basic components, in particular, the communication library and scheduler have very similar implementations.
Current Status GpH has been adopted as a significant part of the Haskell community effort on parallelism. The evaluation strategies library has just been released as part of the mainstream GHC compiler release, building on the GHC-SMP implementation, and there has also been significant recent effort on visualization with the release of the EdenTV and ThreadScope visualizers for Eden and GHC-SMP, respectively. Work is currently underway to integrate GUM and GHC-SMP to give a widespectrum implementatation for GpH, and Eden will also form part of this effort. The SymGrid-Par system is being developed as part of a major UK project to provide support for high-performance computing on massively parallel machines, such as HECToR.
Future Directions The advent of multicore computers and the promise of manycore computers have changed the nature of parallel computing. The GpH model of lightweight multithreaded parallelism is well suited to this new order, offering the ability to easily generate large amounts of parallelism. The adaptivity mechanisms built into the GUM implementation mean that the parallel program can dynamically, and automatically, change its behavior to increase parallelism, improve locality, or throttle back parallelism as required. Future parallel systems are likely to be built more hierarchically than present ones, with processors built from heteregeneous combinations of general-purpose cores, graphics processing units, and other specialist units, using several communication networks and a complex memory hierarchy. Proponents of manycore architectures anticipate that a single processor
G
G
Glasgow Parallel Haskell (GpH)
will involve hundreds or even thousands of such units. Because of the cost of maintaining a uniform memory model across large numbers of systems, when deployed on a large scale these processors are likely to be combined hierarchically into multiple levels of clusters. These systems may then be used to form high performance “clouds.” Future parallel languages and implementations must therefore be highly flexible and adaptable, capable of dealing with multiple levels of communication latency and internal parallel structure, and perhaps fault tolerance. The GpH model with evaluation strategies, supported by adaptive implementations such as GUM forms a good basis for this, but it will be necessary to extend the existing models and implementations to cover more heterogeneous processor types, to deal with multiple levels of parallelism and additional program structure, and to focus more directly on locality issues.
Related Entries Eden Fortress (Sun HPCS Language) Functional Languages Futures
(TFP). Papers also frequently appear in the Journal of Functional Programming (JFP). A survey of Haskell-based parallel languages and implementations, as of , can be found in [], and a general introduction to research in parallel functional programming, as of , can be found in []. Some examples of the use of GpH in larger applications can be found in []. Much of this entry is based on private notes, e-mails, and final reports on the various research projects that have used GpH. The main paper on evaluation strategies is []. The main paper on the GUM implementation is [], and the main paper on the GranSim similator is []. Many subsequent papers have used these systems and ideas. For example, one recent paper describes the SymGrid-Par system []. Further material may be found on the GpH web page at http://www.macs.hw.ac.uk/~dsg/gph/, on the GdH web page at http://www.macs.hw.ac.uk/~dsg/ gdh/, on the GHC web page at http://www.haskell. org/ghc, on the SCIEnce project web page at http:// www.symbolic-computation.org, on the Eden web page at http://www.mathematik.uni-marburg.de/~eden, and on the contributor’s web page at http://www-fp.cs. st-andrews.ac.uk/~kh.
MPI (Message Passing Interface) MultiLisp NESL Parallel Skeletons Processes, Tasks, and Threads Profiling PVM (Parallel Virtual Machine) Shared-Memory Multiprocessors Sisal Speculation, Thread-Level
Bibliographic Notes and Further Reading The main venues for publication on GpH and other parallel functional languages are the International Conference on Functional Programming (ICFP) and its satellite events including the Haskell Symposium; the International Symposium on Implementation and Application of Functional Languages (IFL); the International Conference on Programming Language Design and Implementation (PLDI); and the Symposium on Trends in Functional Programming
Bibliography . Hammond K, Loidl H-W, Partridge A () Visualising granularity in parallel programs: a graphical winnowing system for Haskell. Proceedings of the HPFC’ – Conference on High Performance Functional Computing, Denver, pp –, – April . Hammond K, Michaelson G (eds) () Research directions in parallel functional programming. Springer, Heidelberg . Hammond K, Zain AA, Cooperman G, Petcu D, Trinder PW () SymGrid: a framework for symbolic computation on the grid. Proceedings of the EuroPar ’: th International EuroPar Conference, Springer LNCS , Rennes, France, pp – . Loidl H-W, Trinder P, Hammond K, Junaidu SB, Morgan RG, Peyton Jones SL () Engineering parallel symbolic programs in GpH. Concurrency Pract Exp ():– . Loogen R, Ortega-Mallén Y, Peña Mar R () Parallel functional programming in Eden. J Functional Prog ():– . Marlow S, Jones SP, Singh S () Runtime support for multicore Haskell. Proceedings of the ICFP ’: th ACM SIGPLAN International Conference on Functional Programming, ACM Press, New York . Mohr E, Kranz DA, Halstead RH Jr () Lazy task creation: a technique for increasing the granularity of parallel programs. IEEE Trans Parallel Distrib Syst ():– . Peyton Jones S, Clack C, Salkild J () High-performance parallel graph reduction. Proceedings of the PARLE’ – Conference
Global Arrays Parallel Programming Toolkit
.
.
.
.
.
. .
.
on Parallel Architectures and Languages Europe, Springer LNCS , pp – Peyton Jones S, Hall C, Hammond K, Partain W, Wadler P () The Glasgow Haskell compiler: a technical overview. Proceedings of the JFIT (Joint Framework for Information Technology) Technical Conference, Keele, UK, pp – Peyton Jones S (ed), Hughes J, Augustsson L, Barton D, Boutel B, Burton W, Fasel J, Hammond K, Hinze R, Hudak P, Johnsson T, Jones M, Launchbury J, Meijer E, Peterson J, Reid A, Runciman C, Wadler P () Haskell language and libraries. The revised report. Cambridge University Press, Cambridge Pointon R, Trinder P, Loidl H-W () The design and implementation of Glasgow distributed Haskell. Proceedings of the IFL’ – th International Workshop on the Implementation of Functional Languages, Springer LNCS , Aachen, Germany, pp – Rauber Du Bois A, Trinder P, Loidl H () mHaskell: mobile computation in a purely functional language. J Univ Computer Sci ():– Roe P () Parallel programming using functional languages. PhD thesis, Department of Computing Science, University of Glasgow Trinder P, Hammond K, Loidl H-W, Peyton Jones S () Algorithm + strategy = parallelism. J Funct Prog ():– Trinder P, Hammond K, Mattson J Jr, Partridge A, Peyton Jones S () GUM: a portable parallel implementation of Haskell. Proceedings of the PLDI’ – ACM Conference on Programming Language Design and Implementation, Philadelphia, pp – Trinder P, Loidl H-W, Pointon R () Parallel and distributed Haskells. J Funct Prog (&):–. Special Issue on Haskell
Global Arrays Global Arrays Parallel Programming Toolkit
G
Definition Global Arrays is a high-performance programming model for scalable, distributed-memory, parallel computer systems. Global Arrays is based on the concept of globally accessible dense arrays that are logically shared, yet physically distributed onto the memories of a parallel distributed computer system (Fig. illustrates this concept).
Discussion Introduction Global Arrays (GA) is a high-performance programming model for scalable, distributed-memory, parallel computer systems. GA is a library-based Partitioned Global Address Space (PGAS) programming model. The underlying supported sequential languages are Fortran, C, C++, and Python. GA provides global view access to very large dense arrays through API functions implemented for those languages, under a Single Program Multiple Data (SPMD) execution environment. GA was originally developed as part of the underlying software infrastructure for the US Department of Energy’s NWChem computational chemistry software package. Over time, it has been developed into a standalone package with a rich set of API functions (+) that cater to many needs in scientific application development. GA has been used to enable scalable Physically distributed data
Global Arrays Parallel Programming Toolkit Jarek Nieplocha,† , Manojkumar Krishnan , Bruce Palmer , Vinod Tipparaju , Robert Harrison , Daniel Chavarría-Miranda Pacific Northwest National Laboratory, Richland, WA, USA Oak Ridge National Laboratory, Oak Ridge, TN, USA
Synonyms Global arrays; GA † deceased.
Single, shared data structure
Global Arrays Parallel Programming Toolkit. Fig. Dual view of Global Arrays data structures
G
G
Global Arrays Parallel Programming Toolkit
parallel execution for several major scientific applications including NWChem (computational chemistry, specifically electronic structure calculation), STOMP (Subsurface Transport Over Multiple Phases, a subsurface flow and transport simulator), ScalaBLAST (a more scalable, higher-performance version of BLAST), Molpro (quantum chemistry), TE THYS (unstructured, implicit CFD and coupled fluid/solid mechanics finite volume code), Pagoda (Parallel Analysis of Geodesic Data), COLUMBUS (computational chemistry), and GAMESS-UK (computational chemistry). GA’s development has occurred over the last two decades. For this reason, the number of people involved and their contributions is large. GA’s original development occurred as a co-design effort between the NWChem team and the computer science team focused on GA. The main designer and original developer of GA was Jarek Nieplocha. Robert Harrison led the main effort in the development of NWChem.
Basic Global Arrays There are three classes of operations in Global Arrays: core operations, task parallel operations, and dataparallel operations. These operations have multiple language bindings, but provide the same functionality independent of the language. The current GA library contains approximately operations that provide a rich set of functionality related to data management and computations involving distributed arrays. GA is interoperable with MPI, enabling the development of hybrid programs that use both programming models. The basic components of the Global Arrays toolkit are function calls to create global arrays, copy data to, from, and between global arrays, and identify and access the portions of the global array data that are held locally. There are also functions to destroy arrays and free up the memory originally allocated to them. The basic function call for creating new global arrays is nga_create. The arguments to this function include the dimension of the array, the number of indices along each of the coordinate axes, and the type of data (integer, float, double, etc.) that each array element represents. The function returns an integer handle that can be used to reference the array in all subsequent operations. The allocation of data can be left completely to the toolkit, but if it is desirable to control the distribution
of data for load balancing or other reasons, additional versions of the nga_create function are available that allow the user to specify in detail how data is distributed between processors. The basic nga_create call provides a simple mechanism to control data distribution via the specification of an array that indicates the minimum dimensions of a block of data on each processor. One of the most important features of Global Arrays is the ability to easily move blocks of data between global arrays and local buffers. The data in the global array can be referred to using a global indexing scheme and data can be moved in a single function call, even if it represents data distributed over several processors. The nga_get function can be used to move a block of distributed data from a global array to a local buffer. The arguments consist of the array handle for the array that data is being taken from, two integer arrays representing the lower and upper indices that bound the block of distributed data that is going to be moved, a pointer to the local buffer or a location in the local buffer that is to receive the data, and an array of strides for the local data. The nga_put call is similar and can be used to move data in the opposite direction. The number of basic GA operations is fairly small and many parallel programs can be written with just the following ten routines: ● ● ●
●
● ● ●
●
GA_Initialize(): Initialize the GA library. GA_Terminate(): Release internal resources and finalize execution of a GA program. GA_Nnodes(): Return the number of GA compute processes (corresponds to the SPMD execution environment). GA_Nodeid(): Return the GA process ID of the calling compute process, this is a number between and GA_Nnodes() – 1. NGA_Create(): Create an n-dimensional globally accessible dense array (global array instance). NGA_Destroy(): Deallocate memory and resources associated with a global array instance. NGA_Put(): Copy data from a local buffer to an array section within a global array instance in a onesided manner. NGA_Get(): Copy data from an array section within a global array instance to a local buffer in a
Global Arrays Parallel Programming Toolkit
P2 GA_Get
P0
P1
P3
P4
G
P2 calls GA Get to get part of the global array into its local buffer P0
P1
P2 Determine ownership and data locality
P3
P4
G P0 Issue multiple non-blocking ARMCI_NbGet’s, one for data on each remote destination
Waits for completion of all data transfers
P1
P2
P3
P2
P4
P2 has the data, GA Get is complete
Global Arrays Parallel Programming Toolkit. Fig. Left: GA_Get flow chart. Right: An example: Process P issues GA_Get to get a chunk of data, which is distributed (partially) among P, P, P, and P (owners of the chunk)
one-sided manner (see Fig. for a detailed description of how get() operates). ● GA_Sync(): Synchronize compute processes via a barrier and ensure that all pending GA operations are complete (in accordance to the GA consistency model). ● NGA_Distribution(): Returns the array section owned by a specified compute process.
Example GA Program We present a parallel matrix multiplication program written in Global Arrays using the Fortran language interface. It uses most of the basic GA calls described before, in addition to some more advanced calls to create global array instances with specified data distributions. The program computes the result of C = A × B. Some variable declarations have been omitted for brevity.
Discussion on the Example Program The program is (mostly) a fully functional GA code, except for omitted variable declarations. It creates global array instances with specific data distributions, illustrates the use of the nga_put() and nga_get() primitives, as well as locality information through the nga_distribution() call. The code includes calls to initialize and terminate the MPI library, which are needed to provide the SPMD execution environment to the GA application. (It is possible to write a GA application that does not call the MPI library through the use of the TCGMSG simple message-passing environment included with GA.) The code includes the creation and use of local buffers to be used as sources and targets for put and get operations (lA, lB, lC), which in this case were allocated as Fortran dynamic arrays. Lines – contain the principal part of the example code and illustrate several of the features of
G 1 2 3 4 5 6 7 8 9
Global Arrays Parallel Programming Toolkit
program matmul integer :: sz integer :: i, j, k, pos, g_a, g_b, g_c integer :: nproc, me, ierr integer, dimension(2) :: dims, nblock, chunks integer, dimension(1) :: lead double precision, dimension(:,:), pointer :: lA, lB double precision, dimension(:,:), pointer :: lC call mpi_init(ierr)
10 call ga_initialize() 11 nproc = ga_nnodes() 12 me = ga_nodeid() 13 if (me .eq. 0) then 14 write (∗ , ∗ ) ’Running on: ’, nproc, ’ processors’ 15 end if 16 dims(:) = sz 17 chunks(:) = sz/sqrt(dble(nproc)) ! only runs on a perfect square number of processors 18 nblock(1) = sz/chunks(1) 19 nblock(2) = sz/chunks(2) 20 allocate(dmap(nblock(1) + nblock(2))) 21 pos = 1 22 do i = 1, sz - 1, chunks(1) ! compute beginning coordinate of each partition in the 1st dimension 23 dmap(pos) = i 24 pos = pos + 1 25 end do 26 do j = 1, sz - 1, chunks(2) ! compute beginning coordinate of each partition in the 2nd dimension 27 dmap(pos) = j 28 pos = pos + 1 29 end do 30 ret = nga_create_irreg(MT_DBL, ubound(dims), dims, ’A’, dmap, nblock, g_a)! create a global array instance with specified data distribution 31 ret = ga_duplicate(g_a, g_b, ’B’) ! duplicate same data distribution for array B 32 ret = ga_duplicate(g_a, g_c, ’C’) ! and C 33 allocate(lA(chunks(1), chunks(2)), lB(chunks(1), chunks(2)), lC(chunks(1), chunks(2))) 34 lA(:, :) = 1.0
Global Arrays Parallel Programming Toolkit
G
35 lB(:, :) = 2.0 36 lC(:, :) = 0.0 37 lead(1) = chunks(2) 38 call nga_distribution(g_a, me, tcoordsl, tcoordsh) 39 40 41 42
! initialize global array instances to respective values call nga_put(g_a, tcoordsl, tcoordsh, lA(1, 1), lead) call nga_put(g_b, tcoordsl, tcoordsh, lB(1, 1), lead) call nga_put(g_c, tcoordsl, tcoordsh, lC(1, 1), lead)
43 44 45 46 47
! obtain all tcoordsl1(1) tcoordsl1(2) tcoordsh1(1) tcoordsh1(2)
blocks in the row of the A matrix = 1 = tcoordsl(2) = chunks(1) = tcoordsh(2)
48 49 50 51 52
! obtain all tcoordsl2(1) tcoordsl2(2) tcoordsh2(1) tcoordsh2(2)
blocks in the column of the B matrix = tcoordsl(1) = 1 = tcoordsh(1) = chunks(2)
53 do pos = 1, nblock(1) ! matrix is square 54 call nga_get(g_a, tcoordsl1, tcoordsh1, lA(1, 1), lead) 55 call nga_get(g_b, tcoordsl2, tcoordsh2, lB(1, 1), lead) 56 57 58 59 60 61
do j = 1, n do k = 1, n do i = 1, n lC(i, j) = lC(i, j) + lA(i, k) * lB(k, j) end do end do
62 63 64
! advance coordinates for blocks tcoordsl1(1) = tcoordsl1(1) + chunks(1) tcoordsh1(1) = tcoordsh1(1) + chunks(1)
65 tcoordsl2(2) = tcoordsl2(2) + chunks(2) 66 tcoordsh2(2) = tcoordsh2(2) + chunks(2) 67 end do 68 ! lC contains the final result for the block owned by the process 69 call nga_put(g_c, tcoordsl, tcoordsh, lC(1, 1), lead) 70 ! do something with the result 71 call ga_print(g_c) 72 deallocate(dmap, lA, lB, lC)
G
G 73 74 75 76 77
Global Arrays Parallel Programming Toolkit
ret = ga_destroy(g_a) ret = ga_destroy(g_b) ret = ga_destroy(g_c) call ga_terminate() call mpi_finalize(ierr)
78 end program matmul
GA: data is being accessed using global coordinates (lines – and –) that correspond to the global size of the global array instances that were created previously; in addition, the access to the global array data for arrays g_a and g_b in lines and is done without requiring the participation of the process where that data is allocated (one-sided access). Figure illustrates the concept of one-sided access.
Global Arrays Concepts GA allows the programmer to control data distribution and makes the locality information readily available to be exploited for performance optimization. For example, global arrays can be created by: () allowing the
Process Z ga_get(a, 175, 185, 19, 70, buf, 10) Process Y ga_get(a, 180, 210, 23, 40, buf, 30) Process X ga_get(a, 100, 200, 17, 20, buf, 100)
Global Arrays Parallel Programming Toolkit. Fig. Any part of GA data can be accessed independently by any process at any time
library to determine the array distribution, () specifying the decomposition for only one array dimension and allowing the library to determine the others, () specifying the distribution block size for all dimensions, or () specifying an irregular distribution as a Cartesian product of irregular distributions for each axis. The distribution and locality information is always available through interfaces that allow the application developer to query: () which data portion is held by a given process, () which process owns a particular array element, and () a list of processes and the blocks of data owned by each process corresponding to a given section of an array. The primary mechanisms provided by GA for accessing data are block copy operations that transfer data between layers of memory hierarchy, namely, global memory (distributed array) and local memory. Further, extending the benefits of using blocked data accesses and copying remote locations into contiguous local memory can improve cache performance by reducing both conflict and capacity misses []. In addition, each process is able to access directly the data held in a section of a Global Array that is locally assigned to that process. Data representing sections of the Global Array owned by other processes on SMP clusters can also be accessed directly using the GA interface, if desired. Atomic operations are provided that can be used to implement synchronization and assure correctness of an accumulate operation (floating-point sum reduction that combines local and remote data) executed concurrently by multiple processes and targeting overlapping array sections. GA is extensible as well. New operations can be defined exploiting the low-level interfaces dealing with distribution, locality, and providing direct memory access (nga_distribution, nga_locate_ region, nga_access, nga_release, nga_ release_update). These, e.g., were used to provide
Global Arrays Parallel Programming Toolkit
additional linear algebra capabilities by interfacing with third-party libraries, e.g., ScaLAPACK [].
Global Arrays Memory Consistency Model In shared-memory programming, one of the issues central to performance and scalability is memory consistency. Although the sequential consistency model [] is straightforward to use, weaker consistency models [] can offer higher performance on modern architectures and they have been implemented on actual hardware. GA’s nature as a one-sided, global-view programming model requires similar attention to memory consistency issues. The GA approach is to use a weakerthan-sequential consistency model that is still relatively straightforward to understand by an application programmer. The main characteristics of the GA approach include: ●
GA distinguishes two types of completion of the store operations (i.e., put, scatter) targeting global shared memory: local and remote. The blocking store operation returns after the operation is completed locally, i.e., the user buffer containing the source of the data can be reused. The operation completes remotely after either a memory fence operation or a barrier synchronization is called. The fence operation is required in critical sections of the user code, if the globally visible data is modified. ● The blocking operations (get/put) are ordered only if they target overlapping sections of global arrays. Operations that do not overlap or access different arrays can complete in arbitrary order. ● The nonblocking get/put operations complete in arbitrary order. The programmer uses wait/test operations to order completion of these operations, if desired.
Global Arrays Extensions To allow the user to exploit data locality, the toolkit provides functions identifying the data from the global array that is held locally on a given processor. Two functions are used to identify local data. The first is the nga_distribution function, which takes a processor ID and an array handle as its arguments and returns a set of lower and upper indices in the global address space representing the local data block. The second is the nga_access function, which returns an array index and an array of strides to the locally held
G
data. In Fortran, this can be converted to an array by passing it through a subroutine call. The C interface provides a function call that directly returns a pointer to the local data. In addition to the communication operations that support task parallelism, the GA toolkit includes a set of interfaces that operate on either entire arrays or sections of arrays in the data-parallel style. These are collective data-parallel operations that are called by all processes in the parallel job. For example, movement of data between different arrays can be accomplished using a single function call. The nga_copy_patch function can be used to move a patch, identified by a set of lower and upper indices in the global index space, from one global array to a patch located within another global array. The only constraints on the two patches are that they contain equal numbers of elements. In particular, the array distributions do not have to be identical, and the implementation can perform, as needed, the necessary data reorganization (the so-called MxN problem []). In addition, this interface supports an optional transpose operation for the transferred data. If the copy is from one patch to another on the same global array, there is an additional constraint that the patches do not overlap.
Historical Development and Comparison with Other Programming Models The original GA package [–] offered basic one-sided communication operations, along with a limited set of collective operations on arrays in the style of BLAS []. Only two-dimensional arrays and two data types were supported. The underlying communication mechanisms were implemented on top of vendor-specific interfaces. In the course of years, the package evolved substantially and the underlying code was completely rewritten. This included separation of the GA internal one-sided communication engine from the high-level data structure. A new portable, general, and GAindependent communication library called ARMCI was created []. New capabilities were later added to GA without the need to modify the ARMCI interfaces. The GA toolkit evolved in multiple directions: ●
Adding support for a wide range of data types and virtually arbitrary array ranks. ● Adding advanced or specialized capabilities that address the needs of some new application areas,
G
G ●
●
●
●
Global Arrays Parallel Programming Toolkit
e.g., ghost cells or operations for sparse data structures. Expansion and generalization of the existing basic functionality. For example, mutex and lock operations were added to better support the development of shared-memory-style application codes. They have proven useful for applications that perform complex transformations of shared data in task parallel algorithms, such as compressed data storage in the multireference configuration interaction calculation in the COLUMBUS package []. Increased language interoperability and interfaces. In addition to the original Fortran interface, C, Python, and a C++ class library were developed. Developing additional interfaces to third-party libraries that expand the capabilities of GA, especially in the parallel linear algebra area: ScaLAPACK [] and SUMMA []. Interfaces to the TAO optimization toolkit have also been developed []. Developed support for multilevel parallelism based on processor groups in the context of a sharedmemory programming model, as implemented in GA [, ].
These advances generalized the capabilities of the GA toolkit and expanded its appeal to a broader set of applications. At the same time, the programming model, with its emphasis on a shared-memory view of the data structures in the context of distributed memory systems with a hierarchical memory, is as relevant today as it was in when the project started.
Comparison with Other Programming Models The two predominant classes of programming models for parallel computers are distributed-memory, sharednothing, and Uniform Memory Access (UMA) sharedeverything models. Both the shared-everything and fully distributed models have advantages and shortcomings. The UMA shared-memory model is easier to use but it ignores data locality/placement. Given the hierarchical nature of the memory subsystems in modern computers, this characteristic can have a negative impact on performance and scalability. Careful code restructuring to increase data reuse and replacing finegrained load/stores with block access to shared data can address the problem and yield performance for shared memory that is competitive with message passing [].
However, this performance comes at the cost of compromising the ease of use that the UMA shared-memory model posits. Distributed, shared-nothing memory models, such as message-passing or one-sided communication, offer performance and scalability but they are more difficult to program. The classic message-passing paradigm not only transfers data but also synchronizes the sender and receiver. Asynchronous (nonblocking) send/receive operations can be used to diffuse the synchronization point, but cooperation between sender and receiver is still required. The synchronization effect is beneficial in certain classes of algorithms, such as parallel linear algebra, where data transfer usually indicates completion of some computational phase; in these algorithms, the synchronizing messages can often carry both the results and a required dependency. For other algorithms, this synchronization can be unnecessary and undesirable, and a source of performance degradation and programming complexity. The Global Arrays toolkit [–] attempts to offer the best features of both models. It implements a globalview programming model, based on one-sided communication, in which data locality is managed by the programmer. This management is achieved by calls to functions that transfer data between a global address space (a distributed array) and local storage. In this respect, the GA model has similarities to the distributed shared-memory models that provide, e.g., an explicit acquire/release protocol []. However, the GA model acknowledges that remote data is slower to access than local data and allows data locality to be specified by the programmer and hence managed. GA is related to the global address space languages such as UPC [], Titanium [], and, to a lesser extent, Co-Array Fortran []. In addition, by providing a set of dataparallel operations, GA is also related to data-parallel languages such as HPF [], ZPL [], and Data Parallel C []. However, the Global Array programming model is implemented as a library that works with most languages used for technical computing and does not rely on compiler technology for achieving parallel efficiency. It also supports a combination of task and data parallelism and is fully interoperable with the message-passing (MPI) model. The GA model exposes to the programmer the hierarchical memory of modern high-performance computer systems [], and by recognizing the communication overhead for remote data
Global Arrays Parallel Programming Toolkit
transfers, it promotes data reuse and locality of reference. Virtually all scalable architectures possess nonuniform memory access characteristics that reflect their multilevel memory hierarchies. These hierarchies typically comprise processor registers, multiple levels of cache, local memory, and remote memory. Over time, both the number of levels and the cost (in processor cycles) of accessing deeper levels have been increasing. Scalable programming models must address memory hierarchy since it is critical to the efficient execution of applications.
Related Entries Coarray Fortran MPI (Message Passing Interface) PGAS (Partitioned Global Address Space) Languages UPC
Bibliographic Notes and Further Reading A more detailed version of this entry has been published in the International Journal of High Performance Computing Applications, vol. , no. , May by SAGE Publications, Inc., All rights reserved. © .
Bibliography . Lam MS, Rothberg EE, Wolf ME () Cache performance and optimizations of blocked algorithms. In: Proceedings of the th international conference on architectural support for programming languages and operating systems, Santa Clara, – Apr . Blackford LS, Choi J, Cleary A, D’Azevedo E, Demmel J, Dhillon I, Dongarra J, Hammarling S, Henry G, Petitet A, Stanley K, Walker D, Whaley RC () ScaLAPACK: a linear algebra library for message-passing computers. In: Proceedings of eighth SIAM conference on parallel processing for scientific computing, Minneapolis . Scheurich C, Dubois M () Correct memory operation of cache-based multiprocessors. In: Proceedings of th annual international symposium on computer architecture, Pittsburgh . Dubois M, Scheurich C, Briggs F () Memory access buffering in multiprocessors. In: Proceedings of th annual international symposium on Computer architecture, Tokyo, Japan . CCA-Forum. Common component architecture forum. http:// www.cca-forum.org . Nieplocha J, Harrison RJ, Littlefield RJ () Global arrays: a portable shared memory programming model for distributed memory computers. In: Proceedings of Supercomputing, Washington, DC, pp –
G
. Nieplocha J, Harrison RJ, Littlefield RJ () Global arrays: A nonuniform memory access programming model for highperformance computers. J Supercomput :– . Nieplocha J, Harrison RJ, Krishnan M, Palmer B, Tipparaju V () Combining shared and distributed memory models: Evolution and recent advancements of the Global Array Toolkit. In: Proceedings of POHLL’ workshop of ICS-, New York . Dongarra JJ, Croz JD, Hammarling S, Duff I () Set of level basic linear algebra subprograms. ACM Trans Math Softw :– . Nieplocha J, Carpenter B () ARMCI: a portable remote memory copy library for distributed array libraries and compiler runtime systems. In: Proceedings of RTSPP of IPPS/SDP’, San Juan, Puerto Rico . Dachsel H, Nieplocha J, Harrison RJ () An out-of-core implementation of the COLUMBUS massively-parallel multireference configuration interaction program. In: Proceedings of high performance networking and computing conference, SC’, Orlando . VanDeGeijn RA, Watts J () SUMMA: Scalable universal matrix multiplication algorithm. Concurr Pract Exp :– . Benson S, McInnes L, Moré JJ Toolkit for Advanced Optimization (TAO). http://www.mcs.anl.gov/tao . Nieplocha J, Krishnan M, Palmer B, Tipparaju V, Zhang Y () Exploiting processor groups to extend scalability of the GA shared memory programming model. In: Proceedings of ACM computing frontiers, Italy . Krishnan M, Alexeev Y, Windus TL, Nieplocha J () Multilevel parallelism in computational chemistry using common component architecture and global arrays. In: Proceedings of Supercomputing, Seattle . Shan H, Singh JP () A comparison of three programming models for adaptive applications on the origin. In: Proceedings of supercomputing, Dallas . Zhou Y, Iftode L, Li K () Performance evaluation of two home-based lazy release consistency protocols for shared virtual memory systems. In: Proceedings of operating systems design and implementation symposium, Seattle, pp – . Carlson WW, Draper JM, Culler DE, Yelick K, Brooks E, Warren K () Introduction to UPC and language specification. Center for Computing Sciences CCS-TR--, IDA Center for Computing Sciences, Bowie . Yelick K, Semenzato L, Pike G, Miyamoto C, Liblit B, Krishnamurthy A, Hilfinger P, Graham S, Gay D, Colella P, Aiken A () Titanium: A high-performance Java dialect. Concurr Pract Exp :– . Numrich RW, Reid JK () Co-array Fortran for parallel programming. ACM Fortran Forum :– . High Performance Fortran Forum () High Performance Fortran Language Specification, version .. Sci Program ():– . Snyder L () A programmer’s guide to ZPL. MIT Press, Cambridge . Hatcher PJ, Quinn MJ () Data-parallel programming on MIMD computers. MIT Press, Cambridge . Nieplocha J, Harrison RJ, Foster I () Explicit management of memory hierarchy. Adv High Perform Comput –
G
G
GpH (Glasgow Parallel Haskell)
Gossiping Allgather
GpH (Glasgow Parallel Haskell) Glasgow Parallel Haskell (GpH)
GRAPE Junichiro Makino National Astronomical Observatory of Japan, Tokyo, Japan
Definition GRAPE (GRAvity PipE) is the name of a series of special-purpose computers designed for the numerical simulation of gravitational many-body systems. Most of GRAPE machines consist of hardwired pipeline processors to calculate the gravitational interaction between particles and programmable computers to handle all other works. GRAPE-DR (Greatly Reduced Array of Processor Elements with Data Reduction) replaced the hardwired pipeline by simple SIMD programmable processors.
Discussion Introduction The improvement in the speed of computers has been a factor of in every decade, for the last years. In these years, however, the computer architecture has become more and more complex. Pipelined architecture were introduced in s, and vector architectures became the mainstream in s. In s, a number of parallel architectures appeared, but in the s and s, distributed memory parallel computers built from microprocessors have taken over. The technological driving force of this evolution of computer architecture has been the increase of the number of available transistors in integrated circuits, at least after the invention of integrated circuits in s. In the case of CMOS LSIs, the number of transistors in
a chip doubles in every months. Under the assumption of so-called CMOS scaling, this means that the switching speed doubles in every months. In the last years, the number of transistors available on an LSI chip increased by roughly a factor of one million. One way to make use of this huge number of transistors is to implement application-specific pipeline processors into a chip. The GRAPE series of special-purpose computers is one of such efforts to make efficient use of large number of transistors available on LSIs. In many scientific simulations, it is necessary to solve N-body problems numerically. The gravitational N-body problem is one such example, which describes the evolution of many astronomical objects from the solar system to the entire universe. In some cases, it is important to treat non-gravitational effects such as the hydrodynamical interaction, radiation, and magnetic fields, but the gravity is the primary driving force that shapes the universe. To solve the gravitational N-body problem, one needs to calculate the gravitational force on each body (particle) in the system from all other particles in the system. There are many ways to do so, and if relatively low accuracy is sufficient, one can use the Barnes– Hut tree algorithm [] or FMM []. Even with these schemes, the calculation of the gravitational interaction between particles (or particles and multipole expansions of groups of particles) is the most time-consuming part of the calculation. Thus, one can greatly improve the speed of the entire simulation, just by accelerating the speed of the calculation of particle–particle interaction. This is the basic idea behind GRAPE computers. The basic idea is shown in Fig. . The system consists of a host computer and special-purpose hardware, and the special-purpose hardware handles the calculation of gravitational interaction between particles. The host computer performs other calculations such as the time integration of particles, I/O, and diagnostics.
Host computer
Position, mass GRAPE Acceleration, potential
GRAPE. Fig. Basic structure of a GRAPE system
GRAPE
This architecture accelerates not only the simple algorithm in which the force on a particle is calculated by taking the summation of forces from all other particles in the system, but also the Barnes–Hut tree algorithms and FMM. Moreover, it can be used with individual timestep algorithms [], in which particles have their own times and timesteps and integrated in an event-driven fashion. The use of individual timestep is critical in simulations of many systems including star clusters and planetary systems, where close encounters and physical collisions of two particles require very small timesteps for a small number of particles.
History The GRAPE project started in . The first machine completed, the GRAPE- [], was a single-board unit on which around IC and LSI chips were mounted and wire-wrapped. The pipeline processor of GRAPE- was implemented using commercially available IC and LSI chips. It was a natural consequence of the fact that project members lacked both money and experience to design custom LSI chips. In fact, none of the original design and development team of GRAPE- had the knowledge of electronic circuit more than what was learned in basic undergraduate course for physics students. For GRAPE-, an unusually short word format was used, to make the hardware as simple as possible. The input coordinates are expressed in -bit fixed point format. After subtraction, the result is converted to -bit logarithmic format, in which bit are used for the “fractional” part. This format is used for all following operations except for the final accumulation. The final accumulation was done in -bit fixed point, to avoid overflow and underflow. The advantage of the short word format is that ROM chips can be used to implement complex functions that require two inputs. Any function of two -bit words can be implemented by one ROM chip with -bit address input. Thus, all operations other than the initial subtraction of the coordinates and final accumulation of the force were implemented by ROM chips. The use of extremely short word format in GRAPE- was based on the detailed theoretical analysis of error propagation and numerical experiment []. There are three dominant sources of error in numerical simulations of gravitational many-body systems. The first one
G
is the error in the numerical integration of orbits of particles. The second one is the error in the calculated accelerations themselves. The third one comes from the fact that in many cases, the number of particles used is much smaller than the number of stars in the real systems such as a galaxy. Whether or not the third one should be regarded as the source of error depends on the problem one wants to study. If the problem is, for example, merging of two galaxies, which takes place in relatively short timescale (compared to the orbital period of typical stars in galaxies), the effect of small number of particles can be, and should be, regarded as the numerical error. On the other hand, if we want to study long-term evolution of a star cluster, which takes place in the timescale much longer than the orbital timescale, the evolution of orbits of individual stars is driven by close encounters with other stars. In this case, the effect of small (or finite) number of particles is not an numerical error but what is there in real systems. Thus, the required accuracy of pairwise force calculation depends on the nature of the problem. In the case of the study of the merging of two galaxies, the average error can be as large as % of the pairwise force, if the error is guaranteed to be random. The average error of interaction calculated by GRAPE- is less than %, which was good enough for many problems. In the number format used in GRAPE-, the positions of particles are expressed in the -bit fixed-point format, so that the force between two nearby particles, both far from the origin of coordinates, is still expressed with sufficient accuracy. Also, the accumulation of the forces from different particles is done in the -bit fixedpoint format, so that there is no loss of effective bits during the accumulation. Thus, the primary source of error with GRAPE- is the low-accuracy pairwise force calculation. It can be regarded as random error. Strictly speaking, the error due to the short word format cannot always be regarded as random, since it introduces correlation both in space and time. One could eliminate this correlation by applying random coordinate transformation, but the quantitative study of such transformation has not done yet. GRAPE- used the GPIB (IEEE-) interface for the communication with the host computer. It was fast enough for the use with simple direct summation. However, when combined with the tree algorithm, faster
G
G
GRAPE
communication was necessary. GRAPE-A used VME bus for communication, to improve the performance. The speed of GRAPE- and A was around Mflops, which is around / of the speed of fastest vector supercomputers of the time. Hardware cost of them was around /, of supercomputers, or roughly equal to the cost of low-end workstations. Thus, these firstgeneration GRAPEs offered price-performance two orders of magnitude better than that of general-purpose computers. GRAPE- is similar to GRAPE-A, but with much higher numerical accuracy. In order to achieve higher accuracy, commercial LSI chips for floating-point arithmetic operations such as TI SNACT and Analog Devices ADSP/ were used. The pipeline of GRAPE- processes the three components of the interaction sequentially. So it accumulates one interaction in every three clock cycles. This approach was adopted to reduce the circuit size. Its speed was around Mflops, but it is still much faster than workstations or minicomputers at that time. GRAPE- was the first GRAPE computer with a custom LSI chip. The number format was the combination of the fixed point and logarithmic format similar to what were used in GRAPE-. The chip was fabricated using μm design rule by National Semiconductor. The number of transistors on chip was K. The chip operated at MHz clock speed, offering the speed of about . Gflops. Printed-circuit boards with eight chips were mass-produced, for the speed of . Gflops per board. Thus, GRAPE- was also the first GRAPE computer to integrate multiple pipelines into a system. Also, GRAPE- was the first GRAPE computer to be manufactured and sold by a commercial company. Nearly copies of GRAPE- have been sold to more than institutes (more than outside Japan). With GRAPE-, a high-accuracy pipeline was integrated into one chip. This chip calculates the first time derivative of the force, so that fourth-order Hermite scheme [] can be used. Here, again, the serialized pipeline similar to that of GRAPE- was used. The chip was fabricated using μm design rule by LSI Logic. Total transistor count was about K. The completed GRAPE- system consisted of , pipeline chips ( PCB boards each with pipeline chips). It operated on MHz clock, delivering the speed of . Tflops. Completed in , GRAPE- was
the first computer for scientific calculation to achieve the peak speed higher than Tflops. Also, in and , it was awarded the Gordon Bell Prize for peak performance, which is given to a real scientific calculation on a parallel computer with the highest performance. Technical details of machines from GRAPE- through GRAPE- can be found in [] and references therein. GRAPE- [] was an improvement over GRAPE-. It integrated two full pipelines which operate on MHz clock. Thus, a single GRAPE- chip offered the speed eight times more than that of the GRAPE- chip, or the same speed as that of an eight-chip GRAPE- board. GRAPE- was awarded the Gordon Bell Prize for price-performance. The GRAPE- chip was fabricated with . μ m design rule by NEC. Table summarizes the history of GRAPE project. Figure shows the evolution of GRAPE systems and general-purpose parallel computers. One can see that evolution of GRAPE is faster than that of generalpurpose computers. The GRAPE- was essentially a scaled-up version of GRAPE- [], with the peak speed of around Tflops. The peak speed of a single pipeline chip was Gflops. In comparison, GRAPE- consists of , pipeline chips, each with Mflops. The increase of a factor of in speed was achieved by integrating six pipelines into one chip (GRAPE- chip has one pipeline which needs three cycles to calculate the force from one particle) and using three times higher clock frequency. The advance of the device technology (from μm to . μm) made these improvements possible. Figure shows the processor chip delivered in early . The six pipeline units are visible. Starting with GRAPE-, the concept of virtual multiple pipeline (VMP) is used. VMP is similar to simultaneous multithreading (SMT), in the sense that a single pipeline processor behaves as multiple processors. However, what is achieved is quite different. In the case of SMT, the primary gain is in the latency tolerance, since one can execute independent instructions with different threads. In the case of hardwired pipeline processors, there is no need to reduce the latency. With VMP, the bandwidth to the external memory is reduced, since the data of one particle which exerts force are shared by multiple virtual pipelines, each of which calculates the force on its own particle. This sharing of the
GRAPE
G
GRAPE. Table History of GRAPE project GRAPE-
(/–/)
Mflops, low accuracy
GRAPE-
(/–/)
Mflops, high accuracy( bit/ bit)
GRAPE-A
(/–/)
Mflops, low accuracy
GRAPE-
(/–/)
Gflops, high accuracy
GRAPE-A
(/–/)
Mflops, high accuracy
HARP-
(/–/)
Mflops, high accuracy
GRAPE-A
(/–/)
Hermite scheme Gflops/board some copies are used all over the world GRAPE-
(/–/)
Tflops, high accuracy Some copies of small machines
MD-GRAPE
(/–/)
Gflops/chip, high accuracy programmable interaction
GRAPE-
(/–/)
Gflops/chip, low accuracy
GRAPE-
(/–/)
Tflops, high accuracy
1,000
Peak speed (Tflops)
100 10
Vectors MPPS GRAPEs
1 0.1 0.01 10-3 10-4 10-5
1980
1990
2000
2010
Year
GRAPE. Fig. The evolution of GRAPE and generalpurpose parallel computers. The peak speed is plotted against the year of delivery. Open circles, crosses, and stars denote GRAPEs, vector processors, and parallel processors, respectively
particle can be extended to physical multiple pipelines, as far as the total number of pipeline is not too large. Thus, special-purpose computers based on GRAPE-like pipeline have a unique advantage that their requirement of external memory bandwidth is much smaller than that of general-purpose computers with similar peak performance.
GRAPE. Fig. The GRAPE- processor chip
In the case of GRAPE-, each of six physical pipelines is implemented as eight virtual pipelines. Thus, one GRAPE- chip calculates forces on particles in parallel. The required memory bandwidth was MB/s, for the peak speed of Gflops. A traditional vector processor with the peak speed of Gflops would require the memory bandwidth of
G
G
GRAPE
GB/s. Thus, GRAPE- requires the memory bandwidth around / of that of a traditional vector processor. One processor board of GRAPE- housed GRAPE- chips. Each GRAPE- chip has its own memory to store particles which exert the force. Thus, different processor chips on one processor board calculate the forces on the same particles from different particles, and the partial results are summed up by an hardwired adder tree when the result is sent back to the host. Thus, the summation over chips added only a small startup overhead (less than μs) per one force calculation, which typically requires several milliseconds. The completed GRAPE- system consisted of processor boards, grouped into clusters with boards each. Within a cluster, boards are organized in a × matrix, with host computers. They are organized so that the effective communication speed is proportional to the number of host computers. In a simple configuration, the effective communication speed becomes independent of the number of host computers. The details of the network used in GRAPE- are given in [].
from image particles. The direct Ewald method is rather well suited for the implementation in hardware. In , WINE- was developed. It is a pipeline to calculate the wave-space part of the direct Ewald method. The real-space part can be handled by GRAPE-A or MDGRAPE hardware. In , a group led by Toshikazu Ebisuzaki in RIKEN started to develop MDM [], a massively parallel machine for large-scale MD simulations. Their primary goal was the simulation of protein molecules. MDM consists of two special-purpose hardware, massively parallel version of MD-GRAPE (MDGRAPE) and that of WINE (WINE-). The MDGRAPE- part consisted of , custom chips with four pipelines, for the theoretical peak speed of Tflops. The WINE- part consists of , custom pipeline chips, for the peak speed of Tflops. The MDM effort was followed up by the development of MDGRAPE- [], led by Makoto Taiji of RIKEN. MDGEAPE- achieved the peak speed of Pflops in .
Machines for Molecular Dynamics Classical MD calculation is quite similar to astrophysical N-body simulations since, in both cases, we integrate the orbit of particles (atoms or stars) which interact with other particles with simple pairwise force. In the case of Coulomb force, the force law itself is the same as that of the gravitational force, and the calculation of Coulomb force can be accelerated by GRAPE hardware. However, in MD calculations, the calculation cost of van der Waals force is not negligible, though van der Waals force decays much faster than the Coulomb force (r− compared to r− ). It is straightforward to design a pipelined processor which can handle particle–particle force given by some arbitrary function of the distance between particles. In GRAPE-A and its successors, a combination of table lookup and polynomial approximation is used. GRAPE-A and MD-GRAPE were developed in the University of Tokyo, following these lines of idea. GRAPE-A was built using commercial chips and MDGRAPE used a custom-designed pipeline chip. Another difference between astrophysical simulations and MD calculations is that in MD calculations, usually the periodic boundary condition is applied. Thus, we need some way to calculate Coulomb forces
Related Projects The GRAPE project is not the first project to implement the calculation of pairwise force in particle-based simulations in hardware. Delft Molecular Dynamics Processor [] (DMDP) is one of the earliest efforts. It was completed in early s. For the calculation of interaction between particles, it used the hardwired pipeline similar to that of GRAPE systems. However, in DMDP, time integration of orbits and other calculations are all done in the hardwired processors. Thus, in addition to the force calculation pipeline, DMDP had pipelines to update position, select particles for interaction calculation, and calculate diagnostics such as correlation function. FASTRUN [] has the architecture similar to that of DMDP, but designed to handle more complex systems such as protein molecule. To some extent, this difference in the designs of GRAPE computers and that of machines for molecular dynamics comes from the difference in the nature of the problem. In astronomy, the wide ranges in the number density of particles and timescale make it necessary to use adaptive schemes such as treecode and individual timesteps. With these schemes, the calculation cost per timestep per particle is generally higher than
GRAPE
that of fastest scheme optimized to shared timestep and near-uniform distribution of particles. The approach in which only the force calculation is done in hardware is more advantageous for astronomical N-body problems than for molecular dynamics. Anton [] is the latest effort to speed up the molecular dynamics simulation of proteins by specialized hardware. It is essentially the revival of the basic idea of DMDP, except that pipeline processors for operations other than force calculation were replaced by programmable parallel processors. It achieved the speed almost two orders of magnitude faster than that of general-purpose parallel computers for the simulation of protein molecules in water.
LSI Economics and GRAPE GRAPE has achieved the cost performance much better than that of general-purpose computers. One reason for this success is simply that with GRAPE architecture, one can use practically all transistors for arithmetic units, without being limited by the memory wall problem. Another reason is the fact that arithmetic units can be optimized to their specific uses in the pipeline. For example, in the case of GRAPE-, the subtraction of two positions is performed in -bit fixed point format, not in the floating-point format. Final accumulation is also done in fixed point. In addition, most of arithmetic operations to calculate the pairwise interactions are done in single precision. These optimizations made it possible to pack more than arithmetic units into a single chip with less than M transistors. The first microprocessor with fully pipelined double-precision floating-point unit, Intel , required . M transistors for two (actually one and half) operations. Thus, the number of transistors per arithmetic unit of GRAPE is smaller by more than a factor of . When compared with more recent processors, the difference becomes even larger. The Fermi processor from NVIDIA integrates arithmetic unit (adder and multiplier) with G transistors. Thus, it is five times less efficient than Intel , and nearly times less efficient than GRAPE-. The difference in the power efficiency is even larger, because the requirement for the memory bandwidth is lower for GRAPE computers. As a result, performance per watt of GRAPE- chip, fabricated with the nm design rule, is comparable to that of GPGPU chips fabricated with nm design rule. Thus, as silicon
G
technology advances, the relative advantage of specialpurpose architecture such as GRAPE becomes bigger. However, there is another economical factor. As the silicon semiconductor technology advances, the initial cost to design and fabricate custom chip increases. In , the initial cost for a custom chip was around K USD. By , it has become higher than M USD. By , the initial cost of a nm chip is around M USD. Roughly speaking, initial cost has been increasing as n. , where n is the number of transistors one can fit into a chip. The total budget for GRAPE- and GRAPE- projects is and M USD, respectively. Thus, a similar budget had become insufficient by early s. The whole point of special-purpose computer is to be able to outperform “expensive” supercomputers, with the price of – M USD. Even if a special-purpose computer is –, times faster, it is not practical to spend the cost of a supercomputer for a special-purpose computer which can solve only a narrow range of problems. There are several possible solutions. One is to reduce the initial cost by using FPGA (Field-Programmable Gate Array) chips. An FPGA chip consists of a number of “programmable” logic blocks (LBs) and also “programmable” interconnections. A LB is essentially a small lookup table with multiple inputs, augmented with one flip-flop and sometimes full-adder or more additional circuits. The lookup table can express any combinatorial logic for input data, and with flip-flop, it can be part of a sequential logic. Interconnection network is used to make larger and more complex logic, by connecting LBs. The design of recent FPGA chips has become much more complex, with large functional units like memory blocks and multiplier (typically × bit) blocks. Because of the need for the programmability, the size of the circuit that can be fit into an FPGA chip is much smaller than that for a custom LSI, and the speed of the circuit is also slower. Roughly speaking, the price of an FPGA chip per logic gate is around times higher than that of a custom chip with the same design rule. If the relative advantage of a specialized architecture is much larger than this factor of , its implementation based on FPGA chips can outperform general-purpose computers. In reality, there are quite a number of projects to use FPGAs for scientific computing, but most of them
G
G
GRAPE
turned out to be not competitive with general-purpose computers. The primary reason for this result is the relative cost of FPGA discussed above. Since the logic gate of FPGAs is much more expensive than that of general-purpose computers, the design of a specialpurpose computer with FPGA must be very efficient in gate usage. FPGA-based systems which use standard double- or single-precision arithmetic are generally not competitive with general-purpose computers. In order to be competitive, it is necessary to use much shorter word length. GRAPE architecture with reduced accuracy is thus an ideal target for FPGA-based approach. Several successful approaches have been reported [, ].
GRAPE-DR Another solution for the problem of the high initial cost is to widen the application range by some way to justify the high cost. GRAPE-DR project [] followed that approach. With GRAPE-DR, the hardwired pipeline processor of previous GRAPE systems was replaced by a collection of simple SIMD programmable processors. The internal network and external memory interface was designed so that it could emulate GRAPE processor efficiently and could be used for several other important applications, including the multiplication of dense matrices. GRAPE-DR is an acronym of “Greatly Reduced Array of Processor Elements with Data Reduction.” The last part, “Data Reduction,” means that it has an on-chip tree network which can do various reduction operations such as summation, max/min, and logical and/or. The GRAPE-DR project was started in FY , and finished in FY . The GRAPE-DR processor chip consists of simple processors, which can operate at the clock cycle of MHz, for the Gflops of single precision peak performance ( Gflops double precision). It was fabricated with TSMC nm process and the size is around mm . The peak power consumption is around W. The GRAPE-DR processor board houses four GRAPE-DR chips, each with its own local DRAM chips. It communicates with the host computer through Gen -lane PCI-Express interface. To some extent, the difference between GRAPE and GRAPE-DR is similar to that between traditional GPUs and GPGPUs. In both cases, hardwired pipelines are replaced by simple programmable processors. The main
differences between GRAPE-DR and GPGPUs are (a) processor element of GRAPE-DR is much simpler, (b) external memory bandwidth of GRAPE-DR is much smaller, and (c) GRAPE-DR is designed to achieve near-peak performance in real scientific applications such as gravitational N-body simulation and molecular dynamics simulation, and also dense matrix multiplication. These differences made GRAPE-DR significantly more efficient in both transistor usage and power usage. GRAPE-DR chip, which was fabricated with nm design rule and has mm area, integrates processing elements. The NVIDIA Fermi chip, which is fabricated with nm design rule and has > mm area, integrates the same processing elements. Thus, there is about a factor of difference in the transistor efficiency. This difference resulted in more than a factor of difference in the power efficiency. Whether or not the approach like GRAPE-DR will be competitive with other approaches, in particular GPGPUs, is at the time of writing rather unclear. The reason is simply that the advantage of a factor of is not quite enough, because of the difference in other factors, among which the most important is the development cycle. New GPUs are announced roughly every year, while it is somewhat unlikely that one develops the special-purpose computers every year, even if there is sufficient budget. In years, general-purpose computers become ten times faster, and GPGPUs will also become faster by a similar factor. Thus, a factor of advantage will disappear while the machine is being developed. On the other hand, the transistor efficiency of general-purpose computers, and that of GPUs, has been decreasing for the last years and probably will continue to do so for the next years or so. GRAPEDR can retain its efficiency when it is implemented with more advanced semiconductor technology, since, as in the case of GRAPE, one can use the increased number of transistors to increase the number of processor element. Thus, it might remain competitive.
Future Directions Future of Special-Purpose Processors In hindsight, s was a very good period for the development of special-purpose architecture such as GRAPE, because of two reasons. First, the semiconductor technology reached the point where many
GRAPE
floating-point arithmetic units can be integrated into a chip. Second, the initial design cost of a chip was still within the reach of fairly small research projects in basic science. By now, semiconductor technology reached to the point that one could integrate thousands of arithmetic units into a chip. On the other hand, the initial design cost of a chip has become too high. The use of FPGAs and the GRAPE-DR approach are two examples of the way to tackle the problem of increasing initial cost. However, unless one can keep increasing the budget, GRAPE-DR approach is not viable, simply because it still means exponential increase in the initial, and therefore total, cost of the project. On the other hand, such increase in the budget might not be impossible, since the field of computational science as a whole is becoming more and more important. Even though a supercomputer is expensive, it is still much less expensive compared to, for example, particle accelerators or space telescopes. Of course, computer simulation cannot replace the real experiments of observations, but computer simulations have become essential in many fields of science and technology. In addition, there are several technologies available in between FPGAs and custom chips. One is what is called “structured ASIC.” It requires customization of typically just one metal layer, resulting in large reduction in the initial cost. The number of gates one can fit into the given silicon area falls between those of FPGAs and custom chips. Another possibility is just to use the technology one or two generations older.
Application Area of Special-Purpose Computers Primary application area of GRAPE and GRAPE-DR has been the particle-based simulation, in particular that requires the evaluation of long-range interaction. It is suited to special-purpose computers because they are compute-intensive. In other words, the necessary bandwidth to the external memory is relatively small. Grid-based simulations based on schemes like finite-difference or finite-element methods are less compute-intensive and thus not suited to specialpurpose computers.
G
However, the efficiency of large-scale parallel computers based on general-purpose microprocessors for these grid-based simulation has been decreasing rather quickly. There are two reasons for this decrease. One is the lack of the memory bandwidth. Currently, the memory bandwidth of microprocessors normalized by the calculation speed is around . bytes/flops, which is not enough for most of grid-based simulations. Even so, this ratio will become smaller and smaller in the future. The other reason is the latency of the communication between processors. One possible solution for these two problems is to integrate the main memory, processor cores, and communication interface into a single chip. This integration gives practically unlimited bandwidth to the memory, and the communication latency is reduced by one or two orders of magnitude. Obvious disadvantage of this approach is that the total amount of memory would be severely limited. However, in many application of grid-based calculation, very long time integrations of relatively small systems are necessary. Many of such applications requires memory much less than one TB, which can be achieved by using several thousand custom processors each with around GB of embedded DRAM.
Bibliography . Aarseth SJ () Dynamical evolution of clusters of galaxies, I. MN : . Bakker AF, Gilmer GH, Grabow MH, Thompson K () A special purpose computer for molecular dyanamics calculations. J Comput Phys : . Barnes J, Hut P () A hiearchical O(NlogN) force calculation algorithm. Nature : . Fine R, Dimmler G, Levinthal C () FASTRUN: a special purpose, hardwired computer for molecular simulation. Proteins Struct Funct Genet : . Greengard L, Rokhlin V () A fast algorithm for particle simulations. J Comput Phys : . Hamada T, Fukushige T, Kawai A, Makino J () PROGRAPE-: a programmable, multi-purpose computer for many-body simulations. PASJ :– . Ito T, Makino J, Ebisuzaki T, Sugimoto D () A special-purpose N-body machine GRAPE-. Comput Phys Commun : . Kawai A, Fukushige T () $/GFLOP Astrophysical N-body simulation with a reconfigurable add-in card and a hierarchical tree algorithm. In: Proceedings of SC, ACM (Online) . Kawai A, Fukushige T, Makino J, Taiji M () GRAPE-: a special-purpose computer for N-body simulations. PASJ :
G
G
Graph Algorithms
. Makino J, Aarseth SJ () On a Hermite integrator with AhmadCohen scheme for gravitational many-body problems. PASJ : . Makino J, Fukushige T, Koga M, Namura K () GRAPE-: massively-parallel special-purpose computer for astrophysical particle simulations. PASJ : . Makino J, Hiraki K, Inaba M () GRAPE-DR: -Pflops massively-parallel computer with -core, -Gflops processor chips for scientific computing. In: Proceedings of SC, ACM (Online) . Makino J, Ito T, Ebisuzaki T () Error analysis of the GRAPE- special-purpose N-body machine. PASJ : . Makino J, Taiji M () Scientific simulations with specialpurpose computers – The GRAPE systems. Wiley, Chichester . Makino J, Taiji M, Ebisuzaki T, Sugimoto D () GRAPE: a massively parallel special-purpose computer for collisional N-body simulations. ApJ : . Narumi T, Susukita R, Ebisuzaki T, McNiven G, Elmegreen B () Mol Simul : . Shaw DE, Denero MM, Dror RO, Kuskin JS, Larson RH, Salmon JK, Young C, Batson B, Bowers KJ, Chao JC, Eastwood MP, Gagliardo J, Grossman JP, Ho CR, Ierardi DJ, Kolossváry I, Klepeis JL, Layman T, McLeavey C, Moraes MA, Mueller R, Priest EC, Shan Y, Spengler J, Theobald M, Towles B, Wang SC () Anton: a special-purpose machine for molecular dynamics simulation. In: Proceedings of the th annual international symposium on computer architecture (ISCA ’), ACM, San Diego, pp – . Taiji M, Narumi T, Ohno Y, Futatsugi N, Suenaga A, Takada N, Konagaya A () Protein explorer: a petaflops special-purpose computer system for molecular dynamics simulations. In: The SC Proceedings, IEEE, Los Alamitos, CD–ROM
synchronous, and in each unit of time, each processor either executes one instruction or stays idle.
Techniques Graph problems are diverse. Given a graph G = (V, E), where ∣V∣ = n and ∣E∣ = m, several techniques are frequently used in designing parallel graph algorithms. The basic techniques are described as follows (Detailed descriptions can be found in []). Prefix Sum
Given a sequence of n elements s , s , ⋯, sn with a binary associative operator denoted by ⊕, the prefix sums of the sequence are the partial sums defined by S i = s ⊕ s ⊕ ⋯ ⊕ si , ≤ i ≤ n Using the balanced tree technique, the prefix sums of n elements can be computed in O(log n) time with O(n/ log n) processors. Fast prefix sum is of fundamental importance to the design of parallel graph algorithms as it is frequently used in algorithms for more complex problems. Pointer Jumping
Discussion
Pointer jumping, also sometimes called path doubling, is useful for handling computation on rooted forests. For a rooted forest, there is a parent function P defined on the set of vertices. P(r) is set to r when r does not have a parent. When finding the root of each tree, the pointer jumping technique updates the parent of each node by that node’s grandparent, that is, set P(r) = P(P(r)). The algorithm runs in O(log n) time with O(n) processors. Pointer jumping can also be used to compute the distance between each node to its root. Pointer jumping is used in several connectivity and tree algorithms (e.g., see [, ]).
Parallel Graph Algorithms
Divide and Conquer
Relationships in real-world situations can often be represented as graphs. Efficient parallel processing of graph problems has been a focus of many algorithm researchers. A rich collection of parallel graph algorithms have been developed for various problems on different models. The majority of them are based on the parallel random access machine (PRAM). PRAM is a shared-memory model where data stored in the global memory can be accessed by any processor. PRAM is
The divide-and-conquer strategy is recognized as a fundamental technique in algorithm design (not limited to parallel graph algorithms). The frequently used quicksort algorithm is based on divide and conquer. The strategy partitions the input into partitions of roughly equal size and recursively works on each partition concurrently. There is usually a final step to combine the solutions of the subproblems into a solution for the original problem.
Graph Algorithms David A. Bader , Guojing Cong Georgia Institute of Technology, Atlanta, GA, USA IBM, Yorktown Heights, NY, USA
Graph Algorithms
Pipelining
Like divide-and-conquer, the use of pipelining is not limited to parallel graph algorithm design. It is of critical importance in computer hardware and software design. Pipelining breaks up a task into a sequence of smaller tasks (subtasks). Once a subtasks is complete and moves to the next stage, the ones following it can be processed in parallel. Insertion and deletion with – trees demonstrate the pipelining technique []. Deterministic Coin Tossing
Deterministic coin tossing was proposed by Cole and Vishkin [] to break the symmetry of a directed cycle without using randomization. Consider the problem of finding a three-coloring of graph G. A k-coloring of G is a mapping c : V → {, , ..., k − } such that c(i) ≠ c(j) if < i, j > ∈ E. Initially, the cycle has a trivial coloring with n = ∣V∣ colors. The apparent symmetry in the problem (i.e., the vertices cannot be easily distinguished) presents the major difficulty to reducing the number of colors needed for the coloring. Deterministic coin tossing uses the binary representation of an integer i = ⋯ik ⋯i i to break the symmetry. For example, suppose t is the least significant bit position in which c(i) and c(S(i)) differ (S(i) is the successor of i in the cycle), then the new coloring for i can be set as t + c(i)t . Accelerating Cascades
Accelerating cascades was presented together with deterministic coin tossing in [] for the design of faster algorithms for list ranking and other problems. This technique combines two algorithms, one optimal and the other super fast, for the same problem to get a optimal and very fast algorithm. The general strategy is to start with the optimal algorithm to reduce the problem size and then apply the fast but nonoptimal algorithm. Other Building Blocks for Parallel Graph Algorithms
List ranking determines the position, or rank, of the list items in a linked list. A generalized list-ranking problem is defined as follows. Let X be an array of n elements stored in arbitrary order. For each element i, let X(i).value be its value and X(i).next be the index of its successor. Then, for any binary associative operator ⊕, compute X(i).prefix such that X(head).prefix = X(head).value and X(i).prefix =
G
X(i).value ⊕ X(predecessor).prefix, where head is the first element of the list, i is not equal to head, and predecessor is the node preceding i in the list. Pointer jumping can be applied to solve the list-ranking problem. An optimal list-ranking algorithm is given in []. The Euler tour technique is another of the basic building blocks for designing parallel algorithms, especially for tree computations. For example, postorder/preorder numbering, computing the vertex level, computing the number of descendants, etc., can be done work-time optimally on EREW PRAM by applying the Euler tour technique. As suggested by its name, the power of the Euler tour technique comes from defining a Eulerian circuit on the tree. In Tarjan and Vishkin’s biconnected components paper [] that originally introduced the Euler tour technique, the input to their algorithm is an edge list with the cross-pointers between twin edges < u, v > and < v, u > established. With these cross-pointers it is easy to derive an Eulerian circuit. The Eulerian circuit can be treated as a linked list, and by assigning different values to each edge in the list, list ranking can be used for many tree computation. For example, when rooting a tree, the value is associated with each edge. After list ranking, simply inspecting the list rank for < u, v > and < v, u > can set the correct parent relationship for u and v. Tree contraction systematically shrinks a tree into a single vertex by successively shrinking parts of the tree. It can be used to solve the expression evaluation problem. It is also used in other algorithm, for example, computing the biconnected components [].
Classical Algorithms The use of the basic techniques are demonstrated in several classical graph algorithms for spanning tree, minimum spanning tree, and biconnected components. Various deterministic and randomized techniques have been given for solving the spanning tree problem (and the closely related connected components problem) on PRAM models. A brief survey of these algorithms can be found in []. The Shiloach–Vishkin algorithm (SV) algorithm is representative of several connectivity algorithms in that it adapts the widely used graft-and-shortcut approach. Through carefully designed grafting schemes, the algorithm achieves complexities of O(log n) time and O((m + n) log n)
G
G
Graph Algorithms
work under the arbitrary CRCW PRAM model. The algorithm takes an edge list as input and starts with n isolated vertices and m processors. Each processor Pi ( ≤ i ≤ m) inspects edge ei =< vi , vi > and tries to graft vertex vi to vi under the constraint that vi < vi . Grafting creates k ≥ connected components in the graph, and each of the k components is then shortcut to to a single super-vertex. Grafting and shortcutting are iteratively applied to the reduced graphs G′ = (V ′ , E′ ) (where V ′ is the set of super-vertices and E′ is the set of edges among super-vertices) until only one super-vertex is left. Minimum spanning tree (MST) is one of the most studied combinatorial problems with practical applications. While several theoretic results are known for solving MST in parallel, many are considered impractical because they are too complicated and have large constant factors hidden in the asymptotic complexity. See for a survey of MST algorithms. Many parallel algorithms are based on the Bor˚uvka algorithm. Bor˚uvka’s algorithm is comprised of Bor˚uvka iterations that are used in many parallel MST algorithms. A Bor˚uvka iteration is characterized by three steps: find-min, connectedcomponents, and compact-graph. In find-min, for each vertex v the incident edge with the smallest weight is labeled to be in the MST; connect-components identifies connected components of the induced graph with the labeled MST edges; compact-graph compacts each connected component into a single supervertex, removes self-loops and multiple edges, and relabels the vertices for consistency. A connected graph is said to be separable if there exists a vertex v such that removal of v results in two or more connected components of the graph. Given a connected, undirected graph G, the biconnected components problem finds the maximal-induced subgraphs of G that are not separable. Tarjan [] presents an optimal O(n + m) algorithm that finds the biconnected components of a graph based on depth-first search (DFS). Eckstein [] gave the first parallel algorithm that takes O(d log n) time with O((n + m)/d) processors on CREW PRAM, where d is the diameter of the graph. Tarjan and Vishkin [] present an O(log n) time algorithm on CRCW PRAM that uses O(n + m) processors. This algorithm utilizes many of the fundamental primitives including prefix sum, list ranking, sorting, connectivity, spanning tree, and tree computations.
Communication Efficient Graph Algorithms Communication-efficient parallel algorithms were proposed to address the “bottleneck of processor-toprocessor communication” (e.g., see []). Goodrich [] presented a communication-efficient sorting algorithm on weak-CREW BSP that runs in O(log n/ log(h + )) communication rounds (with at most h data transported by each processor in each round) and O((n log n)/p) local computation time, for h = Θ(n/p). Goodrich’s sorting algorithm is frequently used in communication-efficient graph algorithms. Dehne et al. designed an efficient list-ranking algorithm for coarse-grained multicomputers (CGM) and BSP that takes O(log p) communication rounds with O(n/p) local computation. In the same study, a series of communication-efficient graph algorithms such as connected components, ear decomposition, and biconnected components are presented using the list-ranking algorithm as a building block. On the BSP model, Adler et al. [] presented a communication-optimal MST algorithm. The list-ranking algorithm and the MST algorithm take similar approaches to reduce the number of communication rounds. They both start by simulating several (e.g., O(log p) or O(log log p) ) steps of the PRAM algorithms on the target model to reduce the input size so that it fits in the memory of a single node. A sequential algorithm is then invoked to process the reduced input of size O(n/p), and finally the result is broadcast to all processors for computing the final solution.
Practical Implementation Many real-world graphs are large and sparse. These instances are especially hard to process due to the characteristics of the workload. Although fast theoretic algorithms exist in the literature, large and sparse graph problems are still challenging to solve in practice. There remains a significant gap between algorithmic model and architecture. The mismatch between memory access pattern and cache organization is the most outstanding barrier to high-performance graph analysis on current systems.
Graph Workload Compared with traditional scientific applications, graph analysis is more memory intensive. Graph algorithms put tremendous pressure on the memory subsystem
G
Graph Algorithms
to deliver data to the processor. Table shows the percentages of memory instructions executed in the SPLASH benchmarks [] and several graph algorithms. SPLASH represents typical scientific applications for shared-memory environments. On IBM Power, on average, about % more memory instructions are executed in the graph algorithms. For biconnected components, the number is over %. For memory-intensive applications, the locality behavior is especially crucial to the performance. Unfortunately, most graph algorithms exhibit erratic memory access patterns that result in poor performance, as shown in Table . Table is the cycles per instruction (CPI) construction [] for three graph algorithms on IBM Power. The algorithms studied are betweenness centrality (BC), biconnected components (BiCC), and minimum spanning tree (MST). CPI construction attributes in percentage cycles to categories such as completion, instruction cache miss penalty, and stall. In Table , for all algorithms, significant amount of cycles are spent on pipeline stalls. About –% of the cycles are wasted on the load-store unit stalls, and more than % of the cycles are spent on stalls due to data cache misses. Table clearly shows that graph algorithms perform poorly on current cache-based architectures, and the culprit is the memory access pattern. The floating-point stalls (FPU STL) column is revealing about one other prominent feature of graph algorithms.
FPU STL is the percentage of cycles wasted on floatingpoint unit stalls. In Table , BiCC and MST do not incur any FPU stalls. Unfortunately, this does not mean that the workload fully utilizes the FPU. Instead, as there are no floating-point operations in these algorithms, the elaborately designed floating point units lay idle. CPI construction shows that graph workloads spent most of the time waiting for data to be delivered, and there are not many other operations to hide the long latency to main memory. In fact, of the three algorithms, only BC incurs execution of a few floating-point instructions.
Implementation on Shared-Memory Machines It is relatively straightforward to map PRAM algorithms on to shared-memory machines such as symmetric multiprocessors (SMPs), multi-core and many core systems. While these systems are of shared-memory architecture, they are by no means the PRAM used in theoretical work – synchronization cannot be taken for granted, memory bandwidth is limited, and performance requires a high degree of locality. Practical design choices need to be made to achieve high performance on such systems. Adapting to the Available Parallelism
Nick’s Class (N C) is defined as the set of all problems that run in polylog-time with a polynomial number of processors. Whether a problem P is in N C is a fundamental question. The PRAM model assumes an
Graph Algorithms. Table Percentages of load-store instructions for the SPLASH benchmark and the graph problems. ST, MST, BiCC, and CC stand for spanning tree, minimum spanning tree, biconnected components, and connected components, respectively SPLASH
Graph problems
Benchmark
Barnes
Cholesky
Ocean
Raytrace
ST
MST
BiCC
CC
Load
.%
.%
.%
.%
.%
.%
.%
.%
Store
.%
.%
.%
.%
%
.%
.%
.%
Load+store
.%
.%
.%
.%
.%
.%
.%
.%
Graph Algorithms. Table CPI construction for three graph algorithms. Base cycles are for “useful” work. The “Stall” columns show the percentages of cycles on pipeline stalls followed by stalls due to load-store unit, data cache miss, floating-point unit, fix point unit, respectively Algorithm
Base
GCT
Stall
LSU STL
BC
.
.
.
.
BiCC
.
.
.
MST
.
.
.
DCache STL
FPU STL
FXU STL
.
.
.
.
.
.
.
.
.
.
.
G
G
Graph Algorithms
unlimited number of processors and explores the maximum inherent parallelism of P. Acknowledging the practical restriction of limited parallelism provided by real computers, Kruskal et al. [] argued that nonpolylogarithmic time algorithms (e.g., sublinear time algorithms) could be more suitable than polylog algorithms for implementation with practically large input size. EP (short for efficient parallel) algorithms, by the definition in [], is the class of algorithms that achieve a polynomial reduction in running time with a polylogarithmic inefficiency. With EP algorithms, the design focus is shifted from reducing the complexity factors to solving problems of realistic sizes efficiently with a limited number of processors. Reducing Synchronization
When adapting a PRAM algorithm to shared-memory machines, a thorough understanding of the algorithm usually suffices to eliminate unnecessary synchronization such as barriers. Reducing synchronization thus is an implementation issue. Asynchronous algorithm design, however, is more aggressive in reducing synchronization. It sometimes allows nondeterministic intermediate results but deterministic solutions. In a parallel environment, to ensure correct final results oftentimes a total ordering on all the events is not necessary, and a partial ordering in general suffices. Relaxed constraints on ordering reduce the number of synchronization primitives in the algorithm. Bader and Cong presented a largely asynchronous spanning tree algorithm in [] that employs a constant number of barriers. The spanning tree algorithm for shared-memory machines has two main steps: () stub spanning tree and () work-stealing graph traversal. In the first step, one processor generates a stub spanning tree, that is, a small portion of the spanning tree by randomly walking the graph for O(p) steps. The vertices of the stub spanning tree are evenly distributed into each processor’s queue, and each processor in the next step will traverse from the first element in its queue. After the traversals in step , the spanning subtrees are connected to each other by this stub spanning tree. In the graph traversal step, each processor traverses the graph (by coloring the nodes) similar to the sequential algorithm in such a way that each processor finds a subgraph of the final spanning tree. Work-stealing is used to balance the load for graph traversal.
One problem related to synchronization is that there could be portions of the graph traversed by multiple processors and be in different subgraphs of the spanning tree. The immediate remedy is to synchronize using either locks or barriers. With locks, coloring the vertex becomes a critical section, and a processor can only enter the critical section when it gets the lock. Although the nondeterministic behavior is now prevented, it does not perform well on large graphs due to an excessive number of locking and unlocking operations. In the proposed algorithm, no barriers are introduced in graph traversal. The algorithm runs correctly without, barriers even when two or more processors color the same vertex. In this situation, each processor will color the vertex and set as its parent the vertex it has just colored. Only one processor succeeds at setting the vertex’s parent to a final value. For example, using Fig. , processor P colored vertex u, and processor P colored vertex v, and at a certain time they both find w unvisited and are now in a race to color vertex w. It makes no difference which processor colored w last because w’s parent will be set to either u or v (and it is legal to set w’s parent to either of them; this will not change the validity of the spanning tree, only its shape). Further, this event does not create cycles in the spanning tree under sequential consistency model. Both P and P record that w is connected to each processor’s own tree. When each of w’s unvisited children are visited by various processors, its parent will be set to w, independent of w’s parent.
P2
P1
v
u w
Graph Algorithms. Fig. Two processors P and P see vertex w as unvisited, so each is in a race to color w and set w’s parent pointer. The shaded area represents vertices colored by P , the black area represents those marked by P , and the white area contains unvisited vertices
Graph Algorithms
Choosing Between Barriers and Locks
Locks and barriers are two major types of synchronization primitives. In practice, the choice of using locks or barriers may not be very clear. Take the “graft and shortcut” spanning tree algorithm for example. For graph G = (V, E) represented as an edge list, the algorithm starts with n isolated vertices and m processors. For edge ei = ⟨u, v⟩, processor Pi ( ≤ i ≤ m) inspects u and v, and if v < u, it grafts vertex u to v and labels ei to be a spanning tree edge. The problem here is that for a certain vertex v, its multiple incident edges could cause grafting v to different neighbors, and the resulting tree may not be valid. To ensure that v is only grafted to one of the neighbors, locks can be used. Associated with each vertex v is a flag variable protected by a lock that shows whether v has been grafted. In order to graft v a processor has to obtain the lock and check the flag, thus race conditions are prevented. A different solution uses barriers [] in a two-phase election. No checking is needed when a processor grafts a vertex, but after all processors are done (ensured with barriers), a check is performed to determine which one succeeds and the corresponding edge is labeled as a tree edge. Whether to use a barrier or lock is dependent on the algorithm design as well as the barrier and lock implementations. Locking typically introduces large memory overhead. When contention among processors is intense, the performance degrades significantly. Cache Friendly Design
The increasing speed difference between processor and main memory makes cache and memory access patterns important factors for performance. The fact that modern processors have multiple levels of memory hierarchy is generally not reflected by most of the parallel models. As a result, few parallel algorithm studies have touched on the cache performance issue. The SMP model proposed by Helman and JáJá is the first effort to model the impact of memory access and cache over an algorithm’s performance []. The model forces an algorithm designer to reduce the number of noncontiguous memory accesses. However, it does not give hints to the design of cache-friendly parallel algorithms. Chiang et al. [] presented a PRAM simulation technique for designing and analyzing efficient externalmemory (sequential) algorithms for graph problems. This technique simulates the PRAM memory by
G
keeping a task array of O(N) on disk. For each PRAM step, the simulation sorts a copy of the contents of the PRAM memory based on the indices of the processors for which they will be operands, and then scans this copy and performs the computation for each processor being simulated. The following can be easily shown: Theorem Let A be a PRAM algorithm that uses N processors and O(N) space and runs in time T. Then A can be simulated in O(T ⋅sort(N)) I/Os []. Here, sort(N) represents the optimal number of I/Os needed to sort N items striped across the disks, and scan(N) represents the number of I/Os needed to read N items striped across the disks. Specifically, x x sort(x) = log M B DB B x scan(x) = DB where M = # of items that can fit into main memory, B = # of items per disk block, and D = # of disks in the system. A similar technique can be applied to the cachefriendly parallel implementation of PRAM algorithms for large inputs. I/O efficient algorithms exhibit good spatial locality behavior that is critical to good cache performance. Instead of having one processor simulate the PRAM step, p ≪ n processors may perform the simulation concurrently. The simulated PRAM implementation is expected to incur few cache block transfers between different levels. For small input sizes, it would not be worthwhile to apply this technique as most of the data structures can fit into cache. As the input size increases, the cost to access memory becomes more significant, and applying the technique becomes beneficial. Algorithmic Optimizations
For most problems, parallel algorithms are inherently more complicated than the sequential counterparts, incurring large overheads with many algorithm steps. Instead of lowering the asymptotic complexities, in many cases, reducing the constant factors improves performance. Cong and Bader demonstrates the benefit of such optimizations with their biconnected components algorithm []. The algorithm eliminates edges that are not essential in computing the biconnected components. For any
G
G
Graph Algorithms
input graph, edges are first eliminated before the computation of biconnected components is done so that at most min(m, n) edges are considered. Although applying the filtering algorithm does not improve the asymptotic complexity, in practice, the performance of the biconnected components algorithm can be significantly improved. An edge e is considered as nonessential for biconnectivity if removing e does not change the biconnectivity of the component to which it belongs. Filtering out nonessential edges when computing biconnected components (these edges are put back in later) yields performance advantages. The Tarjan–Vishkin algorithm (TV) is all about finding the equivalence relation R′∗ c [, ]. Of the three conditions for R′c , it is trivial to check for condition which is for a tree edge and a non-tree edge. Conditions and , however, are for two tree edges, and checking involves the computation of high and low values. To compute high and low, every non-tree edge of the graph is inspected, which is very time consuming when the graph is not extremely sparse. The fewer edges the graph has, the faster the Low-high step. Also, when building the auxiliary graph, the fewer edges in the original graph means the smaller the auxiliary graph and the faster the Label-edge and Connected-components steps. Combining the filtering algorithm for eliminating nonessential edges and TV, the new biconnected components algorithm runs in max(O(d), O(log n)) time with O(n) processors on CRCW PRAM, where d is the diameter of the graph. Asymptotically, the new algorithm is not faster than TV. In practice, however, parallel speedups upto with processors are achieved on SUN Enterprise using the filtering technique.
Implementation on Multithreaded Architectures Graph algorithms have been observed to run well on multi-threaded architectures such as the CRAY MTA- [] and its successor, the Cray XMT. The Cray MTA [] is a flat, shared-memory multiprocessor system. All memory is accessible and equidistant from all processors. There is no local memory and no data caches. Parallelism, and not caches, is used to tolerate memory and synchronization latencies. An MTA processor consists of hardware streams and one instruction pipeline. Each stream can have up to outstanding memory operations. Threads from the
same or different programs are mapped to the streams by the runtime system. A processor switches among its streams every cycle, executing instructions from nonblocked streams in a fair manner. As long as one stream has a ready instruction, the processor remains fully utilized. Bader et al. compared the performance of listranking algorithm on SMPs and MTA []. For list ranking, they used two classes of list to test the algorithms: Ordered and Random. Ordered places each element in the array according to its rank; thus, node i is the ith position of the array and its successor is the node at position (i + ). Random places successive elements randomly in the array. Since the MTA maps contiguous logical addresses to random physical addresses, the layout in physical memory for both classes is similar, the performance on the MTA is independent of order. This is in sharp contrast to SMP machines which rank Ordered lists much faster than Random lists. On the SMP, there is a factor of – difference in performance between the best case (an ordered list) and the worst case (a randomly-ordered list). On the ordered lists, the MTA is an order of magnitude faster than the SMP, while on the random list, the MTA is approximately times faster.
Implementation on Distributed Memory Machines As data partitioning and explicit communication are required, implementing highly irregular algorithms is hard on distributed-memory machines. As a result, although many fast theoretic algorithms exist in the literature, few experimental results are known. As for performance, the adverse impact of irregular accesses is magnified in the distributed-memory environment when memory requests served by remote nodes experience long network latency. Two studies have demonstrated reasonable parallel performance with distributed-memory machines [, ]. Both studies implement parallel breadth-first search (BFS), one on BlueGene/L [] and the other on CELL/BE []. The CELL architecture resembles a distributed-memory setting as explicit data transfer is necessary between the local storage on an SPE and the main memory. Neither study establishes a strong evidence for fast execution of parallel graph algorithms on distributed-memory systems. The individual CPU in BlueGene and the PPE
Graph Algorithms
of CELL are weak compared with other Power processors or the SPE. It is hard to establish a meaningful baseline to compare the parallel performance against. Indeed, in both studies, either only wall clock times or speedups compared with other reference architectures are reported. Partitioned global address space (PGAS) languages such as UPC and X [, ] have been proposed recently that present a shared-memory abstraction to the programmer for distributed-memory machines. They allow the programmer to control the data layout and work assignment for the processors. Mapping shared-memory graph algorithms onto distributedmemory machines is straightforward with PGAS languages. Performance wise, straightforward PGAS implementation for irregular graph algorithms does not usually achieve high performance due to the aggregate startup cost of many small messages. Cong, Almasi, and Saraswat presented their study in optimizing the UPC implementation of graph algorithm in []. They improve both the communication efficiency and the cache performance of the algorithm through improving the locality behavior.
Some Experimental Results Greiner [] implemented several connected components algorithms using NESL on the Cray Y-MP/C and TMC CM-. Hsu, Ramachandran, and Dean [] also implemented several parallel algorithms for connected components. They report that their parallel code runs times slower on a MasPar MP- than Greiner’s results on the Cray, but Hsu et al.’s implementation uses one-fourth of the total memory used by Greiner’s approach. Krishnamurthy et al. [] implemented a connected components algorithm (based on ShiloachVishkin []) for distributed memory machines. Their code achieved a speedup of using a -processor TMC CM- on graphs with underlying D and D regular mesh topologies, but virtually no speedup on sparse random graphs. Goddard, Kumar, and Prins [] implemented a connected components algorithm for a mesh-connected SIMD parallel computer, the processor MasPar MP-. They achieve a maximum parallel speedup of less than two on a random graph with , vertices and about one-million edges. For a
G
random graph with , vertices and fewer than a halfmillion edges, the parallel implementation was slower than the sequential code. Chung and Condon [] implemented a parallel minimum spanning tree (MST) algorithm based on Bor˚uvka’s algorithm. On a -processor CM-, for geometric graphs with , vertices and average degree and graphs with fewer vertices but higher average degree, their code achieved a parallel speedup of about , on -processors, over the sequential Bor˚uvka’s algorithm, which was – times slower than their sequential Kruskal algorithm. Dehne and Götz [] studied practical parallel algorithms for MST using the BSP model. They implemented a dense Bor˚uvka parallel algorithm, on a -processor Parsytec CC-, that works well for sufficiently dense input graphs. Using a fixed-sized input graph with , vertices and , edges, their code achieved a maximum speedup of . using processors for a random dense graph. Their algorithm is not suitable for sparse graphs. Woo and Sahni [] presented an experimental study of computing biconnected components on a hypercube. Their test cases are graphs that retain and % edges of the complete graphs, and they achieved parallel efficiencies up to . for these dense inputs. The implementation uses adjacency matrix as input representation, and the size of the input graphs is limited to less than K vertices. Bader and Cong presented their studies [, , ] of the spanning tree, minimum spanning tree, and biconnected components algorithms on SMPs. They achieved reasonable parallel speedups on the large, sparse inputs compared with the best sequential implementations.
Bibliography . Adler M, Dittrich W, Juurlink B, Kutyłowski M, Rieping I () Communication-optimal parallel minimum spanning tree algorithms (extended abstract). In: SPAA’: proceedings of the tenth annual ACM symposium on parallel algorithms and architectures. ACM, New York, pp – . Allen F, Almasi G et al () Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Syst J ():– . Bader DA, Cong G () A fast, parallel spanning tree algorithm for symmetric multiprocessors (SMPs). In: Proceedings of the th international parallel and distributed processing symposium (IPDPS ), Santa Fe
G
G
Graph Algorithms
. Bader DA, Cong G () Fast shared-memory algorithms for computing the minimum spanning forest of sparse graphs. In: Proceedings of the th international parallel and distributed processing symposium (IPDPS ), Santa Fe . Bader DA, Cong G, Feo J () On the architectural requirements for efficient execution of graph algorithms. In: Proceeding of the international conference on parallel processing, Oslo, pp – . Charles P, Donawa C, Ebcioglu K, Grothoff C, Kielstra A, Praun CV, Saraswat V, Sarkar V () X: an object-oriented approach to non-uniform cluster computing. In: Proceedings of the ACM SIGPLAN conference on object-oriented programming systems, languages and applications (OOPSLA), San Diego, pp – . Chiang Y-J, Goodrich MT, Grove EF, Tamassia R, Vengroff DE, Vitter JS () External-memory graph algorithms. In: Proceedings of the symposium on discrete algorithms, San Francisco, pp – . Chung S, Condon A () Parallel implementation of Bor˚uvka’s minimum spanning tree algorithm. In: Proceedings of the tenth international parallel processing symposium (IPPS’), Honolulu, pp – . Cole R, Vishkin U () Deterministic coin tossing and accelerating cascades: micro and macro techniques for designing parallel algorithms. In: STOC’: proceedings of the eighteenth annual ACM symposium on theory of computing. ACM, New York, pp – . Cole R, Vishkin U () Approximate parallel scheduling. part II: applications to logarithmic-time optimal graph algorithms. Info Comput :– . Cong G, Bader DA () An experimental study of parallel biconnected components algorithms on symmetric multiprocessors (SMPs). In: Proceedings of the th international parallel and distributed processing symposium (IPDPS ), Denver . Cong G, Almasi G, Saraswat V () Fast PGAS implementation of distributed graph algorithms. In: Proceedings of the ACM/IEEE international conference for high performance computing, networking, storage and analysis (SC), SC’. IEEE Computer Society, Washington, DC, pp – . CPI analysis on Power. On line, . http://www.ibm.com/ developerworks/linux/library/pacpipower/index.html . Cray, Inc. () The CRAY MTA- system. www.cray.com/ products/programs/mta_/ . Culler DE, Dusseau AC, Martin RP, Schauser KE () Fast parallel sorting under LogP: from theory to practice. In: Portability and performance for parallel processing. Wiley, New York, pp – (Chap ) . Dehne F, Götz S () Practical parallel algorithms for minimum spanning trees. In: Workshop on advances in parallel and distributed systems, West Lafayette, pp – . Eckstein DM () BFS and biconnectivity. Technical Report –, Department of Computer Science, Iowa State University of Science and Technology, Ames . Goddard S, Kumar S, Prins JF () Connected components algorithms for mesh-connected parallel computers. In: Bhatt
.
.
.
.
.
. .
. .
.
. . . . .
.
SN (ed) Parallel algorithms: third DIMACS implementation challenge, – October . DIMACS series in discrete mathematics and theoretical computer science, vol . American Mathematical Society, Providence, pp – Goodrich MT () Communication-efficient parallel sorting. In STOC’: proceedings of the twenty-eighth annual ACM symposium on theory of computing. ACM, New York, pp – Greiner J () A comparison of data-parallel algorithms for connected components. In: Proceedings of the sixth annual symposium on parallel algorithms and architectures (SPAA-), Cape, May, pp – Helman DR, JáJá J () Designing practical efficient algorithms for symmetric multiprocessors. In: Algorithm engineering and experimentation (ALENEX’). Lecture notes in computer science, vol . Springer, Baltimore, pp – Hofstee HP () Power efficient processor architecture and the cell processor. In: International symposium on high-performance computer architecture, San Francisco, pp – Hsu T-S, Ramachandran V, Dean N () Parallel implementation of algorithms for finding connected components in graphs. In: Bhatt SN (ed) Parallel algorithms: third DIMACS implementation challenge, – October . DIMACS series in discrete mathematics and theoretical computer science, vol . American Mathematical Society, Providence, pp – JáJá J () An introduction to parallel algorithms. AddisonWesley, New York Krishnamurthy A, Lumetta SS, Culler DE, Yelick K () Connected components on distributed memory machines. In Bhatt SN (ed) Parallel algorithms: third DIMACS implementation challenge, – October . DIMACS series in discrete mathematics and theoretical computer science, vol . American Mathematical Society, Providence, pp – Kruskal CP, Rudolph L, Snir M () Efficient parallel algorithms for graph problems. Algorithmica ():– Paul WJ, Vishkin U, Wagener H () Parallel dictionaries in – trees. In: Tenth colloquium on automata, languages and programming (ICALP), Barcelona. Lecture notes in computer science. Springer, Berlin, pp – Scarpazza DP, Villa O, Petrini F () Efficient breadth-first search on the Cell/BE processor. IEEE Trans Parallel Distr Syst ():– Shiloach Y, Vishkin U () An O(logn) parallel connectivity algorithm. J Algorithms ():– Tarjan RE () Depth-first search and linear graph algorithms. SIAM J Comput ():– Tarjan RE, Vishkin U () An efficient parallel biconnectivity algorithm. SIAM J Comput ():– Unified Parallel C, URL: http://en.wikipedia.org/wiki/Unified_ Parallel_C Woo J, Sahni S () Load balancing on a hypercube. In: Proceedings of the fifth international parallel processing symposium, Anaheim. IEEE Computer Society, Los Alamitos, pp – Woo SC, Ohara M, Torrie E, Singh JP, Gupta A () The SPLASH- programs: characterization and methodological
Graph Partitioning
considerations. In: Proceedings of the nd annual international symposium computer architecture, pp – . Yoo A, Chow E, Henderson K, McLendon W, Hendrickson B, Çatalyürek ÜV () A scalable distributed parallel breadth-first search algorithm on BlueGene/L. In: Proceedings of supercomputing (SC ), Seattle
Graph Analysis Software SNAP (Small-World Network Analysis and Partitioning) Framework
Graph Partitioning Bruce Hendrickson Sandia National Laboratories, Albuquerque, NM, USA
Definition Graph partitioning is a technique for dividing work amongst processors to make effective use of a parallel computer.
G
units of work) need to be divided into P sets with about the same number of vertices in each. Additionally, the number of edges that connect vertices in two different sets needs to be kept small since these will reflect the need for interprocessor communication. This problem is known as graph partitioning and is an important approach to the parallelization of many applications. More generally, the vertices of the graph can have weights associated with them, reflecting different amounts of computation, and the edges can also have weights corresponding to different quantities of communication. The graph partitioning problem involves dividing the set of vertices into P sets with about the same amount of total vertex weight, while keeping small the total weight of edges that cross between partitions. This problem is known to be NP-hard, but a number of heuristics have been devised that have proven to be effective for many parallel computing applications. Several software tools have been developed for this problem, and they are an important piece of the parallel computing ecosystem. Important algorithms and tools are discussed below.
Discussion
Parallel Computing Applications of Graph Partitioning
When considering the data dependencies in a parallel application, it is very convenient to use concepts from graph theory. A graph consists of a set of entities called vertices, and a set of pairs of entities called edges. The entities of interest in parallel computing are small units of computation that will be performed on a single processor. They might be the work performed to update the state of a single atom in a molecular dynamics simulation, or the work required to compute the contribution of a single row of a matrix to a matrix-vector multiplication. Each such work unit will be a vertex in the graph which describes the computation. If two units have a data dependence between them (i.e., the output of one computation is required as input to the other), then there will be an edge in the graph that joins the two corresponding vertices. For a computation to perform efficiently on a parallel machine each of the P processors needs to have about the same amount of work to perform, and the amount of inter-processor communication must be small. These two conditions can be viewed in terms of the computational graph. The vertices of the graph (signifying
Graph partitioning is a useful technique for parallelizing many scientific applications. It is appropriate when the calculation consists of a series of steps in which the computational structure and data dependencies do not vary much. Under such circumstances the expense of partitioning is rewarded by improved parallel performance for many computational steps. The partitioning model is most applicable for bulk synchronous parallel applications in which each step consists of local computation followed by a global data exchange. Fortunately, many if not most scientific applications exhibit this basic structure. Particle simulations are one such important class of applications. The particles could be atoms in a material science or biological simulation, stars in a simulation of galaxy formation, or units of charge in an electromagnetic application. But by far the most common uses of graph partitioning involve computational meshes for the solution of differential equations. Finite volume, finite difference, and finite element methods all involve the decomposition of a complex geometry into simple shapes that interact only with near neighbors. Various graphs can be
G
G
Graph Partitioning
constructed from the mesh and a partition of the graph identifies subregions to be assigned to processors. Th